Cache Reader Plugin

Moderators: grovkillen, Stuntteam, TD-er

Post Reply
Message
Author
chemmex
Normal user
Posts: 92
Joined: 15 Feb 2019, 16:18

Cache Reader Plugin

#1 Post by chemmex » 12 Mar 2023, 21:38

Inspired by the 20230304 release notes, I have built a custom image with P146 included, but so far a lot of questions remain.

I'm particularly interested in bulk sending of stored data. TD-er, I realize that writing docs takes even more effort than writing code itself, but can you give an idea of how it is intended to work?

Thanks

TD-er
Core team member
Posts: 8643
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: Cache Reader Plugin

#2 Post by TD-er » 12 Mar 2023, 22:07

It can either re-send trough the original task ("Send Bulk" unchecked) , or it can 'dump' it via MQTT ("Send Bulk" checked)
You can also click on the "Download as CSV" button.

When dumping to MQTT, the taskinfo will be sent to a separate MQTT topic (well at least there is a separate field to set a separate topic is you wish)

There are 2 options for sending it to MQTT:
- Send as HEX encoded binary
- Send as CSV (the same layout as on the CSV file export)

The binary HEX encoded is the same format as how it is stored on the flash and a Python decode script is present in the misc folder to give you some inspiration. :)

The CSV option dumping to MQTT does also include some header per message to indicate what is to be expected.
- start file nr
- start pos
- end file nr
- end pos
- Nr of CSV lines
- Length in bytes (incl. newline) of the CSV lines

Then what follows is a number of CSV lines, depending on the nr. of tasks included in the dump and the max. message size.
N.B. on ESP8266 do not use large messages due to memory.

You can set the separator and whether you only want to export tasks which have a plugin set to them.

The "Join Samples with same Timestamp" is just what is states.
This will then also be shown in the "PluginID" and "Task index" columns as those will then show which tasks/plugins were combined in the same line.

You can also set a specific timestamp when saving to the cache controller, using the new command "TaskRunAt"

Example rules set:

Code: Select all

On gps#All Do
  Let,4,%unixtime%
  LoopTimerSet,1,[int#2] // re-set the timer to have at least [int#2] seconds between samples.
  TaskValueSet,gps1,long,[gps#long] 
  TaskValueSet,gps1,lat,[gps#lat] 
  TaskValueSet,gps1,alt,[gps#alt] 
  TaskValueSet,gps1,spd,[gps#spd] 
  TaskRunAt,gps1,[int#4]
  Event,saveAllToFlash
Endon

On Rules#Timer=1 Do
  Let,4,%unixtime%
  Event,saveAllToFlash
Endon

On sunrise#All Do
   // Copy the last measurement to the dummy task
   // Since we have 2 fields left, we also keep the previous value
   TaskValueSet,co2,co2_prev,[co2#co2]
   TaskValueSet,co2,temp_prev,[co2#temp]
   TaskValueSet,co2,co2,[sunrise#co2]
   TaskValueSet,co2,temp,[sunrise#T]
   TaskValueSet,error,sunrise_prev,[error#sunrise]
   TaskValueSet,error,sunrise,[sunrise#err]
Endon

On saveAllToFlash Do

   // Now collect  other data to store into 'dummy' tasks.
   // For this, we use the command TaskValueSet.
   // See: https://espeasy.readthedocs.io/en/latest/Rules/Rules.html#taskvalueset
   
   // First we get the other values from the GPS unit and store it  in the dummy task called "gps2"
   // See: https://espeasy.readthedocs.io/en/latest/Plugin/P082.html#access-to-all-measurement-values
   // N.B. the first 2 arguments of the TaskValueSet command are resp. the task name and variable name of a dummy task
   // So make sure these match with an enabled dummy task
   TaskValueSet,gps2,hdop,[gps#hdop] 
   TaskValueSet,gps2,satvis,[gps#sat_vis] 
   TaskValueSet,gps2,sattracked,[gps#sat_tr] 
   TaskValueSet,gps2,chksumfail,[gps#chksum_fail] 
   // Calling TaskRun will cause the values to be sent to the controller(s) assigned to the task
   TaskRunAt,gps2,[int#4]
   
   // Now we collect the latest samples of the analog voltages
   TaskValueSet,analog,carBat,[bat#Analog.avg3]
   TaskValueSet,analog,backupBat,[batBackup#Analog.avg5]
   // And flush them to the Cache Controller
   TaskRunAt,analog,[int#4]
   
   // Now we collect the latest samples of the BME280
   TaskValueSet,bme2,Temperature,[bme280#Temperature]
   TaskValueSet,bme2,Humidity,[bme280#Humidity]
   TaskValueSet,bme2,Pressure,[bme280#Pressure]
   // And flush them to the Cache Controller
   TaskRunAt,bme2,[int#4]
   
   // Flush current and previous CO2 value to flash
   TaskRunAt,co2,[int#4]
   
   // Store the samples from the last reading from the PMS5003
   TaskRunAt,pms1,[int#4]
   TaskRunAt,pms2,[int#4]
   
   // Collect data from the (optional) Accelerometer 
   TaskRunAt,adxl345,[int#4]
   
   // Collect some sysinfo
   TaskRunAt,sysinfo,[int#4]  
   
   // Collect error states
   TaskRunAt,error,[int#4]  
Endon
This is collecting data to several dummy tasks and then flushing it to the cache controller.

chemmex
Normal user
Posts: 92
Joined: 15 Feb 2019, 16:18

Re: Cache Reader Plugin

#3 Post by chemmex » 12 Mar 2023, 22:34

Thanks a lot.

I am able to save sensor data to the cache files; the question is rather how can I control sending the cached readings to the MQTT. Are there separate commands or it is scheduled by the plugin? So far I am not seeing anything in the logs or at the broker.
Last edited by chemmex on 12 Mar 2023, 22:39, edited 1 time in total.

TD-er
Core team member
Posts: 8643
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: Cache Reader Plugin

#4 Post by TD-er » 12 Mar 2023, 22:39

To force flushing any 'cache' file on the filesystem, give this command:

Code: Select all

cachereader,setreadpos,0,0
To force sending the task info to the MQTT broker:

Code: Select all

cachereader,sendtaskinfo

chemmex
Normal user
Posts: 92
Joined: 15 Feb 2019, 16:18

Re: Cache Reader Plugin

#5 Post by chemmex » 12 Mar 2023, 22:59

No luck with both commands. Below is plugin setup:
146.png
146.png (143.39 KiB) Viewed 2120 times
and task info:
8.png
8.png (11.44 KiB) Viewed 2120 times
Log only shows the command itself

Code: Select all

1483080: HTTP: cachereader,setreadpos,0,0
1483086: CacheReader : SetReadPos,0,0
but nothing is sent and the cachefile size remains the same

TD-er
Core team member
Posts: 8643
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: Cache Reader Plugin

#6 Post by TD-er » 12 Mar 2023, 23:06

The position in the task info screenshot show you have a FilePos of 1824, so it has been sending something.
Do you have some MQTT explorer to look at topics?
Is the node allowed to publish to these topics?
Is the MQTT controller connected and doesn't loose connection?

chemmex
Normal user
Posts: 92
Joined: 15 Feb 2019, 16:18

Re: Cache Reader Plugin

#7 Post by chemmex » 12 Mar 2023, 23:15

Yes to all three questions. The node regularly publishes data from the rules. The broker has no restrictions.
As to the FilePos of 1824, it is also the size of cacheFile, it contains the sensor data collected so far (77 lines in dumped CSV file):

fb.png
fb.png (6.9 KiB) Viewed 2113 times

TD-er
Core team member
Posts: 8643
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: Cache Reader Plugin

#8 Post by TD-er » 12 Mar 2023, 23:29

There might (still) be an issue here if the write pointer is the same as the read pointer.
So maybe you can try to power cycle the unit, to make sure the read pos is actually 0.
The read pos and file nr. are stored in RTC memory, so it will be kept even through reboots, but not power cycle.

chemmex
Normal user
Posts: 92
Joined: 15 Feb 2019, 16:18

Re: Cache Reader Plugin

#9 Post by chemmex » 12 Mar 2023, 23:50

I will try with a clean config tomorrow.

Thanks and good night!

TD-er
Core team member
Posts: 8643
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: Cache Reader Plugin

#10 Post by TD-er » 13 Mar 2023, 00:12

Please first try a power cycle, to see if it will upload the 1st file.
I think it might no longer be a problem if there are >1 cache files.

It will also continue to upload new samples as soon as they are written to the flash.

N.B. forcing download the CSV file will also trigger to save whatever is still in RTC and can be flushed to flash.

chemmex
Normal user
Posts: 92
Joined: 15 Feb 2019, 16:18

Re: Cache Reader Plugin

#11 Post by chemmex » 13 Mar 2023, 11:40

Nothing changed so far after power cycle.

My current config is following:

- two controllers (C005/MQTT and C016/Cache)
- SHT30 sensor task sending data each 20 seconds to Cache Controller
- Cache reader plugin configured as few posts above
- a rule publishing some telemetry data to the broker configured as C005 each minute

I can see that SHT data is cached and file size is growing; I can dump this data as CVS file in the Cache reader plugin interface, everything seems valid

In the plugin, I have Send Bulk checked and MQTT topics configured, also send to C005 controller checked

What is expected but doesn't happen: Cache reader doesn't trigger sending to MQTT, nothing is seen in the log and in the tcpdump

TD-er
Core team member
Posts: 8643
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: Cache Reader Plugin

#12 Post by TD-er » 13 Mar 2023, 11:59

On ESP8266 the RTC memory is very small, like 240 bytes.
So this will flush to the 'cache files' every 10 "samples" (sends to C016)
ESP32 does have a lot more RTC memory, but as some trade-off between not loosing too much data and not wearing out the flash, the data is flushed every 768 bytes (32 "samples")
I find it strange that you don't see anything being written to the MQTT broker, unless the minimal time between messages is set too high. This can be set in the controller and there is a similar setting in the cache reader too.
If you set the minimal interval in the cache reader too low, no other task will be able to send anything to the controller.

Can you add something like this to the rules:

Code: Select all

// An event to send the sysinfo.
// This can also be triggered remote to request the sysinfo via MQTT by sending a command like:
// topic: tracker_v2/%sysname%/cmd
// message: event,sendSysinfo
On sendSysinfo Do
  publish tracker_v2/%sysname%/sysinfo/sysbuild_filename,"%sysbuild_filename%"
  publish tracker_v2/%sysname%/sysinfo/sysbuild_date,"%sysbuild_date%"
  publish tracker_v2/%sysname%/sysinfo/sysbuild_time,"%sysbuild_time%"
  publish tracker_v2/%sysname%/sysinfo/systime,"%systime%"
  publish tracker_v2/%sysname%/sysinfo/unixtime,"%unixtime%"
  publish tracker_v2/%sysname%/sysinfo/mac,"%mac%"
  
  // Also send the taskinfo used by the cache reader task.
  cachereader,sendtaskinfo  
Endon



On MQTT#Connected Do
  // Only automatically send the sysinfo the first time the unit connects to the MQTT broker since boot.
  If [int#1] = 0
    Let,1,1
    AsyncEvent,sendSysinfo
  endif  
Endon
As you can see, by me copy/paste quite extensive rules sets, I have it running on a number of nodes here.
So I really wonder what might be wrong on your config to why it isn't working.

Anyway, do you see the sysinfo being published?
Maybe the data is sent to the MQTT controller before it is connected and with a very short timeout it might already be purged from the queue due to timeouts?
So that's why you might want to trigger such an upload via the MQTT#connected event.

chemmex
Normal user
Posts: 92
Joined: 15 Feb 2019, 16:18

Re: Cache Reader Plugin

#13 Post by chemmex » 13 Mar 2023, 12:41

Can you add something like this to the rules:
CODE: SELECT ALL
In fact, I am publishing some basic system info upon MQTT#Connected, so this part works without problems. Except for this line:

Code: Select all

  // Also send the taskinfo used by the cache reader task.
  cachereader,sendtaskinfo  
This doesn't send anything

Probably, I missed something in my custom build?

chemmex
Normal user
Posts: 92
Joined: 15 Feb 2019, 16:18

Re: Cache Reader Plugin

#14 Post by chemmex » 13 Mar 2023, 13:05

Suddenly, I got it working after full reset of the node, so I am receiving CSV data in the upload topic

TD-er
Core team member
Posts: 8643
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: Cache Reader Plugin

#15 Post by TD-er » 13 Mar 2023, 15:32

Ah good to know.

Post Reply

Who is online

Users browsing this forum: No registered users and 23 guests