SerialProxy - how to get data

[mgrunt]

Hi, how can I get for example value "V" from this stream?

Code: Select all

PID	0xA060
FW	159
SER#	HQ21069E3M5
V	26280
I	750
VPV	31920
PPV	20
CS	3
MPPT	2
OR	0x00000000
ERR	0
LOAD	OFF

I've tried everything but it doesn't work. Thank you.

[Ath]

Can you show a configuration you have tried so far?

[Ath]

Hi, how can I get for example value "V" from this stream?

Code: Select all

PID	0xA060
FW	159
SER#	HQ21069E3M5
V	26280
I	750
VPV	31920
PPV	20
CS	3
MPPT	2
OR	0x00000000
ERR	0
LOAD	OFF

I've tried everything but it doesn't work. Thank you.

The documentation isn't rich on examples, but the basic information is available: https://espeasy.readthedocs.io/en/lates ... #p087-page

This plugin needs a regular expression, in the Lua dialect, and uses groups, to match and split the input.
The regular expression to parse your input could be this:

Code: Select all

(%w+#?)%s+(%w+)

- (Group) any alphanumeric word with optional # at the end
- 1 or more white-space characters (space, tab)
- (Group) any alphanumeric word

When an input that matches the regex is received, the event SerialProxy#v=`(received input)` is generated to be handled in Rules.
If you want to limit the matches that should be processed, the Match Type setting should be changed to Global Match, and Capture filters can be set to check for (group) 0 == V (case sensitive!) to only let the lines using the V value pass.

To handle the data in rules there's a little snag, as rules are by default only processed if the first passed argument is numeric, and the input provided here is not numeric..., but the way to handle that is prefix the event name with * so string values will also cause the rule to be processed.
A test rule I used for a task named SerialProxy looks like this:

Code: Select all

On *serialproxy#v Do
  LogEntry,'Match P1: [serialproxy#group.0], P2: [serialproxy#group.1] V: [serialproxy#next.V] ERR: [serialproxy#next.ERR]'
  If {equals:V:'[serialproxy#group.0]'}=1
    LogEntry,'Processing V value: [serialproxy#group.1]'
  Endif
  If {equals:ERR:'[serialproxy#group.0]'}=1
    LogEntry,'Processing ERR value: [serialproxy#group.1]'
  Endif
Endon

This also shows 2 ways of getting the value for a named argument like V or ERR, either check the first group (0) and take the value from the second group (1), or fetch the 'next' value from the group after the V or ERR group. V and ERR are compared not case-sensitive here.

[mgrunt]

Sorry for not replying earlier I was busy... Thanks for the detailed description! I'm starting to understand how it works. It's a pity that this example is not given in the documentation or I didn't understand it from the documentation. I'll try to play with it in a few days.

I will let you know how it turned out and what my experiences are.

Is there any way to temporarily let the control list everything (not just the result of the expression) that goes to the serial port in the log?

[Ath]

It's a pity that this example is not given in the documentation or I didn't understand it from the documentation. I'll try to play with it in a few days.

It's quite hard to write good documentation, and having a real-world example is really helpful. I'll see how I can use this for documentation.

Is there any way to temporarily let the control list everything (not just the result of the expression) that goes to the serial port in the log?

To reduce the load on the ESP we don't currently log all received data. I could add an option to the settings, but when leaving the regular expression empty it is supposed to pass through all incoming data to Rules, where you can process it, including writing it all to the log.

Edit:
You can test your regex and filter(s) by sending simulated input via the SerialProxy_Test,"some test data" command.

[mgrunt]

Now I tried it and got this result.

RegEx: (%w+)%s+(%w+)$
I tried the dollar string at the end, it might detect a newline for better separation (Isn't that a bad idea?)

Filter: Disabled

Rules: none

Output from MPPT and input to ESP32 (values ​​are now irrelevant...):

Code: Select all

PID	0xA053
FW	164
SER#	HQxxxxxxxxx
V	26280
I	750
VPV	31920
PPV	20
CS	3
MPPT	2
OR	0x00000000
ERR	0
LOAD	OFF
IL	0
H19	5298
H20	68
H21	485
H22	75
H23	517
HSDS	245
Checksum

LOG in ESPEasy

Code: Select all

1664663830: EVENT: serialproxy#v=`H21)135`
1664670793: EVENT: serialproxy#v=`CS)3`
1664677816: EVENT: serialproxy#v=`ERR)0`
1664683808: EVENT: serialproxy#v=`PPV)16`
1664705265: EVENT: Clock#Time=Tue,14:26
1664717921: EVENT: serialproxy#v=`MPPT`
1664717937: EVENT: serialproxy#v=`2`
1664761938: EVENT: serialproxy#v=`PPVI16`
1664762911: EVENT: serialproxy#v=`V`
1664762933: EVENT: serialproxy#v=`13000`
1664763973: EVENT: serialproxy#v=`H19I764t`
1664765264: EVENT: Clock#Time=Tue,14:27
1664765913: EVENT: serialproxy#v=`FW`
1664766060: EVENT: serialproxy#v=`164`
1664768968: EVENT: serialproxy#v=`IL`
1664768989: EVENT: serialproxy#v=`2300`
1664792016: EVENT: serialproxy#v=`IL`
1664792035: EVENT: serialproxy#v=`2300`
1664792953: EVENT: serialproxy#v=`PIDI0xA053`
1664800991: EVENT: serialproxy#v=`I`
1664801011: EVENT: serialproxy#v=`-1180`
1664816054: EVENT: serialproxy#v=`LOAD)ON`
1664818082: EVENT: serialproxy#v=`HSDS)269`

Why is there a separator somewhere ")" or "I" or is there none (and I have two lines for example one line is "IL" a second line is "2300")?

When I use your rule I get in LOG for example this:

Code: Select all

1670252457: EVENT: serialproxy#v=`SER#IHQXXXXXXXX`
1670252462: ACT : LogEntry,'Match P1: , P2: V: ERR: '
1670252474: Match P1: , P2: V: ERR:
1670252477: Calculate: Unknown token
1670252482: Calculate: Unknown token
1670252513: EVENT: serialproxy#v=`22`
1670252518: ACT : LogEntry,'Match P1: , P2: V: ERR: '
1670252529: Match P1: , P2: V: ERR:
1670252533: Calculate: Unknown token
1670252538: Calculate: Unknown token

Furthermore, I have not yet understood how this plugin reads data. Does it continuously read the data into a buffer (how big is it?) or does it read it within, for example, a second and once every interval? Because, for example, I only see the "V" value once every x minutes, while MPPT sends all the above data every second.

Thank you for the detailed explanation above. I already understand something, but not everything yet...

EDIT: Can't the whole thing be broken by the item "Checksum" behind which there is a mostly unprintable character (sometimes also printable)?

[Ath]

This looks like your serial settings on ESP side don't match the outgoing serial on your external device. Baud rate is probably ok (what is the actual baud rate?), but something in the parity/databits/stopbits seems to have a mismatch.

[Ath]

RegEx: (%w+)%s+(%w+)$
I tried the dollar string at the end, it might detect a newline for better separation (Isn't that a bad idea?)

The input is already split on any <CR> received, so that shouldn't be an issue. Similar for a ^ at the start of the regex.
You left out the #? part in the first token "(%w+#?)" to be able to capture the SER# line, unless you want to exclude that of course.

When I use your rule I get in LOG for example this:

That log will only work if the regex is processed, and that processing is skipped (to improve processing speed) if you disable the regex/filtering in the settings.

EDIT: Can't the whole thing be broken by the item "Checksum" behind which there is a mostly unprintable character (sometimes also printable)?

Once you enable the regex, it will ignore that Checksum line, as it probably doesn't have a white-space separator and is missing the second token.

[mgrunt]

I think the received data is fine. I did a little experiment where I redirected the data from the MPPT to the console (HW Serial0). Here is the result:

Code: Select all

50748446: Command unknown: FW164
50748455: Command unknown: SER#HQXXXXXX
50749376: Command unknown: Checksum
50749455: Command unknown: 543CR7
50749463: Command unknown: V13130
50750376: Command unknown: PID0xA053
50750458: Command unknown: 213543CR7
50750466: Command unknown: V13130
50751380: Command unknown: PID0xA053
50751399: Command unknown: CR7
50751473: Command unknown: CS3
50752379: Command unknown: PID0xA053
50752449: Command unknown: FW164
50752519: Command unknown: SER#HQXXXXXX
50752525: Command unknown: SDS270
50753390: Command unknown: FW16
50753471: Command unknown: 1440
50753479: Command unknown: VPV17910
50754393: Command unknown: PID0xA05164
50754524: Command unknown: SER#HQXXXXXX
50755385: Command unknown: PID0xA053
50755465: Command unknown: SER#HQXXXXXX
50755534: Command unknown: 18400
50756388: Command unknown: PID0xA053
50756467: Command unknown: 3CR7
50756475: Command unknown: V13130
50757386: Command unknown: A053
50757458: Command unknown: FW164
50757529: Command unknown: SER#HQXXXXXX
50757535: Command unknown: 70
50758386: Command unknown: D0xA053
50758396: Command unknown: FW164
50758469: Command unknown: 13130
50758537: Command unknown: 270
50759395: Command unknown: Checksum0xA053
50759415: Command unknown: 3130
50759488: Command unknown: MPPT2
50760396: Command unknown: PID
50760467: Command unknown: SER#HQXXXXXX
50760537: Command unknown: V13130
50761386: Command unknown: ecksum
50761457: Command unknown: PID0xA053
50761466: Command unknown: F164
50761534: Command unknown: H23141
50761603: Command unknown: HSDS270
50762388: Command unknown: Checksum
50762458: Command unknown: PID0xA053
50762467: Command unknown: FW164
50762545: Command unknown: H23141
50762614: Command unknown: HSDS270

Here is the data the MPPT sends every second:
(If I connect the MPPT serial line in the PC with the setting 19200-8-N-1, I get these data. I have the same parameters set in the serialproxy plugin.)

Code: Select all

PID	0xA053
FW	164
SER#	HQXXXXXX
V	26280
I	750
VPV	31920
PPV	20
CS	3
MPPT	2
OR	0x00000000
ERR	0
LOAD	OFF
IL	0
H19	5298
H20	68
H21	485
H22	75
H23	517
HSDS	245
Checksum

However, I am not sure whether data recording and processing is suitable for the serialproxy plugin. Because when we look at how the system records data, there are a lot of irrelevant lines where the data is not consistent depending on how the system collects data. Couldn't this be the whole problem?

ATH: Thank you for answers about regex. I will keep this in mind in the next tests.

[mgrunt]

I will still experiment. I try simulating with serial write command, dummy object and Arduino, which will simulate MTTP (data transmission). I'll let you know how it turned out.