[Serial Proxy] LoRa gateway for beehive scales

[chromo23]

Hi all,

it is time to upgrade my beehive scales. Driven by a raspberry pi, they are bound to a power adapter and WiFi transmission.
I now decided to go with a battery/solar powered LoRa solution where nodes send data to a receiver which is connected to ESPEasy and is sending the received data via serial.

I couldn´t find any examples about the serial proxy plugin.
How do i process the data that is send to ESPEasy? Which would be the best syntax?

Let`s say i send the data with this syntax:
"node=1;weight=40;temp1=20;rssi=-30"
or
"node=2;weight=30;temp1=22;rssi=-35"

How do i process this data depending on the node.
Like:

Code: Select all

On Serial#receive Do
	If %eventvalue1% = 1     //if node = 1
	  let,1,%eventvalue2%   //store weight
	  let,2,%eventvalue3%. //store temperature
	  let,3,%eventvalue4%. //store rssi
	Elseif  %eventvalue1% = 2
	  let,4,%eventvalue2%
	  let,5,%eventvalue3%
	  let,6,%eventvalue4%
	Endif
Endon

Can this be done with filtering? I do not understand at all how filtering works and what it is good for.
Using substring is probably not an option i guess since the length of the string is constantly changing....

[Ath]

I did some investigation, and have a couple of findings:
- The regex dialect used is based on Lua, that has some peculiarities: (I'm probably a bit spoiled with all the regex features available in my day to day used programming languages )
- The escape char is '%' instead of the more common '\'
- (result) Group indexes are 0-base instead of the usual 1-base
- There is no support for non-capturing groups
With that knowledge I've cobbled this regex to match your expected input:

Code: Select all

((node)=(%d+);?)((weight)=(%d+);?)((temp%d?)=(%-?%d+);?)((rssi)=(%-?%d+);?)

But I've been unable so far to make any separate value optional, it seems like groups can't be optional, so all are required to be present...

Once that hurdle was passed, the filtering was investigated, and it turns out this filtering is only applied when selecting either Global Match or Global Match inverted. And then it still is an OR match, so if any of the comparisons returns true, the outcome will be true. But at least 1 of the comparisons has to be true.
For the other Match Type options, only the regex match determines the 'output'.

The 'output' is the configured variable (v) having the value of the received string, so no separate parsed results, as you might have hoped

Code: Select all

EVENT: SerialProxy#v=`node=1;weight=40;temp1=20;rssi=-30`

For this to be split up, we might need to implement more string parsing functions, and possibly also 'proper' string handling, but that's not yet available, and also 'quite a task' to implement

[chromo23]

Thanks.
Let me get this straight since i am not sure if i understood correctly:

1. The serial proxy plugin receives data and puts it (if no regex and filtering is active) as a string into the "v"-value
Example with to different inputs:

Code: Select all

15389791: EVENT: SerialProxy#v=`node=1;weight=199;temp1=219;rssi=217`
15389801: EVENT: SerialProxy#v=`node=2;weight=210;temp2=216;rssi=282`

2. With regex you can filter out whole lines if the condition is not met and/or divide strings into groups
Regex:

Code: Select all

((node)=(%d+);?)((weight)=(%d+);?)((temp1)=(%-?%d+);?)((rssi)=(%-?%d+);?)

Output:

Code: Select all

15555788: EVENT: SerialProxy#v=`node=1;weight=268;temp1=287;rssi=261`

Since the Line with "node=2" contains "temp2" and not "temp1" it gets filtered out.

3. If I now add a capture filter like: "Capture Filter 1: 4 == weight" the output in the logs looks like this:

Code: Select all

16041643: P087: Index: 4 Found weight Matches (==)
  16041646: EVENT: SerialProxy#v=`node=2;weight=203;temp2=204;rssi=193`

My conclusion. I can filter out whole lines.. great. but other than that i can not see any use for this plugin, since even though the capture filter works as expected i can not make use of it and the event output is always the whole line in case a capture filter is true. So what is this plugin for if you can´t make use of the serial input?

Btw: This is helping with this form of regex: https://gitspartv.github.io/lua-pattern ... 2B)%3B%3F)

I will try to change the plugin for me in a way, that the event is generated without quotation marks. Then i will send just numbers in csv format:

Code: Select all

SerialProxy#v=2,210,216,282

What do you think? Is there a simpler better/approach?

[Ath]

My conclusion. I can filter out whole lines.. great. but other than that i can not see any use for this plugin, since even though the capture filter works as expected i can not make use of it and the event output is always the whole line in case a capture filter is true. So what is this plugin for if you can´t make use of the serial input?

That's what I was wondering about when investigating this...

I'll see if I can add a feature to fetch the separate parsed-regex results, so it would be a more useful plugin, just give me a few evenings (I'll start today)

Btw: This is helping with this form of regex: https://gitspartv.github.io/lua-pattern ... 2B)%3B%3F)

Thanks!

I will try to change the plugin for me in a way, that the event is generated without quotation marks. Then i will send just numbers in csv format:

Code: Select all

SerialProxy#v=2,210,216,282

What do you think? Is there a simpler better approach?

Well, the quotes have to stay, as else it could be unpredictable what %eventvalue% gets what value, when commas are included... You can send the value to another event to parse it separately (example below)
But more importantly: for backward compatibility

Example: (assuming 4 comma-separated values)

Code: Select all

on SerialProxy#v do
  AsyncEvent,ValueParser=%eventvalue1%
endon

on ValueParser do
  // Expect n values from serial input, 1 = Node, 2 = Weight, 3 = Temp, 4 = RSSI
  if %eventvalue1% > 0
    TaskValueSet,Node%eventvalue1%,Weight,%eventvalue2%
    TaskValueSet,Node%eventvalue1%,Temp,%eventvalue3%
    TaskValueSet,Node%eventvalue1%,RSSI,%eventvalue4%
  endif
endon

[Ath]

Btw: This is helping with this form of regex: https://gitspartv.github.io/lua-pattern ... 2B)%3B%3F)

Hmm, that proves that either the Lua regex parser is quite limited, and/or that site has it's limitations too, as in the %-? parts, the quantifying question mark is either ignored or not supported...

[chromo23]

Hmm, that proves that either the Lua regex parser is quite limited, and/or that site has it's limitations too, as in the %-? parts, the quantifying question mark is either ignored or not supported...

You are right.. I hadn't even noticed

I'll see if I can add a feature to fetch the separate parsed-regex results, so it would be a more useful plugin, just give me a few evenings (I'll start today)

Example: (assuming 4 comma-separated values)

Thats great... it works perfectly!

Code: Select all

on SerialProxy#v do
  AsyncEvent,ValueParser=%eventvalue1%
endon

This doesn´t and maybe it is also a reason the plugin doesn´t work as expected?
To catch the event you need to use a wildcard:

Code: Select all

on SerialProxy* do
  AsyncEvent,ValueParser=%eventvalue1%
endon

[chromo23]

BTW:

I am using an esp8266. Data is send with 9600bps.
For testing i was sending two different lines (as in the example of one of my previous posts) alternating every second.
The esp is constantly missing one or the other line. I tried HW serial ans SW serial. It happend with both configurations. Load of the esp is at 10%. It shouldn't loose information at this speed in my opinion.
Could this be also an issue of the plugin?

Edit: just wrote a routine in rules that checks that no line is missing. it seems that i was maybe only an issue with the log viewer...

[Ath]

To catch the event you need to use a wildcard:

Code: Select all

on SerialProxy* do
  AsyncEvent,ValueParser=%eventvalue1%
endon

Ah, yes, I'm not using the wildcard often, as most of my units handle only numeric data

[chromo23]

Ah, yes, I'm not using the wildcard often, as most of my units handle only numeric data

Me neither
So in general the "pluginname#value" combinatin only gets recognized in rules when it contains a numeric value?

[Ath]

Well, the default behavior is to check the first eventvalue to be numeric, so to be able to handle non-numeric values, the wildcard was introduced. It's documented here, but was already introduced long before that date, probably to accommodate P020 and P044 for handling the RFLINK and P1 data, and maybe even earlier.

[chromo23]

There is another tester for lua pattern: https://montymahato.github.io/lua-pattern-tester/
Maybe we can add one to the docs along with the hint, that the regex is actually lua pattern.
I does the quantifier after an escaped character or class right. But i needed to write an issue there because the whole site disappears when testing "(%()" and entering a test text containing "(".
(I also opened an issue at the other site because of the missing quantifier.)