[Generic - MQTT Import]: No option available

Moderators: grovkillen, Stuntteam, TD-er

Post Reply
Message
Author
fm-maniac
Normal user
Posts: 22
Joined: 07 Nov 2021, 12:39
Location: Germany

[Generic - MQTT Import]: No option available

#1 Post by fm-maniac » 09 Sep 2022, 11:23

How do I get the "advanced" options like mapping values as described in the docs: https://espeasy.readthedocs.io/en/lates ... /P037.html

They are unavailble on the web frontend:
2022-09-09 11_18_47-Window.png
2022-09-09 11_18_47-Window.png (23.45 KiB) Viewed 4396 times
I'm using a build from the latest dirtribution:

Build:⋄ 20116 - Mega
System Libraries:⋄ ESP82xx Core 2843a5ac, NONOS SDK 2.2.2-dev(38a443e), LWIP: 2.1.2 PUYA support
Git Build:⋄ mega-20220809_57b0c0c
Plugin Count:⋄ 47 [Normal]
Build Origin: GitHub Actions
Build Time:⋄ Aug 9 2022 10:01:11
Binary Filename:⋄ ESP_Easy_mega_20220809_normal_ESP8266_4M1M
Build Platform:⋄ Linux-5.10.102.1-microsoft-standard-WSL2-x86_64-with-glibc2.29
Git HEAD:⋄ mega-20220809_57b0c0c

Many thanks!

User avatar
Ath
Normal user
Posts: 3417
Joined: 10 Jun 2018, 12:06
Location: NL

Re: [Generic - MQTT Import]: No option available

#2 Post by Ath » 09 Sep 2022, 12:38

In the documentation there is this note:
NB: To save space it is possible that not all features are available in all builds. The screenshots are taken including all options.
Currently the only builds that include all available options are the MAX builds (requiring an ESP32 with 16MB Flash), or when creating a Custom build including this plugin. That should probably stated more clearly in the documentation, I'll add that.

The procedure for creating a Custom build is documented here, while that mostly describes how to start development for ESPEasy, it is also an effective way of creating a Custom build.

If you can not handle that yourself, it is possible that I create a Custom build and upload here, but I won't be able to periodically update that (or probably don't remember to do that :lol:).
There is a separate effort planned to provide a custom-build service, where you can select the plugins, controllers and options desired, and a .bin file is created for download, but that project has been delayed by other work.
/Ton (PayPal.me)

fm-maniac
Normal user
Posts: 22
Joined: 07 Nov 2021, 12:39
Location: Germany

Re: [Generic - MQTT Import]: No option available

#3 Post by fm-maniac » 09 Sep 2022, 13:36

Thanks for clarification!

I thought as this plugin has the status "normal" it's included fully featured in the normal build.

Not sure if I'm able to do a custom build on my own, as I've never done this before. I'll have to dig into it, if I'll find the time...

Maybe an additional question:

I tried to set a value of the MQTT plugin to a certain value from a rule:

Code: Select all

TaskValueSet,MQTT_in,SetLight,0
But this doesn't work as it does with any other taskvalues. Am I doing somethig wrong or is there another procedure to reset the values?

User avatar
Ath
Normal user
Posts: 3417
Joined: 10 Jun 2018, 12:06
Location: NL

Re: [Generic - MQTT Import]: No option available

#4 Post by Ath » 09 Sep 2022, 14:47

fm-maniac wrote: 09 Sep 2022, 13:36 I thought as this plugin has the status "normal" it's included fully featured in the normal build.

Not sure if I'm able to do a custom build on my own, as I've never done this before. I'll have to dig into it, if I'll find the time...
I'm working on a PR to achieve exactly that, I'll provide a link where you can download from Github Actions in a short while (requires to have a Github account and be logged in to download, but that's all available for free ;))
fm-maniac wrote: 09 Sep 2022, 13:36 I tried to set a value of the MQTT plugin to a certain value from a rule:

Code: Select all

TaskValueSet,MQTT_in,SetLight,0
But this doesn't work as it does with any other taskvalues. Am I doing somethig wrong or is there another procedure to reset the values?
Have you enabled the Tools/Advanced setting "Allow TaskValueSet on all plugins:"? As not having that enabled may be blocking your command.
/Ton (PayPal.me)

User avatar
Ath
Normal user
Posts: 3417
Joined: 10 Jun 2018, 12:06
Location: NL

Re: [Generic - MQTT Import]: No option available

#5 Post by Ath » 09 Sep 2022, 15:33

Once this Github Actions build is finished, you should be able to download the Binaries.zip file, that contains a set of .zip files, one per build, that contain the .bin file you need.
This is from Pull Request #4257

Edit: Updated the Actions build link
Last edited by Ath on 09 Sep 2022, 16:35, edited 1 time in total.
/Ton (PayPal.me)

User avatar
Ath
Normal user
Posts: 3417
Joined: 10 Jun 2018, 12:06
Location: NL

Re: [Generic - MQTT Import]: No option available

#6 Post by Ath » 09 Sep 2022, 16:34

Hadn't tested one configuration, so here's a fixed Actions build link.
/Ton (PayPal.me)

fm-maniac
Normal user
Posts: 22
Joined: 07 Nov 2021, 12:39
Location: Germany

Re: [Generic - MQTT Import]: No option available

#7 Post by fm-maniac » 09 Sep 2022, 16:38

Ath wrote: 09 Sep 2022, 14:47 Have you enabled the Tools/Advanced setting "Allow TaskValueSet on all plugins:"? As not having that enabled may be blocking your command.
Many thanks Ath! This did the trick!

For the github stuff, being a hardware guy I'm not familiar how this works. I've to check if I'm able to make it trough.

User avatar
Ath
Normal user
Posts: 3417
Joined: 10 Jun 2018, 12:06
Location: NL

Re: [Generic - MQTT Import]: No option available

#8 Post by Ath » 09 Sep 2022, 17:04

fm-maniac wrote: 09 Sep 2022, 16:38 Many thanks Ath! This did the trick!
You're welcome :D
fm-maniac wrote: 09 Sep 2022, 16:38 For the github stuff, being a hardware guy I'm not familiar how this works. I've to check if I'm able to make it trough.
I assume you mean the Custom build stuff? Don't worry, this improvement will probably soon be included in a next release of ESPEasy, so no need to do Custom builds for this, and even if you want to go that route, we're here to help you out, and give advice where needed.
/Ton (PayPal.me)

fm-maniac
Normal user
Posts: 22
Joined: 07 Nov 2021, 12:39
Location: Germany

Re: [Generic - MQTT Import]: No option available

#9 Post by fm-maniac » 09 Sep 2022, 17:09

Ath wrote: 09 Sep 2022, 16:34 Hadn't tested one configuration, so here's a fixed Actions build link.
Set up an account, downloaded the binaries from the given link and instantly did an update OTA.

Seems to work. Outstanding service!!! Highly appreciated.

I would be glad to have these options enabled in future releases if possible.

Once again, many thanks!

TD-er
Core team member
Posts: 8643
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: [Generic - MQTT Import]: No option available

#10 Post by TD-er » 09 Sep 2022, 21:44

fm-maniac wrote: 09 Sep 2022, 17:09 [...]
I would be glad to have these options enabled in future releases if possible.
[...]
It will be as that build is a test build for code that will be merged very soon (as in seconds after I post this message)
Thus in the next build it will be as in your test build.

Wiki
Normal user
Posts: 413
Joined: 23 Apr 2018, 17:55
Location: Germany

Re: [Generic - MQTT Import]: No option available

#11 Post by Wiki » 06 Oct 2022, 18:41

I've suffered the same issue. I needed it to configure my solar inverter almost in realtime depending on power consumtion of different power consumers. Thanks to @Ath it was solved before I had to ask for this. Downloaded the binaries, flashed, works.

Thank you - again and again - for your support, its amazing. In some days I will post my little solar inverter regulation story.

Wiki

Code: Select all

pi@raspberrypi:~ $ man woman
No manual entry for woman
pi@raspberrypi:~ $

TD-er
Core team member
Posts: 8643
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: [Generic - MQTT Import]: No option available

#12 Post by TD-er » 06 Oct 2022, 19:06

Wiki wrote: 06 Oct 2022, 18:41 [...]
Thank you - again and again - for your support, its amazing. In some days I will post my little solar inverter regulation story.
[...]
Please do!

Wiki
Normal user
Posts: 413
Joined: 23 Apr 2018, 17:55
Location: Germany

Re: [Generic - MQTT Import]: No option available

#13 Post by Wiki » 07 Oct 2022, 12:37

Short feedback.

I've flashed the binary ESP_Easy_mega_20220909_normal_ESP8266_4M1M.bin onto an existing, since ages productional, stable running Wemos D1 Mini. The options in MQTT (JSON import and filtering) worked instantly. But: device got very unstable, rebooting almost every minute. So on this device I went back to my old binary and took an old ESP32 out of my drawer.

On the ESP32 the MQTT import works fine, the device itself is running stable and has got now finally a useful job.

Code: Select all

pi@raspberrypi:~ $ man woman
No manual entry for woman
pi@raspberrypi:~ $

TD-er
Core team member
Posts: 8643
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: [Generic - MQTT Import]: No option available

#14 Post by TD-er » 07 Oct 2022, 12:44

Sounds like it may run out of memory on the ESP8266.
Is it connecting to a MQTT broker which has lots of messages being sent to the topic you may subscribe to on the ESP8266?

Edit:
You may also want to try this build: https://github.com/letscontrolit/ESPEas ... 3202941073
It includes WiFi fixes and other optimizations which may reduce memory usage.

Wiki
Normal user
Posts: 413
Joined: 23 Apr 2018, 17:55
Location: Germany

Re: [Generic - MQTT Import]: No option available

#15 Post by Wiki » 20 Oct 2022, 11:27

The build for the ESP32 works pretty well, performant and stable. Up to now I have three MQTT Import tasks running, uptime of the device 4 1/2 days (lasst reboot due to powercycle my complete home), free ram between 160k and 185k. Seems as if there is still a lot of air.

For my little project concerning the regulation of my micro solar inverter (max power 600 watt) the possibility of using the MQTT options makes life pretty much easier. Before I tried to accomplish my goal using a Raspberry PI and coding in python. Now my needed informations are only some clicks away - Thank you.

As soon as I have a presentable - better: understandable - solution I will post it here. Probably someone other could take it as an idea for his own project.

Code: Select all

pi@raspberrypi:~ $ man woman
No manual entry for woman
pi@raspberrypi:~ $

TD-er
Core team member
Posts: 8643
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: [Generic - MQTT Import]: No option available

#16 Post by TD-er » 20 Oct 2022, 11:31

Wiki wrote: 20 Oct 2022, 11:27 [...] free ram between 160k and 185k. Seems as if there is still a lot of air.
[...]
There has to be, given what's coming soon :)

Wiki
Normal user
Posts: 413
Joined: 23 Apr 2018, 17:55
Location: Germany

Re: [Generic - MQTT Import]: No option available

#17 Post by Wiki » 20 Oct 2022, 16:03

TD-er wrote: 20 Oct 2022, 11:31
Wiki wrote: 20 Oct 2022, 11:27 [...] free ram between 160k and 185k. Seems as if there is still a lot of air.
[...]
There has to be, given what's coming soon :)
Aha, making me curious.....

Code: Select all

pi@raspberrypi:~ $ man woman
No manual entry for woman
pi@raspberrypi:~ $

TD-er
Core team member
Posts: 8643
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: [Generic - MQTT Import]: No option available

#18 Post by TD-er » 20 Oct 2022, 16:17

MQTT over TLS :)

Sadly ESP32 only, as I really can't get it to work on ESP8266.

Still it will be with some "known issues" which will be documented (to prove they're known)
But rather have people test with it then me trying to iron out all the stuff only to be proven wrong on some very specific setups :)

Wiki
Normal user
Posts: 413
Joined: 23 Apr 2018, 17:55
Location: Germany

Re: [Generic - MQTT Import]: No option available

#19 Post by Wiki » 22 Oct 2022, 10:51

TD-er wrote: 20 Oct 2022, 16:17 MQTT over TLS :)

Sadly ESP32 only, as I really can't get it to work on ESP8266.

Still it will be with some "known issues" which will be documented (to prove they're known)
But rather have people test with it then me trying to iron out all the stuff only to be proven wrong on some very specific setups :)
[OT]
Ah, OK.

This sounds somewhat as if in the future at least the ESP32 might be able to send notifications over SMTPS?
[/OT]

Code: Select all

pi@raspberrypi:~ $ man woman
No manual entry for woman
pi@raspberrypi:~ $

TD-er
Core team member
Posts: 8643
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: [Generic - MQTT Import]: No option available

#20 Post by TD-er » 22 Oct 2022, 20:09

Maybe but first let's make it possible to connect to MQTT using TLS.
That's hard enough.

Wiki
Normal user
Posts: 413
Joined: 23 Apr 2018, 17:55
Location: Germany

Re: [Generic - MQTT Import]: No option available

#21 Post by Wiki » 24 Oct 2022, 16:26

Short report of experiences:

Using the binaries Ath has published on 9th of september, flashed on an ESP32, works stable up to a number of six MQTT Import tasks. The topics are published from other devices every 10-15 secs, mostly JSON filtered topics to/from Domoticz. Adding a seventh import task results in reboots in a frequence of about a minute or so.
I don't know if its interesting, but if wanted and needed just tell me which further informations you would like to have and I will try to post it in the issues section of the project.

Clearly pointed out: I am very fine and grateful with the functionality of the MQTT Import up to now, no need for reactions. I am aware that I am putting a somewhat huge load on the small, tiny device and did never ever imagine before that it would be possible this way. My current small project is just an interim solution to bridge the time until the new energy counter and my Volkszähler will arrive. And its only to cure a kind of curiosity about whats possible. I am collecting power consumption of different devices to calculate an overrall consumption of my home to manipulate the solar inverter accordingly.

Code: Select all

pi@raspberrypi:~ $ man woman
No manual entry for woman
pi@raspberrypi:~ $

TD-er
Core team member
Posts: 8643
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: [Generic - MQTT Import]: No option available

#22 Post by TD-er » 24 Oct 2022, 16:40

What is the reboot reason when using 7 MQTT imports tasks (on your system) ?

I would expect it is either some WDT reset or maybe some exception where it may try to access memory which failed to allocate. (or just no longer there when processing the callback function)

Wiki
Normal user
Posts: 413
Joined: 23 Apr 2018, 17:55
Location: Germany

Re: [Generic - MQTT Import]: No option available

#23 Post by Wiki » 24 Oct 2022, 18:03

I will report tomorrow, when sun is shining ;-)

Code: Select all

pi@raspberrypi:~ $ man woman
No manual entry for woman
pi@raspberrypi:~ $

Wiki
Normal user
Posts: 413
Joined: 23 Apr 2018, 17:55
Location: Germany

Re: [Generic - MQTT Import]: No option available

#24 Post by Wiki » 25 Oct 2022, 11:39

I've took a series of snapshots showing the last reboot reasons:
Absturz5.JPG
Absturz5.JPG (26.52 KiB) Viewed 3350 times
Absturz4.JPG
Absturz4.JPG (29.83 KiB) Viewed 3350 times
Absturz3.JPG
Absturz3.JPG (26.28 KiB) Viewed 3350 times
Absturz2.JPG
Absturz2.JPG (29.83 KiB) Viewed 3350 times
Absturz1.JPG
Absturz1.JPG (27.63 KiB) Viewed 3350 times
Absturz.JPG
Absturz.JPG (50.24 KiB) Viewed 3350 times
Hope, this helps.

Code: Select all

pi@raspberrypi:~ $ man woman
No manual entry for woman
pi@raspberrypi:~ $

Wiki
Normal user
Posts: 413
Joined: 23 Apr 2018, 17:55
Location: Germany

Re: [Generic - MQTT Import]: No option available

#25 Post by Wiki » 25 Oct 2022, 11:55

Noticed two more, different reasons:
Absturz6.JPG
Absturz6.JPG (52.42 KiB) Viewed 3348 times
Absturz7.JPG
Absturz7.JPG (54.54 KiB) Viewed 3348 times

Code: Select all

pi@raspberrypi:~ $ man woman
No manual entry for woman
pi@raspberrypi:~ $

Wiki
Normal user
Posts: 413
Joined: 23 Apr 2018, 17:55
Location: Germany

Re: [Generic - MQTT Import]: No option available

#26 Post by Wiki » 25 Oct 2022, 12:03

And this is how it looks like running with "only" six MQTT Import tasks:
Stabil1.JPG
Stabil1.JPG (55.6 KiB) Viewed 3347 times
Stabil2.JPG
Stabil2.JPG (54.56 KiB) Viewed 3347 times

Code: Select all

pi@raspberrypi:~ $ man woman
No manual entry for woman
pi@raspberrypi:~ $

TD-er
Core team member
Posts: 8643
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: [Generic - MQTT Import]: No option available

#27 Post by TD-er » 25 Oct 2022, 12:20

Hmm most of the resets were when sending data to the MQTT controller.
Maybe we should add some calls to delay(0) when handling MQTT import calls to make sure network traffic is properly handled and no unnoticed disconnects occur.

Wiki
Normal user
Posts: 413
Joined: 23 Apr 2018, 17:55
Location: Germany

Re: [Generic - MQTT Import]: No option available

#28 Post by Wiki » 26 Oct 2022, 10:03

Should I open an issue on github?

Code: Select all

pi@raspberrypi:~ $ man woman
No manual entry for woman
pi@raspberrypi:~ $

User avatar
Ath
Normal user
Posts: 3417
Joined: 10 Jun 2018, 12:06
Location: NL

Re: [Generic - MQTT Import]: No option available

#29 Post by Ath » 26 Oct 2022, 10:06

Wiki wrote: 26 Oct 2022, 10:03 Should I open an issue on github?
That's a good way to keep attention on it :D
/Ton (PayPal.me)

Wiki
Normal user
Posts: 413
Joined: 23 Apr 2018, 17:55
Location: Germany

Re: [Generic - MQTT Import]: No option available

#30 Post by Wiki » 26 Oct 2022, 16:45

....done, reminder set.

Code: Select all

pi@raspberrypi:~ $ man woman
No manual entry for woman
pi@raspberrypi:~ $

Post Reply

Who is online

Users browsing this forum: Google [Bot] and 34 guests