ESPEasy & uPyEasy Strategy

Announcements from the Staff

Moderators: rtenklooster, Voyager, BertB, Stuntteam

Post Reply
Message
Author
LisaM
Normal user
Posts: 513
Joined: 11 Apr 2017, 18:29

ESPEasy & uPyEasy Strategy

#1 Post by LisaM » 27 Jan 2018, 15:39

The uPyEasy beta-1 release is here, complete with binaries and source code (viewtopic.php?f=20&t=4745).
Question is: What's next?

What is going to be our strategy for the foreseeable future?
- Are we going to keep two different applications where ESPEasy is running on ESP8266/ESP32 and uPyEasy running on everything?
- Are we going to (slowly) migrate from ESPEasy to uPyEasy?
- Is ESPEasy going to run on ESP8266 and uPyEasy on everything else?

There many strategies possible, with each having it's own pro's and con's, but we need to agree on which one we choose...

User avatar
grovkillen
Core team member
Posts: 3621
Joined: 19 Jan 2017, 12:56
Location: Hudiksvall, Sweden
Contact:

Re: ESPEasy & uPyEasy Strategy

#2 Post by grovkillen » 27 Jan 2018, 18:18

First, thank you for all your efforts :D

Regarding the strategy for the future I think we should be focusing on the ESP8266 and ESP8285 for ESP Easy and then let uPy Easy be the FW for the rest, is the best choice.

But I also think we need to think about our name. Easy is the common line here and lets try to make the Easy part stand out a bit more. I suggest we try to make the general FW called "Easy" since our users don't really care or know the architecture of the FW.

Just my thoughts ;)

EDIT: maybe refer to ESP Easy as the "lite" version and uPy Easy as the "extensive" version. Or something like that?

EDIT2: is it just me who in daily talk say "uppy" Easy?
ESP Easy Flasher [flash tool and wifi setup at flash time]
ESP Easy Webdumper [easy screendumping of your units]
ESP Easy Netscan [find units]
Official shop: https://firstbyte.shop/
Sponsor ESP Easy, we need you :idea: :idea: :idea:

Shardan
Normal user
Posts: 1156
Joined: 03 Sep 2016, 23:27
Location: Bielefeld / Germany

Re: ESPEasy & uPyEasy Strategy

#3 Post by Shardan » 28 Jan 2018, 14:08

+1
Regards
Shardan

papperone
Normal user
Posts: 497
Joined: 04 Oct 2016, 23:16

Re: ESPEasy & uPyEasy Strategy

#4 Post by papperone » 28 Jan 2018, 14:44

I agree to keep current firmware ESP8266/8285 only while uPy one shoudl target newer/faster hardware.
If we allow both frimware to be compatible on ESP8266 the issue will be to maintain all plugins for both firmware due to the different languages (C++ vs. yPython).
About naming @grovkillen post triggered me the same idea, the word "EASY" shoudl be the "brand" so I woudl propose EASY-ESP and EASY-uPy (and EASY-whatever in the future).
My TINDIE Store where you can find all ESP8266 boards I manufacture --> https://www.tindie.com/stores/GiovanniCas/
My Wiki Project page with self-made PCB/devices --> https://www.letscontrolit.com/wiki/inde ... :Papperone

TD-er
Core team member
Posts: 8643
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: ESPEasy & uPyEasy Strategy

#5 Post by TD-er » 28 Jan 2018, 20:40

Perhaps looking into the expected features may also help the discussion?

The way I see it, one of the main changes should be in the user interface.
  • JSON REST like api to interface
  • Website which communicates with the REST api.
This will force some more generic way of interfacing and hopefully less fishy bugs and less memory use.
Also it will allow a more central management approach.

And another thing I really like is to export/import the settings using some Json dump.
This allows to upgrade/extend settings among versions and also between the C++/uPy version?
And also makes it easier to distribute settings amongst several nodes, or perhaps share part of settings via a forum.

I think one of the main advantages of the uPy approach is that adding new features is often just a matter of calling some import and start working.
And I think there are more programmers for Python than there are for C++.

What are the minimum requirements for uPyEasy?
Will it run on most Sonoff devices?

Perhaps another idea is to group the functionalities into some categories. Some more suitable for uPy, some for all and some for C++?
e.g. larger displays are not really suitable for ESP8266, due to memory restrictions.

LisaM
Normal user
Posts: 513
Joined: 11 Apr 2017, 18:29

Re: ESPEasy & uPyEasy Strategy

#6 Post by LisaM » 28 Jan 2018, 22:30

TD-er wrote: 28 Jan 2018, 20:40 Perhaps looking into the expected features may also help the discussion?

The way I see it, one of the main changes should be in the user interface.
  • JSON REST like api to interface
  • Website which communicates with the REST api.
This will force some more generic way of interfacing and hopefully less fishy bugs and less memory use.
Also it will allow a more central management approach.

And another thing I really like is to export/import the settings using some Json dump.
This allows to upgrade/extend settings among versions and also between the C++/uPy version?
And also makes it easier to distribute settings amongst several nodes, or perhaps share part of settings via a forum.

I think one of the main advantages of the uPy approach is that adding new features is often just a matter of calling some import and start working.
And I think there are more programmers for Python than there are for C++.

What are the minimum requirements for uPyEasy?
Will it run on most Sonoff devices?

Perhaps another idea is to group the functionalities into some categories. Some more suitable for uPy, some for all and some for C++?
e.g. larger displays are not really suitable for ESP8266, due to memory restrictions.
JSON, REST, SOAP are all M2M communication protocols, useless when dealing with end-users. The question is what app resides on the user side to receive these comm protocols from the server?

uPyEasy settings export is actually a json dump of all tables.

uPyEasy GUI is no longer fixed, but html template based. Easy ;) to change...

I always said uPyEasy would require minimal 1MB of flash to work with, so it would also fit on Sonoff devices. Maybe it will be a trimmed version to fit the 1MB flash, then again who needs dozens of sensors if only a couple are actually needed for an Sonoff? The STM32 version is only 700K, ESP32 is bigger but that's mainly espressif c-based junk floating along.

TD-er
Core team member
Posts: 8643
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: ESPEasy & uPyEasy Strategy

#7 Post by TD-er » 28 Jan 2018, 22:44

I agree the distribution of the whole could be trimmed a little. Also with ESPEasy, but then we should add some option to generate a version with the required plugins.
I think that's quite hard to do.

psy0rz
Normal user
Posts: 232
Joined: 02 Feb 2017, 12:12

Re: ESPEasy & uPyEasy Strategy

#8 Post by psy0rz » 29 Jan 2018, 23:22

LisaM wrote: 28 Jan 2018, 22:30 JSON, REST, SOAP are all M2M communication protocols, useless when dealing with end-users. The question is what app resides on the user side to receive these comm protocols from the server?
TD-er and I where discussing a system like this:

The ESP only talks JSON via a REST interface.

The users "app" is just a webapp in the browser: The browser downloads static html and javascript files which communicate via the REST interface.

The REST api will return metadata and regular data. The metadata will tell the webapp which input fields a plugin needs forexample, and the descriptions we want to show for those fields. The webapp uses this data to generate the correct view. (There nice ways to do this in a template-like manner, all processed client-side)

This has multiple advantages:
[*] Core and plugins have one well defined REST (json) api.
[*] No need for processing html and forms on the ESP. (saves a lot of resources)
[*] Heavy lifting and template processing is done on client side by the webapp.
[*] Static html/css/javascript files could be hosted on the ESP, but also somewhere else.
[*] If its hosted somewhere else, the ESP will just do a http redirect and tell the webapp the IP of the ESP.

But perhaps this is something for the next version of either project.
Please support ESPEasy development via Patreon or buy us a coffee.

psy0rz
Normal user
Posts: 232
Joined: 02 Feb 2017, 12:12

Re: ESPEasy & uPyEasy Strategy

#9 Post by psy0rz » 29 Jan 2018, 23:31

LisaM wrote: 27 Jan 2018, 15:39 The uPyEasy beta-1 release is here, complete with binaries and source code (viewtopic.php?f=20&t=4745).
Question is: What's next?

What is going to be our strategy for the foreseeable future?
- Are we going to keep two different applications where ESPEasy is running on ESP8266/ESP32 and uPyEasy running on everything?
- Are we going to (slowly) migrate from ESPEasy to uPyEasy?
- Is ESPEasy going to run on ESP8266 and uPyEasy on everything else?

There many strategies possible, with each having it's own pro's and con's, but we need to agree on which one we choose...
The only reason to keep ESPEasy for now is the sheer amount of plugins it has. If uPyEasy works well, I would expect a lot of plugins to be ported very soon.

Frankly I dont see the point in spending so much time on something that has so many hard to fix bugs (mainly because its C and because of hardware issues like the PUYA flashchips for example) and runs on limited hardware. I'm affraid its just quicker to port a large portion of commonly used plugins to uPyEasy, instead of trying to fix all the bugs and add all the new features to ESPEasy.
Please support ESPEasy development via Patreon or buy us a coffee.

LisaM
Normal user
Posts: 513
Joined: 11 Apr 2017, 18:29

Re: ESPEasy & uPyEasy Strategy

#10 Post by LisaM » 29 Jan 2018, 23:39

psy0rz wrote: 29 Jan 2018, 23:22
LisaM wrote: 28 Jan 2018, 22:30 JSON, REST, SOAP are all M2M communication protocols, useless when dealing with end-users. The question is what app resides on the user side to receive these comm protocols from the server?
TD-er and I where discussing a system like this:

The ESP only talks JSON via a REST interface.

The users "app" is just a webapp in the browser: The browser downloads static html and javascript files which communicate via the REST interface.

The REST api will return metadata and regular data. The metadata will tell the webapp which input fields a plugin needs forexample, and the descriptions we want to show for those fields. The webapp uses this data to generate the correct view. (There nice ways to do this in a template-like manner, all processed client-side)

This has multiple advantages:
[*] Core and plugins have one well defined REST (json) api.
[*] No need for processing html and forms on the ESP. (saves a lot of resources)
[*] Heavy lifting and template processing is done on client side by the webapp.
[*] Static html/css/javascript files could be hosted on the ESP, but also somewhere else.
[*] If its hosted somewhere else, the ESP will just do a http redirect and tell the webapp the IP of the ESP.

But perhaps this is something for the next version of either project.
This will be only a minor change for uPyEasy, although this might be strange to hear. uPyEasy doesn't process any html, that is all done by the utemplate library. uPyEasy just collects all data necessary for generating the template and then send the data to the utemplate library (which combines the data and the html template to generate a dynamic html page). Wether i send it to the utemplate library or to another is really moot to me.
Almost all parts of uPyEasy are exchangeable components, see also the uPyEasy architecture guidelines... (state of the art ;) )

psy0rz
Normal user
Posts: 232
Joined: 02 Feb 2017, 12:12

Re: ESPEasy & uPyEasy Strategy

#11 Post by psy0rz » 29 Jan 2018, 23:49

LisaM wrote: 29 Jan 2018, 23:39
psy0rz wrote: 29 Jan 2018, 23:22
LisaM wrote: 28 Jan 2018, 22:30 JSON, REST, SOAP are all M2M communication protocols, useless when dealing with end-users. The question is what app resides on the user side to receive these comm protocols from the server?
TD-er and I where discussing a system like this:

The ESP only talks JSON via a REST interface.

The users "app" is just a webapp in the browser: The browser downloads static html and javascript files which communicate via the REST interface.

The REST api will return metadata and regular data. The metadata will tell the webapp which input fields a plugin needs forexample, and the descriptions we want to show for those fields. The webapp uses this data to generate the correct view. (There nice ways to do this in a template-like manner, all processed client-side)

This has multiple advantages:
[*] Core and plugins have one well defined REST (json) api.
[*] No need for processing html and forms on the ESP. (saves a lot of resources)
[*] Heavy lifting and template processing is done on client side by the webapp.
[*] Static html/css/javascript files could be hosted on the ESP, but also somewhere else.
[*] If its hosted somewhere else, the ESP will just do a http redirect and tell the webapp the IP of the ESP.

But perhaps this is something for the next version of either project.
This will be only a minor change for uPyEasy, although this might be strange to hear. uPyEasy doesn't process any html, that is all done by the utemplate library. uPyEasy just collects all data necessary for generating the template and then send the data to the utemplate library (which combines the data and the html template to generate a dynamic html page). Wether i send it to the utemplate library or to another is really moot to me.
Almost all parts of uPyEasy are exchangeable components, see also the uPyEasy architecture guidelines... (state of the art ;) )
Very nice. :)
Please support ESPEasy development via Patreon or buy us a coffee.

psy0rz
Normal user
Posts: 232
Joined: 02 Feb 2017, 12:12

Re: ESPEasy & uPyEasy Strategy

#12 Post by psy0rz » 30 Jan 2018, 00:03

For now I think its important we get both projects in a stable state. We need to release a stable ESPEasy v2.0 and a initial stable uPyEasy. After that I would expect a slow migration from ESPEasy to uPyEasy, as uPyEasy starts to mature and will gain more plugins.

uPyEasy seems to be designed very well, and since developing in python is easier, much more bugfree, and allows for dynamic plugin loading. Because of this I would expect an exponential growth the coming year, after a stable release and good documentation. Lisa I assume you're also planning on a plugin repo and an "plugin shop", so that we can just search for a plugin in the webinterace and just click Install?
Please support ESPEasy development via Patreon or buy us a coffee.

LisaM
Normal user
Posts: 513
Joined: 11 Apr 2017, 18:29

Re: ESPEasy & uPyEasy Strategy

#13 Post by LisaM » 30 Jan 2018, 00:27

psy0rz wrote: 30 Jan 2018, 00:03 For now I think its important we get both projects in a stable state. We need to release a stable ESPEasy v2.0 and a initial stable uPyEasy. After that I would expect a slow migration from ESPEasy to uPyEasy, as uPyEasy starts to mature and will gain more plugins.

uPyEasy seems to be designed very well, and since developing in python is easier, much more bugfree, and allows for dynamic plugin loading. Because of this I would expect an exponential growth the coming year, after a stable release and good documentation. Lisa I assume you're also planning on a plugin repo and an "plugin shop", so that we can just search for a plugin in the webinterace and just click Install?
Yes, plugin repo and dynamic install and run. A limited number of 'frozen' core plugins, let's say the top 10 most choosen plugins, and the rest is just dynamically selectable from a repository, downloaded and immediately run. How much plugins you can install depends on the flash size, the more memory the flash drive has the more plugins you can download.
Generic parts of the plugins are small, since most of the code is already in the utils module. Only sensor/function specific code is still in the plugin itself.
Currently the frozen plugins are already dynamically loaded, they are not loaded until you actually create a device that uses this plugin to conserve memory. Loading it dynamically from firmware or flash drive is just a small change.

The hardest part is a number of design decision's:
- Where is the plugin repo located?
- How to select the right plugin? (we need a universal data exchange for that, meaning a metadata library containing information about the plugins itself)

For now i'm busy writing the uPyEasy Plugin Developers Guide, it explains in detail how build an uPyEasy plugin in micropython. After that the uPyEasy Protocol Developers Guide for developing more protocol's (like Andrew is currently doing for Openhab MQTT) is next, closing the series with the uPyEasy System Developers Guide explaining how uPyEasy is functioning and how to add/change it's functionality.

Having http-browser based client-server GUI is somewhere on the backburner, i have some thoughts about that but the data specs must be crystal clear. Going client-server also means an opportunity to do a radical GUI makeover by an UX designer.

TD-er
Core team member
Posts: 8643
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: ESPEasy & uPyEasy Strategy

#14 Post by TD-er » 30 Jan 2018, 01:34

LisaM wrote: 30 Jan 2018, 00:27[...] also means an opportunity to do a radical GUI makeover by an UX designer.
Now that's what I would call a very welcome new feature ;)

User avatar
grovkillen
Core team member
Posts: 3621
Joined: 19 Jan 2017, 12:56
Location: Hudiksvall, Sweden
Contact:

Re: ESPEasy & uPyEasy Strategy

#15 Post by grovkillen » 30 Jan 2018, 06:26

Sounds really awesome Lisa :o 8-)
ESP Easy Flasher [flash tool and wifi setup at flash time]
ESP Easy Webdumper [easy screendumping of your units]
ESP Easy Netscan [find units]
Official shop: https://firstbyte.shop/
Sponsor ESP Easy, we need you :idea: :idea: :idea:

psy0rz
Normal user
Posts: 232
Joined: 02 Feb 2017, 12:12

Re: ESPEasy & uPyEasy Strategy

#16 Post by psy0rz » 06 Feb 2018, 00:18

For my led-animation project i'm just using github as a repository. Only need to regenerate the repository index if plugins are added, but we could do that with Travis.

When the repo is on github we could also make branches/tags and let the user choose if they want to browser "stable/testing/developement" plugins for example.

My repo is here: https://github.com/psy0rz/ledanim/tree/master/web/repo (Github wants us to use other urls to directly acces it, like: https://raw.githubusercontent.com/psy0r ... index.json)

Demo of the this webapplication consisting of static files: http://rawgit.com/psy0rz/ledanim/master ... 168.13.128#

This:
* Downloads and shows the repo
* Lets the user to click an animation. (animations are javascript)
* Generates a special bytecode from the animation and sends it to the ESP8266, which in turn runs the animation in a loop. (specified with host=192.168.13.128)
* The webapp is also simulating the animation in the browser. To do this, I cross compiled the C++ code that runs on the ESP8266 to javascript via Emscripten. This way there is one animation library that behaves exactly the same in both the browser and the ESP8266.

Pretty neat eh? :)

I agree that you should put the webapp on the backburner. Getting a working version + repo out has higher prio.

Keep up the good work,

Edwin
Please support ESPEasy development via Patreon or buy us a coffee.

LisaM
Normal user
Posts: 513
Joined: 11 Apr 2017, 18:29

Re: ESPEasy & uPyEasy Strategy

#17 Post by LisaM » 06 May 2018, 01:18

psy0rz wrote: 06 Feb 2018, 00:18 For my led-animation project i'm just using github as a repository. Only need to regenerate the repository index if plugins are added, but we could do that with Travis.

When the repo is on github we could also make branches/tags and let the user choose if they want to browser "stable/testing/developement" plugins for example.

My repo is here: https://github.com/psy0rz/ledanim/tree/master/web/repo (Github wants us to use other urls to directly acces it, like: https://raw.githubusercontent.com/psy0r ... index.json)

Demo of the this webapplication consisting of static files: http://rawgit.com/psy0rz/ledanim/master ... 168.13.128#

This:
* Downloads and shows the repo
* Lets the user to click an animation. (animations are javascript)
* Generates a special bytecode from the animation and sends it to the ESP8266, which in turn runs the animation in a loop. (specified with host=192.168.13.128)
* The webapp is also simulating the animation in the browser. To do this, I cross compiled the C++ code that runs on the ESP8266 to javascript via Emscripten. This way there is one animation library that behaves exactly the same in both the browser and the ESP8266.

Pretty neat eh? :)

I agree that you should put the webapp on the backburner. Getting a working version + repo out has higher prio.

Keep up the good work,

Edwin
With uPyEasy v0.3.0 i have taken the first step towards this new architecture!

The problem with the ESP32 is that is has 28(!) usable GPIO's. Since uPyEasy is increasingly supporting more and more protocols, the hardware page was getting slower and slower (more standard protocols to display including pins) to the point where it took nearly 20~30 seconds to display the page. The problem was that each pulldown menu was generated and transmitted, with already 9 pulldown menu's on one page it took quite a while and with the pin boot states (28 ! pin's) it was also taking a long time.

I've have changed this, now i only generate a single pulldown menu and ALL other pulldown menu are copied from the first using Javascript (so basically on the client!). It's a primitive form of client-server communication, i'm thinking about going all the way with this new architecture and abandon all GUI on the server side.
This would mean a GUI on the client side (downloaded from the cloud) and a server sitting on the ESP32/STM32 communicating with each other. Due to the component based setup of uPyEasy it means that the GUI is making changes in the uPyEasy Server database.
So far no uPyEasy architecture changes have been necessary, due to the use of html templates. JSON Rest support is standard present in uPyEasy, so no changes there either. The question is which client web app to use...

TD-er
Core team member
Posts: 8643
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: ESPEasy & uPyEasy Strategy

#18 Post by TD-er » 06 May 2018, 21:46

ESPeasy itself is also somewhat moving into that same direction, al be it slowly.
In C++ land we have to code it all ourselves.
So maybe we could discuss together what kind of queries to the host are needed.

Yesterday I made some changes to the Json pages in ESPeasy.
One for the log (/logjson) and the existing one (/json)
Generating these only takes now about 40 msec to deliver to the client and about 1k of RAM.
The log may take a bit longer if it is outputting the full log buffer.

The first steps for ESPeasy will be that the client can request sensor data (/json?tasknr=1 for example, or all sensors with tasknr = 0) to do some live updates in the web page.
And the log will also be stored inside the browser.
Both JSON pages have some TTL value to hint the client when to try and refresh.

The next step will be loading and storing of settings and I guess it makes sense to have both platforms output the same kind of information in the same way.
So I guess some exchange of knowledge and insights may be in place here :)

LisaM
Normal user
Posts: 513
Joined: 11 Apr 2017, 18:29

Re: ESPEasy & uPyEasy Strategy

#19 Post by LisaM » 07 May 2018, 14:49

TD-er wrote: 06 May 2018, 21:46 ESPeasy itself is also somewhat moving into that same direction, al be it slowly.
In C++ land we have to code it all ourselves.
So maybe we could discuss together what kind of queries to the host are needed.

Yesterday I made some changes to the Json pages in ESPeasy.
One for the log (/logjson) and the existing one (/json)
Generating these only takes now about 40 msec to deliver to the client and about 1k of RAM.
The log may take a bit longer if it is outputting the full log buffer.

The first steps for ESPeasy will be that the client can request sensor data (/json?tasknr=1 for example, or all sensors with tasknr = 0) to do some live updates in the web page.
And the log will also be stored inside the browser.
Both JSON pages have some TTL value to hint the client when to try and refresh.

The next step will be loading and storing of settings and I guess it makes sense to have both platforms output the same kind of information in the same way.
So I guess some exchange of knowledge and insights may be in place here :)
For uPyEasy there is no need to do any JSON REST API encoding/decoding since everything is done by (another) (micro)python library. The input/output of these libraries (python dictionaries) match the input/output of the database (also python dictionaries), so it works together seamless.So little coding effort on my side.

I'll go ahead and make a proposal for a JSON REST API standard, based on the information i have in the uPyEasy database.

The reason for all of this is very simple: i need it. I want to have a dashboard displaying current values from uPyEasy, primarily on mobile devices.
Something like this:
dashboard.png
dashboard.png (107.63 KiB) Viewed 37823 times
The supernodes, the most powerfull nodes in a nodes mesh, will have this. The more simple ones probably not.

TD-er
Core team member
Posts: 8643
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: ESPEasy & uPyEasy Strategy

#20 Post by TD-er » 07 May 2018, 19:10

LisaM wrote: 07 May 2018, 14:49 [...] For uPyEasy there is no need to do any JSON REST API encoding/decoding since everything is done by (another) (micro)python library. The input/output of these libraries (python dictionaries) match the input/output of the database (also python dictionaries), so it works together seamless.So little coding effort on my side.

I'll go ahead and make a proposal for a JSON REST API standard, based on the information i have in the uPyEasy database.

The reason for all of this is very simple: i need it. I want to have a dashboard displaying current values from uPyEasy, primarily on mobile devices.
Something like this:
[...]
The supernodes, the most powerfull nodes in a nodes mesh, will have this. The more simple ones probably not.
Great, so we already agree ;)
I know creating an API like this is very easy in Python. (I do also use it at work)
And I know you already have a well thought of data-structure.
So it would be great to try to match both API-like interfaces to be able to use then together in the same ESPeasy grid/mesh, where possible.

It is not my intention to let the current datastructs of ESPeasy (C++) be the limiting factors for an API.

It would be great if ESPeasy C++ and Python could share some functionality like using recorded values of one node in rules on the other, or sync some actions and be integrated in the same web-interface to create a combined dashboard.

LisaM
Normal user
Posts: 513
Joined: 11 Apr 2017, 18:29

Re: ESPEasy & uPyEasy Strategy

#21 Post by LisaM » 09 May 2018, 01:39

TD-er wrote: 07 May 2018, 19:10
LisaM wrote: 07 May 2018, 14:49 [...] For uPyEasy there is no need to do any JSON REST API encoding/decoding since everything is done by (another) (micro)python library. The input/output of these libraries (python dictionaries) match the input/output of the database (also python dictionaries), so it works together seamless.So little coding effort on my side.

I'll go ahead and make a proposal for a JSON REST API standard, based on the information i have in the uPyEasy database.

The reason for all of this is very simple: i need it. I want to have a dashboard displaying current values from uPyEasy, primarily on mobile devices.
Something like this:
[...]
The supernodes, the most powerfull nodes in a nodes mesh, will have this. The more simple ones probably not.
Great, so we already agree ;)
I know creating an API like this is very easy in Python. (I do also use it at work)
And I know you already have a well thought of data-structure.
So it would be great to try to match both API-like interfaces to be able to use then together in the same ESPeasy grid/mesh, where possible.

It is not my intention to let the current datastructs of ESPeasy (C++) be the limiting factors for an API.

It would be great if ESPeasy C++ and Python could share some functionality like using recorded values of one node in rules on the other, or sync some actions and be integrated in the same web-interface to create a combined dashboard.
Done:
url: "/api/v1.0/controller/" + id,
type: 'DELETE',
And...working! :lol:
A few lines of code in the webserver framework was enough, searching how this supposed to work was the most time consuming.

I'll make an uPyEasy/ESPEasy RESTful interface proposal...

And start prototyping with a bootstrap (offline) dashboard retrieving current device values from uPyEasy.

Cheers,

Lisa

TD-er
Core team member
Posts: 8643
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: ESPEasy & uPyEasy Strategy

#22 Post by TD-er » 09 May 2018, 22:13

ESPeasy C++ already has some JSON retrieval for the values. (sensor data also addressable per task)
Maybe good for inspiraction?

Code: Select all

{  
  "System":{  
    "Build":20102,
    "Git Build":"",
    "System libraries":"ESP82xx Core 2_4_1, NONOS SDK 2.2.1(cfd48f3), LWIP: 2.0.3",
    "Plugins":47,
    "Plugin description":" [Normal]",
    "Local time":"2018-05-09 22:07:00",
    "Unit":0,
    "Name":"ESP_Easy",
    "Uptime":965,
    "Last boot cause":"Manual reboot",
    "Load":28,
    "Load LC":12236,
    "Free RAM":13816
  },
  "WiFi":{  
    "Hostname":"ESP-Easy-0",
    "IP config":"DHCP",
    "IP":"192.168.1.75",
    "Subnet Mask":"255.255.255.0",
    "Gateway IP":"192.168.1.1",
    "MAC address":"A0:20:A6:19:2A:BF",
    "DNS 1":"192.168.1.1",
    "DNS 2":"0.0.0.0",
    "SSID":"Lurch4",
    "BSSID":"34:31:C4:B1:8D:C7",
    "Channel":11,
    "Connected msec":21684891,
    "Last Disconnect Reason":201,
    "Last Disconnect Reason str":"(201) No AP found",
    "Number reconnects":2,
    "RSSI":-73
  },
  "Sensors":[  
    {  
      "TaskNumber":1,
      "TaskEnabled":"true",
      "TaskInterval":5,
      "Type":"Display - OLED SSD1306/SH1106 Framed",
      "TaskName":"oled"
    },
    {  
      "TaskValues":[  
        {  
          "ValueNumber":1,
          "Name":"Temperature",
          "Value":24.61
        },
        {  
          "ValueNumber":2,
          "Name":"Humidity",
          "Value":50.27
        },
        {  
          "ValueNumber":3,
          "Name":"Pressure",
          "Value":1006.29
        }
      ],
      "TaskNumber":2,
      "TaskEnabled":"true",
      "TaskInterval":60,
      "Type":"Environment - BMx280",
      "TaskName":"bme"
    }
  ],
  "TTL":60000
}
The TTL I added at the highest level is to suggest a reload time in msec.

User avatar
grovkillen
Core team member
Posts: 3621
Joined: 19 Jan 2017, 12:56
Location: Hudiksvall, Sweden
Contact:

Re: ESPEasy & uPyEasy Strategy

#23 Post by grovkillen » 09 May 2018, 22:58

This approach works really good for my JavaScripts. 8-)

JSON is so nice to parse. Love it!
ESP Easy Flasher [flash tool and wifi setup at flash time]
ESP Easy Webdumper [easy screendumping of your units]
ESP Easy Netscan [find units]
Official shop: https://firstbyte.shop/
Sponsor ESP Easy, we need you :idea: :idea: :idea:

LisaM
Normal user
Posts: 513
Joined: 11 Apr 2017, 18:29

Re: ESPEasy & uPyEasy Strategy

#24 Post by LisaM » 10 May 2018, 18:04

TD-er wrote: 09 May 2018, 22:13 ESPeasy C++ already has some JSON retrieval for the values. (sensor data also addressable per task)
Maybe good for inspiraction?

Code: Select all

{  
  "System":{  
    "Build":20102,
    "Git Build":"",
    "System libraries":"ESP82xx Core 2_4_1, NONOS SDK 2.2.1(cfd48f3), LWIP: 2.0.3",
    "Plugins":47,
    "Plugin description":" [Normal]",
    "Local time":"2018-05-09 22:07:00",
    "Unit":0,
    "Name":"ESP_Easy",
    "Uptime":965,
    "Last boot cause":"Manual reboot",
    "Load":28,
    "Load LC":12236,
    "Free RAM":13816
  },
  "WiFi":{  
    "Hostname":"ESP-Easy-0",
    "IP config":"DHCP",
    "IP":"192.168.1.75",
    "Subnet Mask":"255.255.255.0",
    "Gateway IP":"192.168.1.1",
    "MAC address":"A0:20:A6:19:2A:BF",
    "DNS 1":"192.168.1.1",
    "DNS 2":"0.0.0.0",
    "SSID":"Lurch4",
    "BSSID":"34:31:C4:B1:8D:C7",
    "Channel":11,
    "Connected msec":21684891,
    "Last Disconnect Reason":201,
    "Last Disconnect Reason str":"(201) No AP found",
    "Number reconnects":2,
    "RSSI":-73
  },
  "Sensors":[  
    {  
      "TaskNumber":1,
      "TaskEnabled":"true",
      "TaskInterval":5,
      "Type":"Display - OLED SSD1306/SH1106 Framed",
      "TaskName":"oled"
    },
    {  
      "TaskValues":[  
        {  
          "ValueNumber":1,
          "Name":"Temperature",
          "Value":24.61
        },
        {  
          "ValueNumber":2,
          "Name":"Humidity",
          "Value":50.27
        },
        {  
          "ValueNumber":3,
          "Name":"Pressure",
          "Value":1006.29
        }
      ],
      "TaskNumber":2,
      "TaskEnabled":"true",
      "TaskInterval":60,
      "Type":"Environment - BMx280",
      "TaskName":"bme"
    }
  ],
  "TTL":60000
}
The TTL I added at the highest level is to suggest a reload time in msec.
To big, i'll separate the parts into chunks like sensors and so on.
Changed security to Basic Http Authentication on uPyEasy so that the REST api can also use that.
The device and controller DELETE Rest API is working as POC, the remainder will be easy.

uPyEasy Dashboard REST client basics are set:
- Presentation and styling: Twitter Bootstrap
- REST request management: jQuery
- Templates and event handling: Knockout

These are all delivered through the CDN network. There's no need to have all the webpages sending from uPyEasy, so it won't. uPyEasy will only send the json data through the upyEasy REST api.

TD-er
Core team member
Posts: 8643
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: ESPEasy & uPyEasy Strategy

#25 Post by TD-er » 10 May 2018, 18:29

I didn't know the "Twitter Bootstrap" yet, so I will read into it.

The reason I made the JSON loadable as a whole, and also per task is to make the load move to the browser, not the ESP.
A single query takes 30 - 70 msec and if you query all tasks at once, you can get complete the request also in 30 - 70 msec.
On the ESP8266 it simply takes that long to process. But when streaming has started, it doesn't take much time to deliver all at once and I'm generating the stream while it is being transferred.

How do you plan on knowing when new values are there to process?
I've looked into keeping a connection open, but that may give some issues, since the maximum number of open connections is quite limited.

And I don't know much about other existing push technologies.
So that's why I added the TTL attribute to the JSON, to hint when a new fetch would be in order.

Edit:
Isn't "Twitter Bootstrap" just "Bootstrap" ?
Or am I missing something here?

LisaM
Normal user
Posts: 513
Joined: 11 Apr 2017, 18:29

Re: ESPEasy & uPyEasy Strategy

#26 Post by LisaM » 10 May 2018, 18:44

TD-er wrote: 10 May 2018, 18:29 Isn't "Twitter Bootstrap" just "Bootstrap" ?
Twitter created it, open sourced it and named it Twitter Bootstrap. It's becoming more and more just Bootstrap...
The reload time depends on the sensor delay time, if they are all 60s it doesn't make sense to reload every 6 s. So essentially the shortest delay time is determining the reload time.

TD-er
Core team member
Posts: 8643
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: ESPEasy & uPyEasy Strategy

#27 Post by TD-er » 10 May 2018, 20:58

About the reload time, that's indeed what I output as TTL.value.
For the log view, the TTL is determined on the amount of entries read from the log buffer.

Post Reply

Who is online

Users browsing this forum: No registered users and 14 guests