Setting up an MQTT system for Weather Sensing and Astro Control

[Gina]

This is showing lots of promise 👍  Lots to learn though but making progress.

This shows ROR control.  3 buttons.

And this is a rough Dashboard.  I shall separate it into groups or tabs.  The ROR control will be on a different ESP32 with different name.  There will also be roof status indicators.

Edited August 15, 2020 by Gina

[Padraic M]

12 hours ago, Gina said:

Does anyone here use Node-RED and how useful do you find it?

Yes Gina, I've been using node red to control my sensor network for years. I find it extremely useful and very reliable. I use it in two ways - on an Amazon web services linux server as the core of the sensor network, and also on Raspberry PIs when I need to solve any local problems like interfacing from a Flukso power meter to the cloud, or consuming publicly-available open-data datasets etc. Quick and easy to set up, and very easy to create the flows.  Node red on my AWS linux box has no problem managing 10's of mqtt messages per second, running 24x7 for weeks at a time. CPU and memory usage on an AWS t2.micro box is minimal.

I don't use the dashboard - found it fiddly, and never figured out a way to embed it in my own website (also actually hosted from within node red).

The central set-up uses mysql for configuration data, and influxdb for time series sensor data. Node red talks to both databases. Grafana is the dashboard front-end. Communications from sensors and access points to the server uses mosquitto mqtt which is very straightforward, both within node red, and as a stand-alone linux mqtt broker in AWS. Node red can create mosquitto topics on the fly, just by specifying the 'topic' property on input to a mqtt output node. The topic doesn't have to pre-exist.

Nginx proxy server sits in front of everything, and controls access to the node red admin gui, the custom website, mosquitto, and granfana. The databases are firewalled and not accessible from the public network.

Sample screenshot from the website, which uses the Bootstrap framework:
 

Sample grafana screenshot:

I should add that a lot of work has gone into this.... and more is required. My brother in law uses esp8266s with openhab to get a similar result, but I haven't gone down that route. My sensors are a mixture of custom-designed boards with TI msp430 mpu, nrf24l01 radio transceiver, esp8266, and Dragino HE. the msp430 is an ultra low-power arduino equivalent; I don't use arduino as I'm proficient in C and am interested in embedded C programming so I wouldn't get any benefit from the arduino IDE. Dragino is an openwrt linux module - I'm a big fan, having first gotten to grips with openwrt on the TP-Link wr703n router and the Dragino is a big step forward from that. I see raspberry PIs as overkill for what I do - it's really a small form-factor desktop rather than an embedded platform. However, I do have a bunch of them lying around for various uses including a cross-compile platform for the esp8266. I'm gradually moving towards having all sensors based on esp8266 - it's such a versatile chip, although philosophically I have an issue with simple sensors having a complete TCP/IP stack built in, and using up to 200mA in active mode (the msp430 /nrf24 combo uses 20mA max).

I use oshpark.com for PCB printing (very cheap and good) and assemble myself, or for more difficult work, macrofab.com prints and assembles small batch boards (very good but quite expensive). I only use them if I'm using components that are too small to hand-solder (like a bmp280!) and if I've gone past the break-out board prototyping stage.

HTH!

Padraic.

[Gina]

Conclusions so far :-

1. MQTT seems very good
2. Node-RED also seems well worthwhile
3. Node-RED Dashboard should work for control but has a crude display and too few "widgets" to display data

Edited August 16, 2020 by Gina

[stash_old]

3 hours ago, Gina said:

Node-RED Dashboard should work for control but has a crude display and too few "widgets" to display data

True statement but make sure you look around for the Widgets to display data - plenty on Github and there are other "dashboards" or write your own!

For me and my Astro stuff I want functionality/reliabilitty/setup speed/ease/no Internet dependence -  not looks so the std Dashboard is fine for me - for a super all singing "dashboard"/system I would use something like OpenHab.

For std sensors/Motor control I dont even bother with Json formatting - no point IMO especially when doing something like Moonlite focuser control protocol - if you are clever you can use the MQTT Tipic to get your message routed to the correct piece of Arduino Code (or any other code) without having to decode - e.g. topics of \moonlite\movein or \moonlite\moveout can be used to "perform" a piece of code specifically to do that function - if you see what I mean. Anyways thats my way, everyone has there own opinion.

So by tomorrow I expect to see a fully working system - LOL

 

[Gina]

I plan to go with the Node-RED Dashboard for the time being - "pretty" can come later if I feel like bothering.  One thing I would like in a dashboard is a wind rose or a 360 degree pointer to show wind direction.  I do have plans for a mechanical wall display for weather conditions though to build eventually.  Showing wind direction in compass points will do for a start - ATM I have nothing for wind display and as they say "anything is better than nothing" 😁

Edited August 16, 2020 by Gina

[discardedastro]

MQTT's great for sensor data - but I'm not sure I would use it for control data. It doesn't have very good mechanics for handling failures; well, any, really.

You can guarantee delivery to a broker but you can't guarantee the broker delivers it. This means you either get to implement some timeout-based RPC mechanics - i.e. wait for a reply - at which point you're often better off with a synchronous system like HTTP etc - or just assume that your message got there. In which case things like "move focuser in" *wait* "stop moving focuser" can go badly wrong, e.g. what if the stop command didn't arrive? You also get no feedback from MQTT itself about message delivery, and MQTT won't retry to send. Things like RabbitMQ/AMQP or Kafka work better in this regard in allowing consumers to retry message handling or receipt in guaranteed-sequence semantics, even in the face of network failures and potentially hardware errors like reboots mid-command.

Grafana is a fine thing for graphing; I use a system based on InfluxDB, MQTT, and some small Python scripts that take MQTT/HTTP data and feed it into InfluxDB. Grafana then pulls data from Influx for display. That works pretty nicely - though I'd consider TimescaleDB next time around.

JSON is a great thing for messaging but the size overhead is pretty nasty. I've been using CBOR for a few things here and there as an alternative, which works really well and is very compact, but haven't tried it on embedded platforms yet.

Edited August 16, 2020 by discardedastro

[Gina]

QoS 2 should virtually guarantee that a message gets through (providing everything is working, or course) though nothing is infallible.  Where i'm relying on control to keep expensive equipment safe (such as ROR) I shall keep control local as far as possible.  eg. with the ROR control, the rain detector will connect directly to the ESP32 controlling the motor - not rely on MQTT.

 

Edited August 16, 2020 by Gina

[Gina]

Well I said I had a lot to learn!  Just found that the gauge widget already there has a "compass" variation.

[Gina]

All this is rather premature in one way - I haven't finished the hardware yet!  OTOH I like to see that there's a way through to the end of a project, if possible, though there are times when I make it up as I go along!

Edited August 16, 2020 by Gina

[jiberjaber]

19 minutes ago, Gina said:

Well I said I had a lot to learn!  Just found that the gauge widget already there has a "compass" variation.

Yep - that's how I did mine initially - there's a wind rose for Graffina but it's a little bit flaky to set up (ok once running though.)

 

[Gina]

I'm gradually getting to grips with Node-RED and the Dashboard but so far I haven't found a way of setting the order or position of widgets in the dashboard.  This is just testing ATM.

Edited August 16, 2020 by Gina

[Gina]

Gradually adding bits and bobs...  Think I'm nearly ready to set up the outside and observatory sensor ESP32 unit for use with MQTT.

I've set the Pressure to show up to 3 hours of data, the Temperature covers the last hour - might change that for a longer period.

[Gina]

I have the outside/obsy sensor unit working with MQTT.  Just one little problem, looks like the DHT22 has come unplugged.  ATM it's just connected with jumper wires.  Think I'll bring it indoors and connect it properly!

Edited August 16, 2020 by Gina

[stash_old]

4 hours ago, discardedastro said:

It doesn't have very good mechanics for handling failures; well, any, really.

That must be why Nasa/IBM are suggesting it to be used across the world wide telescope control system 🙂 - they even suggest using ASCOM across MQTT . Its TCP that carries the error correction at a physical level and its up to your error procotol that is carried by MQTT to control error control. I used it for Moonlite Focuser control via Nano or ESP and it fine.  MQTT is a fast one to many and many to many message distribution protocol.

Dont know if its the same Wind Vane Widget but https://flows.nodered.org/flow/b9a57175ac5a5e240c9916bcce136dca

[Gina]

Interesting link but how do you use it - no instructions, just a huge wadge of code.

[Gina]

Problem - the indoor ESP32 unit has stopped connecting to MQTT.

[Gina]

I now have it plugged into USB on my Mint box and running the Serial Monitor.  It connects briefly every so often.  Anyone any idea of what might have gone wrong?  The test LED doesn't come on even with the Dashboard switch on.  It sends data though.

Edited August 16, 2020 by Gina

[Gina]

Time for a cuppa...

[Gina]

The indoor ESP32 is working properly again now.  Wonder if there was some problem being caused by the obsy unit.  Hope it doesn't mean the RPi Zero W with broker, Node-RED and Dashboard wasn't being overloaded!!  OTOH the outside unit was working fine.

[Padraic M]

You will need to handle mqtt disconnect messages and automatically reconnect. Not saying that this is you current problem, but....

I find that wireless is always the most unstable link in the chain, whether Wi-Fi or other. I agree with local control for critical control applications. Detect and resend should be fine for non-critical control applications, and best effortsmay work for most sensor data.

[Gina]

Quote

You will need to handle mqtt disconnect messages and automatically reconnect.

Yes, I do that.

[discardedastro]

3 hours ago, stash_old said:

That must be why Nasa/IBM are suggesting it to be used across the world wide telescope control system 🙂 - they even suggest using ASCOM across MQTT . Its TCP that carries the error correction at a physical level and its up to your error procotol that is carried by MQTT to control error control. I used it for Moonlite Focuser control via Nano or ESP and it fine.  MQTT is a fast one to many and many to many message distribution protocol.

Dont know if its the same Wind Vane Widget but https://flows.nodered.org/flow/b9a57175ac5a5e240c9916bcce136dca

I can't find anything on the NASA/IBM story but IBM are superb at press releases and have been strong promotors of MQTT for a long while.

Edit: I did actually find it - IBM were promoting a "Cognitive Telescope" project which appears to have turned into a few CS undergrad projects and went no further. MQTT was part of the picture largely to align with IBM's cloud telemetry portfolio. It's all vanished off the internet now as far as I can tell - just a lot of dead links that don't show up in the internet archive etc.

But MQTT is a perfectly good message broker which is suitable for control applications if and only if you put an application layer on top of it to handle the fact that it employs UDP-style semantics by default, if you want to talk network protocols. If a TCP packet does not arrive at the final destination, the sender is aware and can take action. If a MQTT frame is sent to a destination but not acknowledged, it is considered delivered. QOS 1 in MQTT only guarantees delivery semantics between broker and client, not between two clients, and does not guarantee delivery if your client suffers from byzantine failure modes (e.g. received it, parsed the frame and then blew up - this is considered received - compare to HTTP RESTful APIs where a bidirectional exchange between the two final endpoints occurs to validate proper receipt and processing; even QOS 2 in MQTT doesn't guarantee application delivery).

If NASA want to use it for control applications with an application that handles all this atop the basic protocol then that's all jolly good but it is not something you get for free with MQTT - you have to think about it as a developer - and therefore where I'd generally advise against using it as a building block for applications where you need that - because many other equally-accessible things can do that "for free" 😊 MQTT is superb for fire-and-forget telemetry, which is what it was designed for.

It's worth noting MQTT isn't alone in having some of these problems and error management as part of your control protocol is essential regardless of transport - MQTT (as well as similar messaging/fanout brokers in common modes of operation) just makes you handle more of this yourself than you get out of the box with other approaches.

ASCOM over MQTT is a protocol dumpster fire waiting to happen. With my network/systems engineering hat on, I've seen "oh just bung protocol X into protocol Y as a transport/encapsulation to modernise it" go horribly, horribly wrong plenty of times before, even with active protocol conversion going on. Lots of assumptions made by applications and even protocol semantics can break in incredibly hard to debug/fix ways at the application layer. ASCOM Alpaca is a better way to do ASCOM-over-IP since they can do protocol-aware translation at the network boundary, though ASCOM, being a basically "closed" ecosystem, isn't the way to go for this really imho.

The whole ecosystem for remote instrument control is a mess, but stuff like INDI (which is network-native and so has an understanding of these sorts of client/server-at-a-distance protocol considerations baked in, as well as being properly open-source and open-community) is the way to go. I'd stick to HTTP RESTful interfaces where required for control or ensure you're doing application level confirmation of commands in an RPC-style manner to ensure that the remote application both received your message and is able to act on it.

Doing local fallback for safety is a sensible approach as Gina is already doing.

Edited August 16, 2020 by discardedastro

[Gina]

Nearly all my MQTT system deals with just useful information and you can see when it doesn't arrive.  The only control I'm using it for is opening and closing the obsy roof and I have status feedback to check the command I've sent has worked.  Also, I can see the obsy from my living room windows.  (With lantern at night.)  The status will return various messages depending on the state of the roof.

• Opening
• Open
• Closing
• Closed
• Stopped
• Fault

All these fit in my standard 8 character message strings.

Edited August 16, 2020 by Gina

[stash_old]

13 hours ago, discardedastro said:

I can't find anything on the NASA/IBM story but IBM are superb at press releases and have been strong promotors of MQTT for a long while.

Edit: I did actually find it - IBM were promoting a "Cognitive Telescope" project which appears to have turned into a few CS undergrad projects and went no further. MQTT was part of the picture largely to align with IBM's cloud telemetry portfolio. It's all vanished off the internet now as far as I can tell - just a lot of dead links that don't show up in the internet archive etc.

But MQTT is a perfectly good message broker which is suitable for control applications if and only if you put an application layer on top of it to handle the fact that it employs UDP-style semantics by default, if you want to talk network protocols. If a TCP packet does not arrive at the final destination, the sender is aware and can take action. If a MQTT frame is sent to a destination but not acknowledged, it is considered delivered. QOS 1 in MQTT only guarantees delivery semantics between broker and client, not between two clients, and does not guarantee delivery if your client suffers from byzantine failure modes (e.g. received it, parsed the frame and then blew up - this is considered received - compare to HTTP RESTful APIs where a bidirectional exchange between the two final endpoints occurs to validate proper receipt and processing; even QOS 2 in MQTT doesn't guarantee application delivery).

If NASA want to use it for control applications with an application that handles all this atop the basic protocol then that's all jolly good but it is not something you get for free with MQTT - you have to think about it as a developer - and therefore where I'd generally advise against using it as a building block for applications where you need that - because many other equally-accessible things can do that "for free" 😊 MQTT is superb for fire-and-forget telemetry, which is what it was designed for.

It's worth noting MQTT isn't alone in having some of these problems and error management as part of your control protocol is essential regardless of transport - MQTT (as well as similar messaging/fanout brokers in common modes of operation) just makes you handle more of this yourself than you get out of the box with other approaches.

ASCOM over MQTT is a protocol dumpster fire waiting to happen. With my network/systems engineering hat on, I've seen "oh just bung protocol X into protocol Y as a transport/encapsulation to modernise it" go horribly, horribly wrong plenty of times before, even with active protocol conversion going on. Lots of assumptions made by applications and even protocol semantics can break in incredibly hard to debug/fix ways at the application layer. ASCOM Alpaca is a better way to do ASCOM-over-IP since they can do protocol-aware translation at the network boundary, though ASCOM, being a basically "closed" ecosystem, isn't the way to go for this really imho.

The whole ecosystem for remote instrument control is a mess, but stuff like INDI (which is network-native and so has an understanding of these sorts of client/server-at-a-distance protocol considerations baked in, as well as being properly open-source and open-community) is the way to go. I'd stick to HTTP RESTful interfaces where required for control or ensure you're doing application level confirmation of commands in an RPC-style manner to ensure that the remote application both received your message and is able to act on it.

Doing local fallback for safety is a sensible approach as Gina is already doing.

Thanks for you verbose opinion 🙂

[discardedastro]

32 minutes ago, stash_old said:

Thanks for you verbose opinion 🙂

That's fair - my apologies for something of a rant that didn't contribute much. I thought it worth calling out the things to consider with MQTT vs other protocols people might be familiar with.