Setting up an MQTT system for Weather Sensing and Astro Control

[Gina]

No problem with developing the sensor sketches - they are coming on nicely.  My main concern is displaying the dashboard.  It keeps losing the connection.  The RPi is only about a metre from the WiFi AP.  Maybe the WiFi on the RPi Zero W is poor.  Wondering if I would be better off with an RPi 3B which has Ethernet and to use a wired connection rather than WiFi.

Opinions most welcome.

[davefi]

You could look at the Pi3A+ as a mid-way point. I did have a Pi0 running my UDP listener and MQTT client to push my weather data to my MQTT server on my other VLAN, but it kept missing UDP packets.

I swapped out the a 3A+ and its been rock solid. Its still WiFi only. I have so many different Pi's knocking about it was easy to try and see if it worked any better, but if you already have a Pi3B and are able to wire up anyway, then that's likely the best solution.

[Gina]

I'll change to the RPi 3B then and Ethernet.  The IP address will be different so means changing the broker IP in all the clients though - slight nuisance.  I'll leave it until tomorrow.

[Gina]

Now looking at the wind vane optical sensor connections to the ESP32.  These are analog so need the ADC pins.  There are 15 and I need 4.  But if using WiFi (which we are) there is a limitation on which pins to just 6 - see blue panel below.  I shall use the top 4 on the left, next to the EN pin.

 

[Gina]

Looking at the actual ESP32 module the top 2 pins on the left are labelled VP and VN so I think I'll use the next 4 - D34, D35, D32, D33.  These are all ADC1 so should be fine.  I'm using D4 (GPIO4) for the anemometer Hall sensor.

[Gina]

Found an example of a single track Gray encoder.  That would produce a much smaller and simpler wind vane encoder, but this is a 5 bit coding yielding 30 different positions.  The down side is an extra optical sensor and more complex decoding.  Probably the "sledge hammer" method of a look-up table would be to simplest approach.  Also, the smoothing algorithm may not work.  So whilst this is very interesting I'm afraid it's probably "too clever"!!

Link here :- Gray Code

https://en.wikipedia.org/wiki/File:Animated_Graycode.gif

Edited August 22, 2020 by Gina

[Gina]

Guess I'll just use what I've got rather than try such a "way-out" solution.  I've already got the wind vane with built-in Gray encoder printed and just have one or maybe two more parts to print and I shall have the vane hardware finished.  Have the stripboard and components to solder though.  There are 4 resistors needed for the optical sensors.  Not sure about the IR emitters - I'll connect them all in series and see if a resistor is needed to run off the 5v supply.  I'll sort that out tomorrow - I've finished for tonight.

[wimvb]

4 hours ago, Gina said:

Wondering if I would be better off with an RPi 3B which has Ethernet and to use a wired connection rather than WiFi.

Opinions most welcome.

I prefer ethernet for more critical applications, and wifi for devices that get moved around.

[wimvb]

You will have historic data for as long as the Pi is active. If the Pi goes down, you'll lose the complete history. I'm at the moment looking at a video by the "guy with the swiss accent", Andreas Spiess. His solution looks nice. I hadn't heard of influx before.

 

[Padraic M]

Influx is ideal for sensor data. It's optimised for storing time series data on the basis that most of the data will be written but rarely read. Grafana is the generally the dashboard of choice.

Great memo here https://www.influxdata.com/blog/running-the-tick-stack-on-a-raspberry-pi/ on running it on Pi. 

[Gina]

I  shall look at all ideas for logging weather data and displaying and also the Arduino UNO hardware that I used before plus other ideas.  I think I would like to save the data to an SD card.  I haven't decided on a format for certain but comma delimited text looks a prime candidate.  I like simple.  I still hope to make a wall weather display with gauges and a 360 degree wind direction dial.  Servo motors would make good gauges I'm thinking with stepper motors for the wind direction.

[Gina]

Here's the latest Dashboard screenshot, before I unplug the RPi Zero and replace it with a 3B.  Wind Speed isn't running - I've dismantled the test rig.

Edited August 23, 2020 by Gina

[Gina]

RPi Zero W now replaced with the RPI 3B with wired comms, all clients updated to the new broker IP address and everything working.  The microSD card from the Zero now in the 3B and all Node-RED links were preserved.  Display seems solid!  What a difference!!  Of course, logged data is gone but it was getting to the state where the display was just occasionally working for a few seconds and virtually useless.

Edited August 23, 2020 by Gina

[Gina]

[Gina]

Another problem.  Think Node-RED needs the new broker IP address but I can't see where to set it.  Getting this :-

[Gina]

Seems alright now.  Tried changing the IP address in the browser again and this time it seems to have worked.  That's additional to changing the IP address for viewing the dashboard and in all the clients.

[Padraic M]

Top corner menu - select "Configuration Nodes"

Double-click your mqtt broker node and update the IP address in the configuration menu. You'll need to redeploy your flow.

 

Edited August 23, 2020 by Padraic M

[Gina]

Quote

Double-click your mqtt broker node

I don't seem to have that.  But it's working so no longer a problem.  Thanks anyway.

Node-RED is on the same device as the broker so shouldn't need an IP address.

Edited August 23, 2020 by Gina

[Gina]

Still getting the problem with the Dashboard display!!  Is the dashboard too "busy" - too many widgets?

[discardedastro]

I've got InfluxDB for some bits at home and it works... okay. It's a nightmare to figure out query syntax, stuff like migrations/backups/data archival is a nightmare.

We're currently looking to migrate our InfluxDB setup at work to TimescaleDB which is similar but built on PostgreSQL and so avoids a lot of the problems InfluxDB has with data management (a lot of which is sadly now in their commercial release - which we have at work, but it's a five-figures entry price).

https://www.timescale.com/

https://github.com/timescale/timescaledb - the open source version

Whichever solution you end up with Grafana is an awesome bit of software for dashboards - but may not be quite as configurable as what you have there. Much better for long-term data analysis, though, so may be a useful counterpart. You can use Timescale with standard PostgreSQL drivers which node-red has.

If the dashboard is busy I'd be checking the number of concurrent connections you're limiting the broker to?

As for Ethernet vs WiFi - there is no question, for anything you want 24/7 reliability on, stick to Ethernet.

[Gina]

This morning's dashboard display.  Wind Speed not connected - currently working on the wind sensor connections.

Edited August 24, 2020 by Gina

[wimvb]

9 hours ago, discardedastro said:

We're currently looking to migrate our InfluxDB setup at work to TimescaleDB which is similar but built on PostgreSQL and so avoids a lot of the problems InfluxDB has with data management (a lot of which is sadly now in their commercial release - which we have at work, but it's a five-figures entry price).

Can TimescaleDB run on a raspberry pi (under raspbian)? The examples I’ve found so far, either run it as a cloud service or with the db on a different computer.

[discardedastro]

10 hours ago, wimvb said:

Can TimescaleDB run on a raspberry pi (under raspbian)? The examples I’ve found so far, either run it as a cloud service or with the db on a different computer.

Should do - it's just PostgreSQL, basically. The Debian packages ought to work (though they may not have armhf) but the install from source would definitely work:

https://docs.timescale.com/latest/getting-started/installation/debian/installation-source

I'd personally want to keep my long-term data on something with a bit more of a resilient storage solution - SD card failures are still sadly common in high-write situations, even with industrial-use-rated cards. I've seen some creative use of USB hubs and USB sticks/SD cards for low-cost, low-speed RAID, used with ZFS (RAIDZ2), for low-cost-but-robust-as-heck storage on a Pi - but it's not fast, and you can't boot off it, so for my money I'd go for something a bit more PC-like. Lenovo used to have heavy rebates/discounts on their entry level ThinkServer line - not sure if that's still going on.

[davefi]

You can always use SSD's with the RPi's. I've not used SD cards on any of my permanent Pi deployments - and a 120GB SSD is the same price as an SD card these days.

[wimvb]

I use an emmc card on my Rock sbc. AfaIk these are more reliable than SD, and certainly faster. Just ordered a Rock64Pro with 64 GB emmc memory to use as MQTT broker, dashboard, etc. If I find it reliable enough, I will upgrade the Rock64 that controls my imaging rig to a Rock64Pro.