Setting up an MQTT system for Weather Sensing and Astro Control

[davefi]

Yeah emmc is certainly better than SD based storage. I'm just a bit obsessed with RPi so haven't looked at many other SBC's 🙂 I've only just started playing with the Wemo D1 Mini's, I really like their form factor. I've got some ESP32's in the D1 Mini form factor coming to have a play with as well. Just need to find the time to actually complete some projects now!

[Gina]

I'm still having a problem displaying the dashboard.  I'm wondering if I'm trying to display too much information.  Or too much changing information.  Most of my sensors are sending messages every 5s and several in quick succession.  It really isn't necessary to have the data updated this often and generally the actual values don't change that quickly.  The only reason for the rapidly changing data is noise.  The only truly rapidly changing data is wind gusts and even that doesn't want displaying every few seconds.  I think it would be better to average the data and send MQTT messages far less often.

Or am I barking up the wrong tree here?

Edited August 25, 2020 by Gina

[wimvb]

My suggestion is, do some averaging on the esp, and send/display data every minute. That should also take care of the noise.

[Gina]

I shall do that then.

[stash_old]

You might also want to check that you are NOT sending random "Nan" values from the ESP - I have found that some devices/sensors provide rubbish non numeric values (hence NAN - not a number) which if not filtered out somewhere(best done in the ESP) before displaying the actual data create problems.

I agree that there seems very little reason to provide data from most devices every 5s - obviously any rain detection needs to send its started raining ASAP however the timings can and should be controlled ,IMHO, from within the ESP.

I doubt you "large" display is causing your issues as many Home Automation systems have far more complicated display layouts. If you still suspect that is the issue you can of course split the display into tabbed windows easily in Node-red Dashboard and see if that is really the issue.

But I go with Wimvb first try the 60s refresh/send loop.

[Gina]

I have now applied averaging in all the currently running clients.  Taking readings every 5s as before but accumulating data over 12 intervals then dividing by 12, sending the messages to MQTT and resetting ready for the next 12 intervals.  I have also reset the RPi to clear all old data and started again.  Looking better so far but it's "early days"!

[Gina]

I'm still displaying readings to the default 2 decimal places when the readings are nothing like that accurate.  In fact integer values would suit the reading accuracy better.  I see no point in showing the atmospheric pressure more accurately than integer values but I may round temperature and humidity to tenths.

Edited August 26, 2020 by Gina

[Gina]

Since the atmospheric pressure often changes very slowly I've decided to round to one decimal place.  Same for temperatures but pressures I've set to zero decimal places.  Just a matter of setting the appropriate number of decimal places in the dtostrf function which converts numbers to a char array for MQTT.

[Gina]

This is the dashboard display.  The pressure shows where I had rounding to integer (which seemed unsuitable).  The wind sensor is awaiting development.

[Gina]

6 hours later and the display is still good with no "lost connection" errors!!  Seem to have cured the problem.

[Gina]

Dashboard display still fine.  Hoping to make progress on the sketch for the rain sensors today - lots of data processing.

[Gina]

I like this system for displaying current conditions and recent history.  Of course, standard weather data logging and display is another matter.  That may come later as it's something I've always been interested in but there's lots to do before that.  There are more sensors and data processing to add to the current system plus the ROR control.  Although in some ways a weather station and control of the observatory roof are two distinct systems, they do relate to each other and covering both in one MQTT system makes sense.

[Gina]

The automatic Y range of the history graphs works quite well for the pressure but less well for temperatures but with the Node-RED system being so easy to use, it's easy to alter the ranges to suit current conditions manually, unlike physical gauges. 

Here is a new display with more suitable temperature ranges IMO.

Edited August 27, 2020 by Gina

[rogmet]

Here's the sketch I put on my sensors that's been running beautifully for several years at this stage.

What's good about it is that it limits the frequency of posting to a max of once per second, it will only post if there's a specified difference between the last post and the current reading, and if it's unfeasibly calm, it'll force a posting every 5 minutes.

I just did some chopping and changing, all credit should go to the original author of course. I was using a Wemos D1 Mini (ESP8266) so pins may need to be updated.

ESP_BME280_MQTT_LWAT.ino

[Gina]

Node-RED Dashboard still fine.

[Gina]

Latest -trying to get wind instruments working.  Mainly problems with the coding.

[Gina]

Now have the Wind sensors working.

[Gina]

Latest.

[Gina]

I'm getting a strange problem with the wind speed.  Whether this is anything to do with MQTT or just the sketch in the wind ESP32 I don't know.  Until today the speed calculations have been fine but now the mean speed keeps resetting then building up again to the correct value.  These are enlargements of the mean speed history in the dashboard and the MQTT Explorer.   I'm wondering if the stronger breeze today could be anything to do with it. 

Unfortunately, to upload a new sketch to the ESP32 in the wind client means taking the wind sensor mast and unit down and connecting the ESP32 to the Mint box USB.  I seem to remember reading something about uploading a new Arduino sketch over WiFi but can't find it now.

 

[discardedastro]

4 hours ago, Gina said:

I'm getting a strange problem with the wind speed.  Whether this is anything to do with MQTT or just the sketch in the wind ESP32 I don't know.  Until today the speed calculations have been fine but now the mean speed keeps resetting then building up again to the correct value.  These are enlargements of the mean speed history in the dashboard and the MQTT Explorer.   I'm wondering if the stronger breeze today could be anything to do with it. 

Unfortunately, to upload a new sketch to the ESP32 in the wind client means taking the wind sensor mast and unit down and connecting the ESP32 to the Mint box USB.  I seem to remember reading something about uploading a new Arduino sketch over WiFi but can't find it now.

Depending on how you're doing your averaging, could be hitting data type limits and wrapping to a negative? If you're averaging on device by doing (x+y+...+z)/n, depending on n the sum of xyzetc could exceed a short/small int.

[JamesF]

28 minutes ago, discardedastro said:

Depending on how you're doing your averaging, could be hitting data type limits and wrapping to a negative? If you're averaging on device by doing (x+y+...+z)/n, depending on n the sum of xyzetc could exceed a short/small int.

I was wondering that too, but I don't know enough about the platform to comment really.

I've just this evening tracked down an oacapture bug that only showed up on the RPi and only for certain types of camera, and went away if I changed some code completely unrelated to what I eventually discovered (after quite a few days of looking) was the actual problem.  Having fixed it, I have no idea why it didn't fail all the time everywhere.  Debugging can be so much fun :D

James

[discardedastro]

1 minute ago, JamesF said:

I was wondering that too, but I don't know enough about the platform to comment really.

I've just this evening tracked down an oacapture bug that only showed up on the RPi and only for certain types of camera, and went away if I changed some code completely unrelated to what I eventually discovered (after quite a few days of looking) was the actual problem.  Having fixed it, I have no idea why it didn't fail all the time everywhere.  Debugging can be so much fun

Those bugs are the best sort of bugs!

This is why I really love working with Rust - while it's a bit slower to write than C and friends, it's super easy to debug and very hard to write code that breaks in odd ways - the compiler doesn't let you. I also had some fun with rust-fuzz to fuzz a binary parser I wrote a few weeks back - throwing automatically-generated random data that attempts to probe absolutely every code path and potential error can throw up errors very easily (especially ones you didn't even consider possible!) when you're dealing with third party input. I keep meaning to have a go with Rust on the ESP32 - there's now a good uC toolchain developing for a (pretty broad) subset of Rust.

[JamesF]

Without wishing to derail Gina's thread, Rust (along with a few others) is a language I've noted as something to look into at some point.  I am a little wary of new languages however.  I reckon they take about twenty years to properly "bed in" and become stable enough to be reliably usable.  Some newer ones that have quite vocal followings can be an absolute nightmare when you want to run multiple applications, often ending up in "Dependency Hell" when one needs specific versions of certain packages/libraries whilst another requires often conflicting/incompatible versions of the same.  Few things make me feel more like giving the developers a good slap

James

Edited September 3, 2020 by JamesF

[discardedastro]

2 minutes ago, JamesF said:

Without wishing to derail Gina's thread, Rust (along with a few others) is a language I've noted as something to look into at some point.  I am a little wary of new languages however.  I reckon they take about twenty years to properly "bed in" and become stable enough to be reliably usable.  Some newer ones that have quite vocal followings can be an absolute nightmare when you want to run multiple applications, often ending up in "Dependency Hell" when one needs specific versions of certain packages/libraries whilst another requires often conflicting/incompatible versions of the same.  Few things make me feel more like giving the developers a good slap

And there's definitely still a bit of that in Rust, though it has at least got a proper package manager (glares at Go and Elixir) baked in pretty well so managing versions of stuff is at least trivial. I think Rust's one of the more promising ones for doing astro coding in - it's designed specifically to focus on systems programming and so lends itself well to doing "always on" applications and that fuzzy area between "safety critical" and "really just don't want this to fall over at 3am and leave the observatory roof open..." where you don't need formal, provable correct code but do want a bit more compile-time scrutiny 🙂 I can heartily recommend giving it a go. But yes, let's stop derailing Gina's thread!

[Gina]

An interesting departure from topic 😁  Though not entirely unrelated.

The problem has been there overnight and still bad.  I have been through the calculations and can't find any "overloaded" numeric types.  I plan to set up another ESP32 for testing as the running one is up inside the wind unit and would require everything taking down and bringing the"works" indoors to update or swap ESP32s.