Setting up an MQTT system for Weather Sensing and Astro Control

[JamesF]

I reckon it could be.   When I went swimming today I could see half of the Blackdown Hills were covered in mist as I dropped into Wellington, so there's clearly a lot of wet stuff hanging around down your way.

James

[Gina]

Indeed there is!

[Gina]

That may well be alright but there's still a problem with so few messages getting through from the Living Room sensor (topic esp32).  20m or more between messages.  It was fine until I connected the outside sensor.  Makes me wonder if the RPi Zero W is up to the job.  I now have an RPi 3B I could try.  Guess just a matter of taking the card out of the Zero and putting it in the 3B (or is it?). 

Having an Ethernet port and could use a wired connection rather than WiFi could be an advantage of the RPi 3B.

Edited August 18, 2020 by Gina

[Padraic M]

Yes, humidity may be correct - you'll just need to leave it be (or bring it inside) to see if it adjusts.

Puzzled by the pressure spikes. I've found the BMP280 sensor to be very reliable - the calculation is so complicated you're unlikely to end up with an incorrect result! I just checked my database and the min/max values for the last two days are 986-991. No outliers in 33,000+ data points. 

Also your spikes are invalid pressure figures - 400hPa, 1,200hPa etc...? Is it windy where the outdoor sensor is located?

Possibly it's a failed call to the sensor, and the un-initialised result is being recorded anyway...? A quick fix could be to apply thresholds and drop the invalid values, but you'd prefer to get an explanation for it.

I can't imagine that the RPi Zero mqtt would struggle to keep up with such a small number of messages. When I see drop-outs, it's almost always because of a wireless connection issue. I find wireless routers so unreliable  - even modern routers. I have a Chromecast dongle that manages to disrupt almost all communications on its wifi channel. 

If you disconnect the outside sensor does the indoor one start recording again?

btw, I've just installed MQTT Explorer. Hadn't come across it before, and I'm enjoying watching my messages flow through. Thanks for that tip.

[Gina]

Raining this morning and it's coming straight down so little or no breeze.   Here's the current dashboard display.  I've limited the outside/pressure range to 980-1020 to show the pressure more sensibly.  Yes, I could scrap the invalid values but wonder why I'm getting them.  The forecast is for the rain to stop later this morning and become sunny so can see what happens then.

No new readings from esp32 (Living Room) since last night.  I have a separate WiFi AP connected to the router - a Ubiquity, which has a good reputation and gives good signal strength out in the obsy about 20m away.  Of course, the Living Room ESP32 is quite close to the AP (<2m) so that shouldn't be a problem.

[Gina]

Another screenshot of the dashboard display and have had a bunch of messages from the esp32.

[Gina]

[Gina]

Tried to find out how many clients a Raspberry Pi Zero W MQTT broker can support but so far have found nothing.  Anyone know???

Edited August 19, 2020 by Gina

[7170]

1 hour ago, Gina said:

I've limited the outside/pressure range to 980-1020 to show the pressure more sensibly.  Yes, I could scrap the invalid values but wonder why I'm getting them.

Are you running the sensor in Forced Mode, with oversampling at say 8x or 16x? As this isn't always the default using libraries. If you still get a bad reading after that I would be suspecting an issue with the sensor or power supply problem myself.

[Gina]

I've no idea what Forced Mode is - I've just followed the tutorials so I'll have to do some research.

[wimvb]

2 hours ago, Gina said:

Tried to find out how many clients a Raspberry Pi Zero W MQTT broker can support but so far have found nothing.  Anyone know???

Maybe it’s not the number of clients, but the timing. Is there a way to find out when and for how long each client is sending information? I would think that in general, mqtt can support many connected clients, but not sending simultaneously. Where is the Raspberry pi located? Near the living room esp? Otherwise signal strength could be an issue.

Just guessing.

Btw, have you read this?

http://www.steves-internet-guide.com/mqtt-works/

Edited August 19, 2020 by wimvb

[skybadger]

The ESP stack is sensitive to needing reconnects and has historic issues with connections being dropped, so your code needs to check the client connection is still viable and remake it if necessary. A reboot typically fixes this until the next drop, which may be hours or minutes away. The ESp8266 needs a combination of LWIP 1.4 and high bandwidth settings to enable client connections and in the loop coding to reconnecti on drop. The MQTT broker will have capacity for hundreds or thousands of connections, limited principally  by memory since there is no timing guarantee on messages. 

 

[skybadger]

Here's an example from mine. 

[skybadger]

As far as bad readings go. I would check my loop timings. I'v had sensors that are quite sensitive to both i2c timings - too slow and they get upset and to polling timings, too short or too long compared to acquisition time and they get upset. 

The worst one for timing has been the HTU21D and the worst one for default readings has been the BMP280 due to i2c timings. Setting i2c speed back to 100KHz from 50 KHz solved that. 

[Gina]

21 minutes ago, wimvb said:

Maybe it’s not the number of clients, but the timing. Is there a way to find out when and for how long each client is sending information? I would think that in general, mqtt can support many connected clients, but not sending simultaneously. Where is the Raspberry pi located? Near the living room esp? Otherwise signal strength could be an issue.

Just guessing.

The RPi is less than a metre from the living room esp and both are less than 2m from the WiFi AP. I don't know of any way to find out when and for how long each client is sending information but the QoS is 2 so I would have thought it would send the data until it got a received message.  Or is this the problem in some way?  The outside sensors are also set to QoS of 2.

[stash_old]

Never had a bad problem with MQTT on a RPI but then I use a RPI3 not a zero. Also ,as Skybadger says, you have to allow ,in your code,reconnects for Wifi and dont use too tight loops which block the ESP Stack but I haven't changed any of the default ESP stack or MQTT parameters. This was all using ESP8266(Wemos) not a ESP32. Zero's had a bad reputation for Wifi drop outs at one time (google Zero Wifi problems) but I would have thought they had sorted them by now (or maybe not) - assuming you are using Std Raspian not Mint etc on RPI zero.

Plus I think you can access the Broker LOG via MQTT-explorer or equiv (else Linux file in \var etc) to see any problems that might be shown.

[Gina]

20 minutes ago, skybadger said:

The ESP stack is sensitive to needing reconnects and has historic issues with connections being dropped, so your code needs to check the client connection is still viable and remake it if necessary. A reboot typically fixes this until the next drop, which may be hours or minutes away. The ESp8266 needs a combination of LWIP 1.4 and high bandwidth settings to enable client connections and in the loop coding to reconnecti on drop. The MQTT broker will have capacity for hundreds or thousands of connections, limited principally  by memory since there is no timing guarantee on messages.

The client connection is tested and remade if needed.

void reconnect() {
  // Loop until we're reconnected
  while (!client.connected()) {
    Serial.print("Attempting MQTT connection...");
    // Attempt to connect
    if (client.connect("ESP8266Client")) {
      Serial.println("connected");
      // Subscribe
      client.subscribe("esp32/output");
      client.subscribe("esp32/roof");
    } else {
      Serial.print("failed, rc=");
      Serial.print(client.state());
      Serial.println(" try again in 5 seconds");
      // Wait 5 seconds before retrying
      delay(5000);
    }
  }
}
void loop() {
  if (!client.connected()) {
    reconnect();
  }
  client.loop();

  long now = millis();
  if (now - lastMsg > 5000) {
    lastMsg = now;

It then goes on to read sensors and send the data.

[Gina]

19 minutes ago, skybadger said:

As far as bad readings go. I would check my loop timings. I'v had sensors that are quite sensitive to both i2c timings - too slow and they get upset and to polling timings, too short or too long compared to acquisition time and they get upset. 

The worst one for timing has been the HTU21D and the worst one for default readings has been the BMP280 due to i2c timings. Setting i2c speed back to 100KHz from 50 KHz solved that. 

I'm not too concerned about pressure spikes as I can kill those in code.  For most of the time the readings are fine and the odd missed reading would not be a problem.  I could sort this out later when I have got more important issues sorted out.

[Gina]

26 minutes ago, stash_old said:

Never had a bad problem with MQTT on a RPI but then I use a RPI3 not a zero. Also ,as Skybadger says, you have to allow ,in your code,reconnects for Wifi and dont use too tight loops which block the ESP Stack but I haven't changed any of the default ESP stack or MQTT parameters. This was all using ESP8266(Wemos) not a ESP32. Zero's had a bad reputation for Wifi drop outs at one time (google Zero Wifi problems) but I would have thought they had sorted them by now (or maybe not) - assuming you are using Std Raspian not Mint etc on RPI zero.

Plus I think you can access the Broker LOG via MQTT-explorer or equiv (else Linux file in \var etc) to see any problems that might be shown.

I now have an RPi 3B which I could try instead of the Zero.  I think I may be able to just put the microSD card from the Zero into the 3B to run the software but the IP address will be different.

MQTT Explorer shows that the broker never received the messages from the Living Room esp.  Do you think I should try an ESP8266 for some of my clients?  I need the ESP32 for some for the number and types of GPIO pins but the Living Room sensor is only reading temperature, humidity and pressure.  Outside and obsy sensor could also be ESP8266 but the obsy roof needs an ESP32 for the GPIOs as does the wind sensor.  I went for all ESP32s just to keep things simple (all the same type of module).  I also thought that being a later device the ESP32 may be better but that was only a feeling, I think.

Edited August 19, 2020 by Gina

[stash_old]

20 minutes ago, Gina said:

Do you think I should try an ESP8266 for some of my clients?

The only reason I did was that 3.5yrs ago the ESP32 was new new to me and I would have expect,after initial teething,the esp32 would be better supported depending on the Manu.

Gut feelings are normally not far off :-).

Have you any debug messages (that you can switch on or add) in your Living Room ESP to show WiFi disconnects, MQTT broker connection failures etc - always a thin line having Debug messages as they can impact on the flow IMO.

A suggestion - you could switch off all your devices(clients) but 1 see how it behaves and add a new device in a orderly manner(say 1hr apart) checking for problems!

[Gina]

I have switched off debug messages from the outside/obsy client and there are no debug messages showing.  The esp (Living Room) client is not generating any debug messages.  Tried the esp client unit on a bench PSU instead of USB - no difference.  Now back on USB and this is the Serial Monitor screenshot.

This is the complete sketch.  It has a few "test" only parts as well as BME280 sensor readings.

/*********
  Rui Santos
  Complete project details at https://randomnerdtutorials.com  
*********/

#include <WiFi.h>
#include <PubSubClient.h>
#include <Wire.h>
#include <Adafruit_BME280.h>
#include <Adafruit_Sensor.h>

// Replace the next variables with your SSID/Password combination
const char* ssid = "Ubiquity";
const char* password = "********";  // changed from my proper WiFi PW

// Add your MQTT Broker IP address, example:
//const char* mqtt_server = "192.168.1.144";
const char* mqtt_server = "192.168.1.250";

WiFiClient espClient;
PubSubClient client(espClient);
long lastMsg = 0;
char msg[50];
int value = 0;

//uncomment the following lines if you're using SPI
/*#include <SPI.h>
#define BME_SCK 18
#define BME_MISO 19
#define BME_MOSI 23
#define BME_CS 5*/

Adafruit_BME280 bme; // I2C
//Adafruit_BME280 bme(BME_CS); // hardware SPI
//Adafruit_BME280 bme(BME_CS, BME_MOSI, BME_MISO, BME_SCK); // software SPI
float temperature = 0;
float humidity = 0;
float pressure = 1000.0;
String statusString = "Closed";

// LED Pin
const int ledPin = 4;

void setup() {
  Serial.begin(115200);
  // default settings
  // (you can also pass in a Wire library object like &Wire2)
  //status = bme.begin();  
  if (!bme.begin(0x76)) {
    Serial.println("Could not find a valid BME280 sensor, check wiring!");
    while (1);
  }
  setup_wifi();
  client.setServer(mqtt_server, 1883);
  client.setCallback(callback);

//  pinMode(ledPin, OUTPUT);
}

void setup_wifi() {
  delay(10);
  // We start by connecting to a WiFi network
  Serial.println();
  Serial.print("Connecting to ");
  Serial.println(ssid);

  WiFi.begin(ssid, password);

  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }

  Serial.println("");
  Serial.println("WiFi connected");
  Serial.println("IP address: ");
  Serial.println(WiFi.localIP());
}

void callback(char* topic, byte* message, unsigned int length) {
  Serial.print("Message arrived on topic: ");
  Serial.print(topic);
  Serial.print(". Message: ");
  String messageTemp;
  
  for (int i = 0; i < length; i++) {
    Serial.print((char)message[i]);
    messageTemp += (char)message[i];
  }
  Serial.println();

  // Feel free to add more if statements to control more GPIOs with MQTT

  // If a message is received on the topic esp32/output, you check if the message is either "on" or "off". 
  // Changes the output state according to the message
  if (String(topic) == "esp32/output") {
    Serial.print("Changing output to ");
    if(messageTemp == "on"){
      Serial.println("on");
      digitalWrite(ledPin, HIGH);
    }
    else if(messageTemp == "off"){
      Serial.println("off");
//      digitalWrite(ledPin, LOW);
    }
  }
}

void reconnect() {
  // Loop until we're reconnected
  while (!client.connected()) {
    Serial.print("Attempting MQTT connection...");
    // Attempt to connect
    if (client.connect("ESP8266Client")) {
      Serial.println("connected");
      // Subscribe
      client.subscribe("esp32/output");
      client.subscribe("esp32/roof");
    } else {
      Serial.print("failed, rc=");
      Serial.print(client.state());
      Serial.println(" try again in 5 seconds");
      // Wait 5 seconds before retrying
      delay(5000);
    }
  }
}
void loop() {
  if (!client.connected()) {
    reconnect();
  }
  client.loop();

  long now = millis();
  if (now - lastMsg > 5000) {
    lastMsg = now;
    
    // Temperature in Celsius
    temperature = bme.readTemperature();   
    // Uncomment the next line to set temperature in Fahrenheit 
    // (and comment the previous temperature line)
    //temperature = 1.8 * bme.readTemperature() + 32; // Temperature in Fahrenheit
    
    // Convert the value to a char array
    char tempString[8];
    dtostrf(temperature, 1, 2, tempString);
    Serial.print("Temperature: ");
    Serial.println(tempString);
    client.publish("esp32/temperature", tempString);

    humidity = bme.readHumidity();
    
    // Convert the value to a char array
    char humString[8];
    dtostrf(humidity, 1, 2, humString);
    Serial.print("Humidity: ");
    Serial.println(humString);
    client.publish("esp32/humidity", humString);

    pressure = bme.readPressure() / 100.0F;
    
    // Convert the value to a char array
    char presString[8];
    dtostrf(pressure, 1, 2, presString);
    Serial.print("Pressure: ");
    Serial.println(presString);
    client.publish("esp32/pressure", presString);
    
    
    // Send roof ststus
    char statString[8] = "Closed";
    Serial.print("ROR Status: ");
    Serial.println(statString);
    client.publish("esp32/roof/status", statString);
  }
}

 

[Gina]

Here's the latest dashboard display.  The rain has eased off a bit so I'll go out and turn off the outside/obsy client.

[Gina]

Now have outside/obsy sensor unit disconnected (and brought indoors to add the second BME280).  The transmissions from the esp (Living Room) have resumed at normal rate!!

[stash_old]

The MQTT client name/ID isn't dupilcated is it - has to be unique

[stash_old]

The only thing I THINK I can suggest if to check for WiFi status in your loop - what happens if the wifi has disconnected the MQTT should fail. Just needs a Wifi status check and if not connected connect Wifi.  Unless I have missed something. Just move 3 m3 of firewood so brain and old bones complaining.

Just checked my old code I don't check Wifi status either LOL - rest very much like yours except I left myself this note

  //
    //     change client id else problems ******************************************
    //

 

And send data every 60secs

Edited August 19, 2020 by stash_old