Setting up an MQTT system for Weather Sensing and Astro Control

[wimvb]

The general problem in the maker society is that many tutorials and code are proof of concept, but still far away from stable ”production” versions. Software solutions may work in a specific configuration, but not generally.

[Gina]

Guess I'll just have to keep my fingers crossed then.  Might be a good idea to have the rain sensor on the same unit as the roof controller (as in the current design which doesn't use MQTT).  No particular reason to have it connected to a different one anyway.

[davefi]

3 hours ago, stash_old said:

If your really get the automation bug    https://www.home-assistant.io/                                          

I was pondering suggesting that myself. I currently have a commercial weather station that broadcasts its data via it's hub over UDP. I have a python listener script running on a Pi3A+, which decodes the UDP data, does come calculations to derive additional data, and pushes it to my broker running on a Pi3B. Home Assistant MQTT sensors are set up to listen to the topics, and display the results in the Home Assistant front end (also running on another Pi3B) - I haven't made it very fancy, yet!

Certainly not suggesting to use a commercial station, in fact I am looking to build my own now off the back of several threads on here (I ended up on these forums looking at the DIY All Sky Camera's) - but more to give an idea of what can be done with the data and MQTT.

As for the Pi's, I have dozens of them and I have never killed any (same can not be said for the PiCamera modules, however!). I do only use Pimoroni and Pi Hut for the hardware though, but I would be surprised if that matters.

Hope that's of some interest - back to lurking and reading Observatory Build threads for me 🙂

 

[stash_old]

14 hours ago, wimvb said:

The general problem in the maker society is that many tutorials and code are proof of concept, but still far away from stable ”production” versions. Software solutions may work in a specific configuration, but not generally.

Have to disagree I am afraid the "maker society" is no different than other tutorials with proof of concept. However systems like Home Assistant and many more IOT or Maker systems are just like Indilib or Ascom - created and well supported by open source developers.

While I agree that for a MQTT broker the ESP32 would not be suitable, running a simple low usage Web Server on a ESP8266 or the later ESP32 is perfectly feasable - I .like many applications  , use one to configure initial "comms" on both ESP type with no problems and it is reliable. However some "ESP" modules are better than others - I use "real" Wemos devices which I have found to be very good.

Lets face it there is nothing perfect in this world and IT is no different.

Good luck to Gina and enjoy the project - just let me know when you have  a Talking interface to your Astro set up LOL

[Gina]

Glad to have a more optimistic opinion!  😁

[Gina]

The RPi Zero has arrived.  Not sure I can use it yet though as I'm waiting for delivery of a mini HDMI cable.  Depends if it will work with the microSD card I produced for the RPi 3B+  Have to say, I'm impressed with The Pi Hut - with RM 24 delivery.

[jiberjaber]

42 minutes ago, Gina said:

The RPi Zero has arrived.  Not sure I can use it yet though as I'm waiting for delivery of a mini HDMI cable.  Depends if it will work with the microSD card I produced for the RPi 3B+  Have to say, I'm impressed with The Pi Hut - with RM 24 delivery.

You can set the wifi and SSH settings using a config file in the /boot directory so you could get it up and running headless? 

https://desertbot.io/blog/setup-pi-zero-w-headless-wifi

Basically create a blank file called SSH and add your wifi details to the supplicant file as shown... 

Edited August 14, 2020 by jiberjaber

[Gina]

I'll check the data on the card. 

I thought of a wired connection but when I came to plug the Ethernet cable into the RPi - lo and behold - no Ethernet port, so it has to be WiFi.

[Gina]

Have microSD card in card reader and there's no SSH file and no wpa_supplicant.conf and yet I had SSH working over WiFi on the RPi 3B+.  Guess the problem I had deleted these.

[Gina]

Put empty file SSH into boot directory and made wpa_supplicant.conf file with the info in it.

gina@Mint-Desktop / $ cd media
gina@Mint-Desktop /media $ ls
gina
gina@Mint-Desktop /media $ cd gina
gina@Mint-Desktop /media/gina $ ls
3630-3533  512-NTFS-1  512-NTFS-2  6363-3732  boot  rootfs  ssdata2
gina@Mint-Desktop /media/gina $ cd boot
gina@Mint-Desktop /media/gina/boot $ ls
bcm2708-rpi-b.dtb         COPYING.linux  LICENCE.broadcom
bcm2708-rpi-b-plus.dtb    fixup4cd.dat   overlays
bcm2708-rpi-cm.dtb        fixup4.dat     ssh
bcm2708-rpi-zero.dtb      fixup4db.dat   start4cd.elf
bcm2708-rpi-zero-w.dtb    fixup4x.dat    start4db.elf
bcm2709-rpi-2-b.dtb       fixup_cd.dat   start4.elf
bcm2710-rpi-2-b.dtb       fixup.dat      start4x.elf
bcm2710-rpi-3-b.dtb       fixup_db.dat   start_cd.elf
bcm2710-rpi-3-b-plus.dtb  fixup_x.dat    start_db.elf
bcm2710-rpi-cm3.dtb       issue.txt      start.elf
bcm2711-rpi-4-b.dtb       kernel7.img    start_x.elf
bootcode.bin              kernel7l.img   wpa_supplicant.conf
cmdline.txt               kernel8.img
config.txt                kernel.img
gina@Mint-Desktop /media/gina/boot $ 

Now to see if it works.  ie. if I can SSH into the RPi Zero.

[Gina]

Got as far as password again.

gina@Mint-Desktop ~ $ ssh 192.168.1.250
The authenticity of host '192.168.1.250 (192.168.1.250)' can't be established.
ECDSA key fingerprint is SHA256:lEMA9QPPY5y8I57L7qlLbXpK95DZXF8hlbdZJJPvtd0.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.1.250' (ECDSA) to the list of known hosts.
gina@192.168.1.250's password: 
Permission denied, please try again.
gina@192.168.1.250's password: 
Permission denied, please try again.
gina@192.168.1.250's password: 

Where is the user password stored?  Can I change it by editing a file?  Or do I have to start again and use the default PW "raspberry".  Neither that or the one I set up now work.

[Gina]

I think it's back to square one!  Or almost.  I have the OS in my downloads and the supplicant data in the editor.  Seems I won't need keyboard or monitor connected to the RPi Zero.

[wimvb]

4 hours ago, stash_old said:

Have to disagree I am afraid the "maker society" is no different than other tutorials with proof of concept. However systems like Home Assistant and many more IOT or Maker systems are just like Indilib or Ascom - created and well supported by open source developers.

Absolutely so. Systems that catch on generally evolve into well supported, stable systems. But this takes time. When I started with INDI, a new version could break things that worked before. Then, when Jasem Mutlaq decided to divide versions into stable and nightly builds, this was a leap forward in general stability. something similar happened with the Rock64 sbc.
My point is rather that tutorials show proof of concept, and that more investigation is generally required in order to get to a stable system/setup. This is not unique for the maker community. But the abundance of tutorials and examples makes it very obvious here.

Oh, @Gina, this is absolutely no criticism to what you post here.  I enjoy your threads, and have learned much that has helped me in my diy projects.

Edited August 14, 2020 by wimvb

[stash_old]

And when you have done Gina build one of these - https://create.arduino.cc/projecthub/brettoliver49/arduino-barometer-eb5bdd

 

[Gina]

I certainly could though what I have in mind is a bit different.  I want temperature, humidity, wind speed, wind direction, rainfall, rain status, cloud status, etc. in addition to pressure and its trends.  Trouble is lack of free wall space.  The space above the fireplace has a giant clock with 3ft dial and many of my watercolour paintings plus router and WiFi AP.  East wall also has paintings and my moon dial clock plus a large bookshelf and air-con unit.  And a window, vintage barometer etc.

[Gina]

OK.  SSH into RPi Zero and change of host name and password done plus file system expanded.  Now to try SSH with new host name and PW.

[Gina]

Not surprisingly, I think I've done something wrong in changing host name and password.  I decided on a host name of MQTT_Broker_Gina though this system seems to ignore case unlike most Linux systems.

gina@Mint-Desktop ~ $ ssh pi@raspberrypi
The authenticity of host 'raspberrypi (192.168.1.250)' can't be established.
ECDSA key fingerprint is SHA256:eH9cpckOI0MiUwVEkp18LQ17PoYs0DoJtoLHcePf5yg.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'raspberrypi' (ECDSA) to the list of known hosts.
pi@raspberrypi's password: 
Linux raspberrypi 4.19.118+ #1311 Mon Apr 27 14:16:15 BST 2020 armv6l

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.

SSH is enabled and the default password for the 'pi' user has not been changed.
This is a security risk - please login as the 'pi' user and type 'passwd' to set a new password.

pi@raspberrypi:~ $ sudo raspi-config
New password: 
Retype new password: 
passwd: password updated successfully

Welcome to fdisk (util-linux 2.33.1).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.


Command (m for help): Disk /dev/mmcblk0: 14.9 GiB, 15931539456 bytes, 31116288 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x55d719e5

Device         Boot  Start      End  Sectors  Size Id Type
/dev/mmcblk0p1        8192   532479   524288  256M  c W95 FAT32 (LBA)
/dev/mmcblk0p2      532480 31116287 30583808 14.6G 83 Linux

Command (m for help): Partition number (1,2, default 2): 
Partition 2 has been deleted.

Command (m for help): Partition type
   p   primary (1 primary, 0 extended, 3 free)
   e   extended (container for logical partitions)
Select (default p): Partition number (2-4, default 2): First sector (2048-31116287, default 2048): Last sector, +/-sectors or +/-size{K,M,G,T,P} (532480-31116287, default 31116287): 
Created a new partition 2 of type 'Linux' and of size 14.6 GiB.
Partition #2 contains a ext4 signature.

Command (m for help): 
Disk /dev/mmcblk0: 14.9 GiB, 15931539456 bytes, 31116288 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x55d719e5

Device         Boot  Start      End  Sectors  Size Id Type
/dev/mmcblk0p1        8192   532479   524288  256M  c W95 FAT32 (LBA)
/dev/mmcblk0p2      532480 31116287 30583808 14.6G 83 Linux

Command (m for help): The partition table has been altered.
Syncing disks.

Connection to raspberrypi closed by remote host.
Connection to raspberrypi closed.
gina@Mint-Desktop ~ $ ssh pi@raspberrypi
ssh: Could not resolve hostname raspberrypi: Name or service not known
gina@Mint-Desktop ~ $ ssh pi@MQTT_Broker_Gina
ssh: Could not resolve hostname mqtt_broker_gina: Name or service not known
gina@Mint-Desktop ~ $ 

 

[Gina]

Looks like another restart!!  Maybe it doesn't like that host name.  Checking Network Clients on router...

Aha - no underlines.

[Gina]

That was it - I'm in 😀

gina@Mint-Desktop ~ $ ssh pi@MQTTBrokerGina
The authenticity of host 'mqttbrokergina (192.168.1.250)' can't be established.
ECDSA key fingerprint is SHA256:eH9cpckOI0MiUwVEkp18LQ17PoYs0DoJtoLHcePf5yg.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'mqttbrokergina' (ECDSA) to the list of known hosts.
pi@mqttbrokergina's password: 
Linux MQTTBrokerGina 4.19.118+ #1311 Mon Apr 27 14:16:15 BST 2020 armv6l

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Fri Aug 14 16:38:52 2020 from 192.168.1.79
pi@MQTTBrokerGina:~ $ 

 

[Gina]

Now running the usual :-

sudo apt-get update -y
sudo apt-get upgrade -y

 

[Gina]

That completes setting up the Pi Zero W for headless SSH access over WiFi.

[Gina]

Next job is :-  Setup MQTT Server on Raspberry Pi Zero

This installs both the Mosquitto broker and clients so the RPi can be used as both broker and a client.  I may use it for either the indoors sensors or the mechanical weather wall display with several dials and gauges.  Thinking of the latter as the RPi has lots of GPIO pins.

Edited August 14, 2020 by Gina
typos

[Gina]

OK...  Running this sequence :-

Quote

1.     Starting with a fresh installation of Raspbian Buster Lite (burned here with Balena Etcher), enter sudo apt update then sudo apt upgrade in the command line to make sure you’ve got the latest and greatest running on your system.

2.     Add the open source MQTT broker Mosquitto, along with Mosquitto client software on the Pi with sudo apt-get install mosquitto mosquitto-clients.

3.     The new host address will be the IP address of the Pi, which is available by typing ifconfig. Installation will start the broker and allow it to start on boot. The mosquitto-clients portion of the install allows you to run a client on the machine as well as a broker.

4.     In the Pi terminal, subscribe to the test topic with the command mosquitto_sub -h localhost -t “test”. This sends a subscription message to the MQTT broker running on the same system and lets it listen to any messages sent to this topic.

Sections 1. to 3. completed.  In section 3. ifconfig confirms the IP address as correct.  In section 4. I copied the command into the Terminal but nothing happened.  Does the RPi need rebooting or am I completely misunderstanding this?

[Gina]

Tried rebooting - not solved it though.

pi@MQTTBrokerGina:~ $ sudo reboot
Connection to mqttbrokergina closed by remote host.
Connection to mqttbrokergina closed.
gina@Mint-Desktop ~ $ ssh pi@MQTTBrokerGina
pi@mqttbrokergina's password: 
Linux MQTTBrokerGina 5.4.51+ #1333 Mon Aug 10 16:38:02 BST 2020 armv6l

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Fri Aug 14 17:45:07 2020 from 192.168.1.79
pi@MQTTBrokerGina:~ $ mosquitto_sub -h localhost -t “test”

And there it sits doing nothing!  Evidently I'm not understanding this.  Can someone please tell me what I'm doing wrong?

[Gina]

I've GOT IT!! 😀

Each command has to be on a separate SSH connection to the broker.  The first Terminal is just sitting there waiting for messages from clients.  When another terminal is opened and SSH'd to the broker subscribing to "test" and sending the message "Hello", then "Hello" appears in the first Terminal.

pi@MQTTBrokerGina:~ $ mosquitto_sub -h localhost -t “test”

“Hello”

The MQTT system is working albeit with broker and clients on the same machine (RPi).  😀

Edited August 14, 2020 by Gina