Triple imaging rig networking problem.

[Gina]

I have a triple imaging rig with 3 RPi boards and using wired LAN to the house.  Indoors the rig is controlled by KStars/Ekos running in Linux Mint on my tower system.

All was working fine a few nights ago but then on a later night I kept getting "Broken pipe" errors and network disconnection. This wasn't all at once but one system at a time with a few minutes in between.

LAN consists of router plus one network switch indoors and a second network switch on the rig.  Both Netgear 5 port units.  I also have a WiFi LAN if that's relevant.

On the rig are 2 RPi 3B boards running Raspian and indiserver with drivers plus an RPi 4B running Astroberry Server.  The first RPi 3B controls the EQ8 mount and has 2 CMOS cameras, ASI 1600MM-Cool for imaging and ASI 178MM for guiding.  The other RPi boards run imaging only.  Except that I also have remote focus for the imaging cameras using the Astroberry Focuser driver.  The remote systems run servers and the Mint machine runs 3 instances of KStars/Ekos as clients, controlling everything via 3 individual SSH channels.

Cable to obsy is CAT6 outdoors rated cable allowing gigabit data rate.

Nothing has changed between the night the kit worked and later when it lost networking.  The only thing I can think of is that it got warmer.  I have fan cooling through the box with the 3 RPi boards and HATs.  The main obsy power of 13.8v feeds the rig with a 12/24v to 5v 10A voltage dropper/regulator feeding the RPi boards.   1000µF electrolytic and 0.1µF ceramic capacitors are connected across the incoming power connector.

Any thoughts on what could have gone wrong and/or what I can do to fix it would be much appreciated.  Lots of clear night skies and kit problems!!!  Frustration unlimited!!

[Dr_Ju_ju]

Yes, it may just be the temperature, I'm having issues with some Arduino focuser's not want to play ball, but if I spray some coolant on them they come back to life....  

Do you have a temperature probe within the RPi box ?  to at least tell you what your facing ....

[Gina]

No I don't have a temperature sensor in the box.  Not sure just where I'd out it.

[Dr_Ju_ju]

I've not looked too closely, but I believe the RPi's themselves have an in-built sensor, so following theis may give you the info https://pimylifeup.com/raspberry-pi-temperature/

[Gina]

It works over SSH.

[pete_l]

On very rare occasions I have had the vcgencmd process hang with this "broken pipe" state (though it didn't report it as it was run from cron). This resulted in a borked NFS connection from the pi-zero-w that the problem occurred on so it no longer copied images back to the server,

A side effect was that the number of processes backed up behind this increased massively (the command ran every 10 seconds) and filled the swap space. I think that is what froze the system and killed the network connection.
It only happened twice. The first time I put it down to just one of those things. The second time I investigated and changed my monitoring script from using vcgencmd to get the CPU temperature to looking for the data in /sys (or was it /proc?)/

[Gina]

I did check the swap when I was having the problem and none was in use.  Other system functions too.  I always check such things if I get any problem.

Edited July 17, 2021 by Gina

[JonCarleton]

Odd issue.  I'd worry more about cables than the Pi hardware or configuration.  Ethernet cables that sit in a humid environment sometimes corrode a bit and cause that issue.  As to the temperature issue...also a possibility.  I'd throw a box fan out there and see if it would make the problem go away.  If so, it is fairly easy to install little fans in your Pi boxes.  There is a possibility that the switch itself is having temp issues (hence the overkill of a box fan pointed at your equipment).

I run all my stuff for each setup on a single Pi4 8G.  It has a fan case and stays cool.  This allows the software to run on the Pi and I just export the keyboard, mouse and HDMI video into the house with a KVM ethernet extender.  No network traffic.