KStars/EKOS/Indi crashing during imaging sessions

[pete_81]

Hi guys,
Think this is the best place to post this issue which I've seen several times before without thinking too much of it.
So over the w/e, we've all been enjoying some really clear nights and I set my rig up on Thursday night on patio, looking to get in some nice imaging time before the shorter days don't make it quite as worth it!
Previously I'd wait until I'd got enough images, clouds rolled in, or bed beckoned, so I'd dismantle setup and retreated. This was the case again Thursday night when clouds came in. But from Friday until this morning, setup has been left out meaning a quick (<15min) startup time, and hoping to leave camera imaging all night long.

M42, IC405 & M81 were the planned targets and happy to say I'm pleased with what I got done, but let down by crashes of unknown origin.
KStars, EKOS & Indi, internal guiding, logging into RPi using the hotspot and astroberry.local/desktop approach.

Each night (well, early morning!), a crash has obviously halted any image grabbing. But no idea what's been causing it. With this issue previously, I'd still be up so be able to restart things. But over this w/e, I'd left the setup doing what it wanted, hoping to see successfully completed imaging session in the morning.
The logfiles (verbose) don't really tell me anything about what caused kstars to close (Indi still running as reopening kstars=>ekos and reconnecting the devices brings up the "another instance of Indi found... closing in 5" dialog box). A quick scan through them and last entries tend to be

[2022-03-17T22:50:46.170 GMT INFO ][           org.kde.kstars.indi] - Canon DSLR EOS 550D :  "[INFO] Starting 60 seconds exposure. "

or

[2022-03-19T00:18:57.957 GMT INFO ][     org.kde.kstars.ekos.guide] - "Dithering in progress."

So not really sure what's off.
I have noted plenty of lines (latest one had ~400 lines out of 2500) with occurrences of:

[                       default] - WARNING: Phonon::createPath: Cannot connect  Phonon::MediaObject ( no objectName ) to  Phonon::AudioOutput ( no objectName ).

but due to the number of these (and that they occur from just after autoguiding starts), I'm assuming that's not one to chase down (just yet)?

Last night, I thought I'd try starting via terminal, so 'kstars' at the prompt, and hoped either this would give more help on what the cause may be, or perhaps make it run more stable. Last line in the terminal is the highly descriptive linux error, "Segmentation fault"! 

I've tried searching for similar issues in the Indi forum, and whilst there are reports of folks getting miffed with crashes (consequently moving back to Windows & ascom), they are few and far between so hope there is a solution.
Anyone experienced similar, know how to get around the issue, or is this one to post on the Indi forums? Could it be a connection (eg USB) or RAM issue? Although nothing extra is running on the RPi which is mounted to the dovetail under the telescope, so not used for any other purposes. Left outside, it has been cold?

Log files of the crashed sessions over the w/e attached.
Thanks in advance.

logs.zip

[astrojon77]

I don’t have any advice for you at the moment but just wanted to add that I’ve been seeing image download failures and crashes with KStars/Ekos running on a Raspberry Pi. 
 

Sometimes the whole of Kstars crashes, just leaving PHD happily guiding. Other times, the image download freezes and I Ekos stops responding. 
 

I think (hope) it’s a software issue as usually a reboot of the RPi will be enough. But it’s very frustrating to have an imaging session fail.

I will have a look at my logs. 

[Realtimedoctor]

Hi there, 

I've been using ekos/kstars for about two years now. Only over the last couple of months, I've been seeing more an more frustrating nights, with random crashes just like you. 

The mount continues to run whole ekos crashes, and when I try to restart it gives me the same prompt that another instance is already running. 

Sometimes alos having issues with ekos not playing well with the camera, and the images won't download. After much faf, I narrowed it down to EKOS at times consuming too much ram and hence ekos just crashes ( I don't know why). The work around which has worked for me in the meantime is putting kstars into "low resource mode". That seems to have stopped crashes, and everything runs smoothly if left untouched, most of the time. 

This is one of the reasons why I'm upgrading my pi to a dedicated mini pc which will run u Ubuntu, and indi. Having more ram and compute power, I hope these issues will be gone. 

I'd suggest trying to see if you have low resource mode checked or not. 

Cheers, 

Nish

[pete_81]

Thanks Nish and Jonathan,
I've seen a couple of things suggesting it was a RAM issue, with resource limiting mode solving the issue, (indi forum), but as it was 3 years ago (and on RPi3), would have thought such an issue would have disappeared.
Can others shed any light on things (and if so, then we can all ask for ekos/indi/kstars folks to look into it more).
Rather frustrating that the idea is to automate things and I'm then wondering if I'm wasting imaging time! And the amount spent on RPi and cabling to get nice setup and things don't work as intended! Not quite as frustrating as the cloud cover though!😂

Others seeing similar?

[astrojon77]

From a bit of a searching lots of people seem to make suggests regarding USB power / hubs and number of USB ports in use, or limiting the USB rate. I think I’ll experiment along those lines. 

[Realtimedoctor]

Just to add an update. 

I setup my new pc with dual boot into Linux, and running kstars last night, I had zero issues. Everything works way too faster and smoother than the pi could achieve. 

Maybe was just lucky, but yesterday was the first full night of imaging without any hiccups whatsoever. 

I'm though, thinking of giving NINA a try, since now I've a Windows PC at the mount. 

Cheers,

Nish 

[pete_81]

 

 

Haven't delved deep enough into things, but I know the resource limiter was enabled with the FITS viewer in KStars settings.
I've disabled all the FITS viewer data as it doesn't do anything for me other than show hopefully that the image is correctly being captured (eg DSLR in 'M' rather than 'Av' (forgetting to reset after flats!)) or some other issue. Generally I'd download images to MBP after starting to see how things look anyway - and I've got RawTherapee on the RPi so could do preview of that to see what the image looks like rather than letting EKOS/FITSviewer do the work.

I tried to do the EKOS debugger version of kstars, but the adding to APT ppa

sudo apt-add-repository ppa:mutlaqja/ppa

got the error:

aptsources.distro.NoDistroTemplateException: Error: could not find a distribution template for Raspbian/buster

My Kstars build is currently:

3.5.7, Build: 2022-01-29T12:50:57Z

I've just thought that I can monitor RAM with a little script similar to the temperature monitoring one that's recommended on most of the RPi forums.
So written that, it'll output a text file once started and just log time, temp, CPU & RAM. That way, I can track possible crashes to how much RAM is used, and compare with timestamps in the KStars log. Might show nothing, no harm trying!
Script and output:

#!/usr/bin/env bash
set -eu -o pipefail
SECONDS=${1:-36000}

current_cpu_temp_in_c() {
	/opt/vc/bin/vcgencmd measure_temp | cut -d"t" -f2 | cut -d"'" -f1
}
RAM_Summary() {
	free -h | head -n 2 | cut -b 1-6,15-20,27-33,71-79	# Total,Used,Available...
}
RAM_Available() {
	free -h | tail -n 2 | head -n 1 | cut -b 75-79
}
RAM_Used() {
	free -h | tail -n 2 | head -n 1 | cut -b 27-31
}

echo "New Data Starting Here" >> "/home/astroberry/Astro Imaging/SystemLog.txt"
for i in `seq 1 $SECONDS`; do
	NOW=$(date '+%Y-%m-%d %H:%M:%S,')
#	echo $NOW "T$(current_cpu_temp_in_c)ºC. CPU:$(actual_cpu_clock_speed). RAM available: $(RAM_Available); RAM used: $(RAM_Used)" >> "/home/astroberry/Astro Imaging/SystemLog.txt"
	echo $NOW "T$(current_cpu_temp_in_c)ºC. RAM available: $(RAM_Available); RAM used: $(RAM_Used)" >> "/home/astroberry/Astro Imaging/SystemLog.txt"
	sleep 1
done

New Data Starting Here
2022-03-23 17:55:05, Temp=41.3ºC. RAM available: 3.2Gi; RAM used: 323Mi
2022-03-23 17:55:06, Temp=40.9ºC. RAM available: 3.2Gi; RAM used: 324Mi
2022-03-23 17:55:07, Temp=40.9ºC. RAM available: 3.2Gi; RAM used: 324Mi
2022-03-23 17:55:08, Temp=40.9ºC. RAM available: 3.2Gi; RAM used: 323Mi
2022-03-23 17:55:09, Temp=40.4ºC. RAM available: 3.2Gi; RAM used: 324Mi
2022-03-23 17:55:10, Temp=40.9ºC. RAM available: 3.2Gi; RAM used: 324Mi
2022-03-23 17:55:11, Temp=40.9ºC. RAM available: 3.2Gi; RAM used: 325Mi

Simple, but might shed light on some issues should they occur.
Must remember to start the script from the terminal so a CTRL+C will stop writing to it! Then it just appends rather than overwriting if started again.

Tonight is meant to be clear again so might get it running and leave to see if disabling all the FITS options has worked, or if a crash occurs, if it's related to a memory issue or not. Time will tell!

[Rallemikken]

Go here -- https://www.astroberry.io/docs/index.php?title=Astroberry_Server#FAQ  Scroll down to "Advanced installation". Not sure, but I think Jasem's PPA is for Ubuntu/Mint only. If you have Raspian on your Pi, then it is Debian-based. This repo is for Buster, but I have used it on Bullseye. Some additinal buster packages must be installed if you use Bullseye (which is the latest). Check out http://www.agle.no/astro/kommeigang.html for some hidden clues.

[pete_81]

Coming back to this, with over a week since last imaging session.
Might look into the installation of Bullseye, but for the moment, I'm obviously interested to see why astroberry has such issues as it's meant for this purpose...
Same thing as before with the crashing.

So I'd disabled all FITS options so that no images are displayed after capture, and also ran my custom script to track memory issues. My findings are non-conclusive as there seems to be plenty of memory available - the KStars log just ends after a 60sec exposure started, and pretty obvious from the tracking output when KStars crashes:

 

[2022-03-26T00:33:03.451 GMT INFO ][           org.kde.kstars.indi] - "CR2" file saved to "/home/astroberry/Astro Imaging/Light/UV-IR/M82_Light_UV-IR_60_secs_2022-03-26T00-33-03_077.cr2"
[2022-03-26T00:33:03.452 GMT INFO ][   org.kde.kstars.ekos.capture] - "Received image 77 out of 400."
[2022-03-26T00:33:03.478 GMT DEBG ][                       default] - WARNING: Phonon::createPath: Cannot connect  Phonon::MediaObject ( no objectName ) to  Phonon::AudioOutput ( no objectName ).
[2022-03-26T00:33:03.487 GMT INFO ][   org.kde.kstars.ekos.capture] - "Capturing 60.000-second  image..."
[2022-03-26T00:33:03.500 GMT INFO ][           org.kde.kstars.indi] - Canon DSLR EOS 550D :  "[INFO] Starting 60 seconds exposure. "

 

2022-03-26 00:33:55, Temp=25.3ºC. RAM available: 2.5GB; RAM used: 1.1GB.
2022-03-26 00:33:56, Temp=25.3ºC. RAM available: 2.5GB; RAM used: 1.1GB.
2022-03-26 00:33:57, Temp=25.3ºC. RAM available: 2.5GB; RAM used: 1.1GB.
2022-03-26 00:33:58, Temp=23.8ºC. RAM available: 2.5GB; RAM used: 1.1GB.
2022-03-26 00:33:59, Temp=25.8ºC. RAM available: 2.5GB; RAM used: 1.1GB.
2022-03-26 00:34:00, Temp=25.8ºC. RAM available: 2.5GB; RAM used: 1.1GB.
2022-03-26 00:34:01, Temp=25.3ºC. RAM available: 2.4GB; RAM used: 1.1GB.
2022-03-26 00:34:02, Temp=25.8ºC. RAM available: 2.5GB; RAM used: 1.1GB.
2022-03-26 00:34:03, Temp=26.2ºC. RAM available: 2.5GB; RAM used: 1.0GB.
2022-03-26 00:34:04, Temp=25.3ºC. RAM available: 2.5GB; RAM used: 1.0GB.
2022-03-26 00:34:05, Temp=25.8ºC. RAM available: 3.1GB; RAM used: 417MB.
2022-03-26 00:34:06, Temp=26.2ºC. RAM available: 3.1GB; RAM used: 442MB.
2022-03-26 00:34:07, Temp=25.3ºC. RAM available: 3.1GB; RAM used: 429MB.
2022-03-26 00:34:08, Temp=24.8ºC. RAM available: 3.1GB; RAM used: 429MB.
2022-03-26 00:34:09, Temp=25.3ºC. RAM available: 3.1GB; RAM used: 429MB.
2022-03-26 00:34:10, Temp=24.8ºC. RAM available: 3.1GB; RAM used: 429MB.
2022-03-26 00:34:12, Temp=24.3ºC. RAM available: 3.1GB; RAM used: 429MB.
2022-03-26 00:34:13, Temp=24.3ºC. RAM available: 3.1GB; RAM used: 429MB.
2022-03-26 00:34:14, Temp=25.8ºC. RAM available: 3.1GB; RAM used: 429MB.

It seems to crash 1 min after the exposure starts, so somewhere around the attempted download of the ~22MB image CR2 fille.

What's frustrating more is that I closed my VNC at around a quarter past midnight, after starting the session at 8:30pm, so lost all night from midnight to ~4am having had almost 4hrs of successful imaging up to that point.
Is anyone else having such issues or frustrations?

What are the alternatives for RPi (my installation was with astroberry and blanenaEtcher, then adding conky and anydesk; other than that, its use is for imaging)?
Thanks again folks

[Avocette]

I use the ‘plain vanilla’ Astroberry installation on two RPis simultaneously. I had suffered very occasionally from issues like  KStars 3.5.6 crashing, but much more so when I updated one setup a month ago to 3.5.7, so I went back straight back to 3.5.6 (just copied the old SSD image). Now since Radek announced he had updated his repo to KStars 3.5.8 with INDI 1.9.5 and PHD 2.6.11, I have started running this version and things are going well. I note in Jasem’s Ecosphere blog that there were a lot of bug fixes including for such things as memory leaks.

[pete_81]

Follow up with bug report after simulator crashes:

Github

And more responses from the follow up on Indi.

Edited May 20, 2022 by pete_81