Raspberry Pi AllSky Camera

[Pete6]

49 minutes ago, jiberjaber said:

Yep - I've got those modifications running also from when you first mentioned it.  I just need to have a think about the best place to put the commands to pick up the weather data then I can simplify the code I have reading the files. I think if I run them every 2 mins it should grab at a sufficiently realtime rate for the camera annotation. 

 

If you want to run a job every 2 minutes, open up webcam and find the part where I write out the crontab file and stick a line or two in there.  There are plenty of examples in that code .  

echo "*/2 * * * * $HOME/myjob> /dev/null 2>&1">>$HOME/webcamcron

should do the trick.  That way it will get written to the crontab file each day when webcam runs itself and writes a new crontab file.

[jiberjaber]

Yep - thats what I have done, it calls a script every 2 mins which calls mosquitto_sub -C1 for each of the three topics I need to grab data from the server running them concurrently as one updates every 5 seconds and another updates ever 2 mins! These are piped to a tmp file which is then copied to the final file when teh mqtt stuff has finished so if the webcam updates there is always data there for it to put in the banner (other wise the pipe file just blanks till the data is pipped!) I added an additional file which has a timestamp in so I can see that teh mqtt script was successful quickly, but I will probably turn this off now I am happy it is working.

 

Edited May 23, 2020 by jiberjaber

[uhb1966]

Nice work! IMHO looks like you've got a lot of light polluton - are you doing narrowband?

Best, U.

[Pete6]

8 hours ago, uhb1966 said:

Nice work! IMHO looks like you've got a lot of light polluton - are you doing narrowband?

Best, U.

No this is not doing narrow band.  In fact I only just got the Raspberry Pi Cam V2 with its Sony IMX 219 sensor to gather enough light down the M12 mount to its 1/4 (I think()" sensor.  There us also not IR filter so in fact this is about as wide a band as I can make it.

My goal was to make a very cheap but usable AllSky Camera.  Using a Pi Zero W you can make a working setup for well under £40 not including the case.  Jiberjaber has used a drinks bottle for that.  So not too expensive.  If you skip using the temperature prob you can assemble the whole device with zero soldering and standard Raspberry Pi parts..

Shortly (within the next couple of weeks) I shall be installing one at my dark sky astronomy club site in (nearly) darkest South Carolina http://charlotteastronomers.org/  Hopefully I shall have a link that I can access from home that I can post here.  The club already has an AllSky Cam so if you look at the site you'll find it but that is not narrow band either.  This is not really a narrow band game.  The idea is to see the whole sky regardless of the weather.

Pete

Edited May 24, 2020 by Pete6

[SamAndrew]

I'm about to have a go at setting this up. New to the Pi and just got hold of the HQ camera. I see from discussion on the pi forum there is a patch that allows you to start the image capture in burst mode that means it only takes a couple of seconds on top of the exposure length to take the photo. 

I just did sudo rpi-update and then the following allows you to go straight into busrt mode, e.g. 25 second exposure at gain 10:

raspistill -t 10 -md 3 -bm -ex off -ag 10 -ss 25000000 -st -o Desktop/test8.jpg
Hopefully this means you can do much more frequent exposures if desired.

[Pete6]

Hi Sam,  This sounds pretty exciting to me.  I have no experience using the HQ camera although I have read all about it.  It seems a mighty camera.  It should work with AllSky Camera but given the foregoing, I ain't tested it so you are a (the?) pioneer here.  I will give you all the help I can which may turn out to be worth exactly what you pay for it...

One thing has emerged from testing and that is the Pi (even a 4) seems to take a fair while to completely clear itself after processing an image.  This is why we have had so much discussion about how often a Pi Zero can take an image at night.  This was also why I limited my image size to 800 x 600 pixels.  I hate wasting pixels but It just took too long.  There is a utility on the Pi called htop and that will show multiple instances of raspistill if it starts to run into trouble.  If this happens just increase the variables shotD (Day) or shotN (Night) which determine how often - in minutes - the Pi takes an image.  

In general, more frequent images = less time to process them, bigger images = more processing time needed.  Pretty obvious.  The HQ camera has a larger sensor and higher resolution with more pixels.  However it seems to be able to do at least some processing within the device so maybe it will all even out.  The Raspberry Pi forum is your friend here and member jamesh is the Raspberry Pi employee who seems to be the oracle of the cameras.

Please stay in touch here as this looks to be very interesting.

Kind regards

Pete

[jiberjaber]

So there's a bit of a problem at the moment, the movies are not quite right which I think is because of the output of Sunwait and the loss of astro dark here.  I suspect there needs to be an exception in the code to handle detecting the "--:--" and in that case using another time, perhaps Nautical? 

[Pete6]

Jason,  None of this should be happening.  Could you please either post here or PM me the /home/pi/webcam file.  I have no idea what has happened but I will find out. 

Actually if you need to post it on Dropbox or Google Drive let me have the link.

Thanks

Edited May 29, 2020 by Pete6

[jiberjaber]

2 hours ago, Pete6 said:

Jason,  None of this should be happening.  Could you please either post here or PM me the /home/pi/webcam file.  I have no idea what has happened but I will find out. 

Actually if you need to post it on Dropbox or Google Drive let me have the link.

Thanks

Hi Pete,

I think it's because at my location we now have no astro dark so when the script runs the Astro rise is "--:--" and this breaks the script (I've not looked in detail as I have some image processing to catch up on so this is just a theory   )

Here's the output from webcam:

pi@raspberrypi:/home/allsky $ sudo ./webcam
raspistill not running
N /usr/bin/raspistill -q 100 -ISO auto -co 70 -awb greyworld -n -ex night -ag 9.0 -dg 2.0 -ss 6000000 -w 800 -h 600 -o /run/shm/webcam.jpg

E V E N T                                      HH:MM OFFSET
---------------------------------------------  ----- ------
Astronomical Rise (18 degrees below horizon):  --:-- (Midnight sun) -00:10
Astronomical Rise (Real)                    :  --:-- (Midnight sun)
Nautical Rise (12 degrees below horizon)    :  02:46 -00:10
Nautical Rise (Real)                        :  02:56
Civil Rise (6 degrees below horizon)        :  03:52 -00:10
Civil Rise (Real)                           :  04:02
CAMERA switches from Night to Day           :  03:54 -00:08
VIDEO switches from Night to Day            :  03:59 -00:03
Sunrise (0 degrees below horizon)           :  04:45 00:00
Sunrise (Real)                              :  04:45
Sunset (0 degrees below horizon)            :  21:05 00:00
Sunset (Real)                               :  21:05
Civil Set (6 degrees below horizon)         :  21:48 -00:00
Civil Set (Real)                            :  21:48
CAMERA switches from Day to Night           :  21:57 -00:09
VIDEO switches from Day to Night            :  22:00 -00:12
Nautical Set (12 degrees below horizon)     :  22:54 -00:00
Nautical Set (Real)                         :  22:54
Astronomical Set (18 degrees below horizon) :  --:-- (Midnight sun) -00:00
Astronomical Set (Real)                     :  --:-- (Midnight sun)
Sat 30 May 01:51:55 BST 2020
pi@raspberrypi:/home/allsky $
 

 

webcam

[Pete6]

webcam uses Civil Dawn and Dusk times to switch modes.  The camera seemed happier with this.  You can select whichever Dawn / Dusk set you wish in lines 218 - 230 in the webcam file.  Currently this is set to Civil but it can be changed to Nautical or Astronomical.  I display these last two Dawn . Duck times only for information.  They do not do anything inside the code.  It's an astronomer's tool so I display all the Dawn / Dusk daily timings.

Only the Civil times are currently used to switch camera modes and videos..  Hence the Midnight sun text is just informational.

I changed one thing in your file.  At line 136 

twiSetVid=-00:15 # Offset advance or delay for the Video Day / Night switch.

I did this to give a 6 minute interval between the camera switching to night mode and the night video start.  This is how my system runs currently.

There is a text file in /home/allsky named daily_times.txt  This is a copy of the table that webcam puts out.  It is there purely for reference so that times are accessibly without re-running webcam.

Please try this as this is how my Pi Zero W is currently working.

Please let me know what happens.

Pete

Jason-webcam

[jiberjaber]

On 30/05/2020 at 14:49, Pete6 said:

webcam uses Civil Dawn and Dusk times to switch modes.  The camera seemed happier with this.  You can select whichever Dawn / Dusk set you wish in lines 218 - 230 in the webcam file.  Currently this is set to Civil but it can be changed to Nautical or Astronomical.  I display these last two Dawn . Duck times only for information.  They do not do anything inside the code.  It's an astronomer's tool so I display all the Dawn / Dusk daily timings.

Only the Civil times are currently used to switch camera modes and videos..  Hence the Midnight sun text is just informational.

I changed one thing in your file.  At line 136 

twiSetVid=-00:15 # Offset advance or delay for the Video Day / Night switch.

I did this to give a 6 minute interval between the camera switching to night mode and the night video start.  This is how my system runs currently.

There is a text file in /home/allsky named daily_times.txt  This is a copy of the table that webcam puts out.  It is there purely for reference so that times are accessibly without re-running webcam.

Please try this as this is how my Pi Zero W is currently working.

Please let me know what happens.

Pete

Jason-webcam 29.07 kB · 0 downloads

Hi Peter, thanks for that I've updated my version of the webcam script to cover the revised twiSetVid value but I don't think this is where the problem is at. 

There is still something wrong however.... It seems to effect the movie making for both day and night movies, no longer are the a combination of timed exposures, but seem to be stuck at a particular time. The movie is about the right length, so the concatmovie script is being run and adding a frame each time, the problem is it keeps adding the same frame each time - I can see this as I have the timestamp on the frames!

Why this is occuring I am not sure, if I manually run copypic and concatmovie, it works. So my suspicion is still around the other thing thats changes which is the times used as a result of no astro-dark. At present this is corrupting the webpages as the file it is stored in is now in a different format as posted above, I think the " --:-- (Midnight sun) -00:10" stuff is breaking the scripting somewhere (perhaps in the webcam script or the web pages)

 

[SamAndrew]

Ok finally managed to have a crack at it tonight; got too many projects on the go at the moment!

Have currently got it taking a night photo every minute. No luck with the videos though, it's not saving pictures to the night folder. Will trouble shoot that one tomorrow. Connected a temp sensor so will also see if that will display.

 

[SamAndrew]

Ok I can confirm using all 12 megapixels on the HQ camera is daft! sticking to 800 x 600  

[jiberjaber]

22 minutes ago, SamAndrew said:

Ok I can confirm using all 12 megapixels on the HQ camera is daft! sticking to 800 x 600  

Yes - I tried using the full res of the V2 camera and reverted to the lower res, plus the lower res supports 2x2 binning (not sure if it's automatic)

To debug the picture thing, try running the scripts manually, you might be suffering from what I had which is a low size image which still contained the correct image but Pete's code deletes any images below a certain size (110kb) to solve an issue he was experiencing, I commented out the deletion and mine has been fine since (well expect the current issue of the video not being created.)

If the image is not being deleted, there should be a copy of the picture in the /home/allsky/pics/night/ directory with a timestamp in the filename.

If you run the copypic.sh is should copy a picture from the /run/shm directory and rename it with a timestamp.  If this isn't happening, edit the file and comment out the line "/usr/bin/find . -name "webcam-*.jpg" -type 'f' -size -110k -delete" and re-run to test.  If you then get a file, check it is actually and OK image, if so, leave the comment in.

The same command is in a couple of the scripts, so you'll need to comment it out in all of them if the test solves your issue. so concatday.sh concatnight.sh newdaymovie.sh and newnightmovie.sh I think thats all but worth looking through just in case

 

 

[SamAndrew]

20 hours ago, jiberjaber said:

Yes - I tried using the full res of the V2 camera and reverted to the lower res, plus the lower res supports 2x2 binning (not sure if it's automatic)

To debug the picture thing, try running the scripts manually, you might be suffering from what I had which is a low size image which still contained the correct image but Pete's code deletes any images below a certain size (110kb) to solve an issue he was experiencing, I commented out the deletion and mine has been fine since (well expect the current issue of the video not being created.)

If the image is not being deleted, there should be a copy of the picture in the /home/allsky/pics/night/ directory with a timestamp in the filename.

If you run the copypic.sh is should copy a picture from the /run/shm directory and rename it with a timestamp.  If this isn't happening, edit the file and comment out the line "/usr/bin/find . -name "webcam-*.jpg" -type 'f' -size -110k -delete" and re-run to test.  If you then get a file, check it is actually and OK image, if so, leave the comment in.

The same command is in a couple of the scripts, so you'll need to comment it out in all of them if the test solves your issue. so concatday.sh concatnight.sh newdaymovie.sh and newnightmovie.sh I think thats all but worth looking through just in case

 

 

Thanks this helped, tried running copypic.sh and it was just an issue with permissions as I'd missed a step. Day video is now working.

Currently running on a pi4; will give my Zero a go, but think I may go with the 4 as I want to try and add some image analysis to work out if the sky is cloudy and then send a safety message to my observatory to trigger a shutdown.

I've got an enclosure, dome and power supply sorted; the sooner I can get this together the sooner I can liberate my ASI 120MC from my current all sky camera and use it for another project

[jiberjaber]

OK - so a while ago I adjusted my version of the code to use the buffer image rather than the plain webcam image for constructing the day/night videos. The reason for this is I wanted the timestamp and data in the video. This worked great - or so I thought...

Of late I have been having a problem where the video would be just made up of 1 frame repeated and I was struggling to understand why. 

Today it dawned on me, the 'buffer.jpg' is only generated when the live view web page is being viewed - if you go days without viewing the live view, the buffer.jpg is not updated, it doesn't need to in it's original use case as it was only for the web page display. As a result, my modification was correctly grabbing the buffer picture but the picture never changed as the live view wasn't being executed (no views) .

I need to work out how to run the php code to get the same effect of banner creation but on a regular schedule rather than on demand when teh live view web page is viewed.

 

[jiberjaber]

I see "Ralph" has fixed the issue with the suntimes text on the associated webpage - he's put a fix up in the issues section of github but I am unsure if it only fixes it for when we have no astrodark or not? Might need some code elsewhere to ensure the format is correct for both cases?

Also needs an edit to the webcam.php file to ensure correct text for sunset time. changing line 64 to 

Quote

$SSet=substr($mtimes,39,5); // was 24,5 for sunset, this fixes astrodark issue.

Before:

After:

 

I've also figured out how to get the banner text updated when no views have occured on the live video page, it's a bit of a brute force approach, I added a line to the crontab through the webcam script to execute webcam.php every 2 mins using wget.

Quote

# Generate the buffer.jpg file to ensure there is a banner version of the image available every 2 mins.
# this ensures that even with no visits to teh allsky live view page the banner is updated.
echo "# ***  U P D A T E   B A N N E R    T E X T   I M A G E   E V E R Y   2   M I N S.">>$HOME/webcamcron
echo "*/2 * * * * /usr/bin/weget -q -O temp.txt http://localhost/webcam.php > /dev/null 2>&1">>$HOME/webcamcron
echo "">>$HOME/webcamcron

 

 

[jiberjaber]

Aha - so I suspected the fix might only work for locations with no astrodark and I think I was correct, testing with my lat/long and Pete's location details (Pete has astrodark at the moment, I don't) revealed that the solution is to resolve teh issue at source where the sunwait data is generated in the webcam script. This I've tested and seems to work. So need to revert the webcam.php and suntime.php back to the original version and add the following in to the webcam script.  It basically checks for the presence of "--:--" which is associated with the 'Midnight Sun' text which screws up the formatting of all the rest of the code tht relies on the sunwait generated data.

From line 165, add the if...fi statement blocks to look like the following:

Quote

 

# Astronomical dawn and set. 18 degrees below horizon
AstroRise=$(sunwait list 1 astronomical rise offset $twiRise $lon $lat)
if [[ $AstroRise == *"--:--"* ]]; then
AstroRise='--:--'
fi

AstroRiseReal=$(sunwait list 1 astronomical rise $lon $lat)
if [[ $AstroRiseReal == *"--:--"* ]]; then
AstroRiseReal='--:--'
fi

AstroSetReal=$(sunwait list 1 astronomical set $lon $lat)
if [[ $AstroSetReal == *"--:--"* ]]; then
AstroSetReal='--:--'
fi

AstroSet=$(sunwait list 1 astronomical set offset $twiSet $lon $lat)
if [[ $AstroSet == *"--:--"* ]]; then
AstroSet='--:--'
fi

 

 

 

[jiberjaber]

Wow - this has been a busy day on this.  I thought a bit more about the use of the banner text version and decided it was better to not have the execution of the php in cron but to just execute it before the picture is copied, so to this end I removed the cron entry part listed above and modified the copypic.sh script.  There is now a true/false option so if you want to use the banner version of the image in the movies you can, if not, then it is ignored, if you want to give this a go, replace copypic.sh with the following:

 

Quote

 

# Filename: copypic.sh

#Set this true to use the banner text version of the image in movie creation
BannerVersion=true
#BannerVersion=false

if [ "$BannerVersion" = true ]; then
# Generate the buffer.jpg file to ensure there is a banner version of the image available.
/usr/bin/wget -q -O temp.txt http://localhost/webcam.php 
/bin/cp -a /dev/shm/buffer.jpg "/home/allsky/pics/webcam-$(date +'%s').jpg"
else
/bin/cp -a /run/shm/webcam.jpg "/home/allsky/pics/webcam-$(date +'%s').jpg"
fi

# Remove any files that are too small as they were grabbed at changeover that may be incomplete.
cd /home/allsky/pics
#/usr/bin/find . -name "webcam-*.jpg" -type 'f' -size -110k -delete
cd $HOME
# end

 

 

Edited June 6, 2020 by jiberjaber

[jiberjaber]

I suspect the amature waterproofing of my allskycamera coke bottle version hasn't past the test of truth! LOL Suspect I'll have to get a proper dome cover for it now... 

 

[Gina]

Welcome to the drenched ASC club!!

[jiberjaber]

43 minutes ago, Gina said:

Welcome to the drenched ASC club!!

Yep - will await what it looks like in the morning, might have to remove and dry it out, shame as I was happy about where I had got it working... must get round to consolidating all the bits I've done somewhere!

[Gina]

My lens is still drying out.  Still showing internal misting.

[jiberjaber]

5 hours ago, Gina said:

My lens is still drying out.  Still showing internal misting.

Mine looks like a very foggy day! I can just tell that there is sun in the sky but the roof is no longer visible - suspect I'll have to take it down and open it up to dry out... !

[Gina]

Yes, I think you will.  I would definitely advise it.