Jocular: a tool for astronomical observing with a camera

[Steve in Boulder]

25 minutes ago, Martin Meredith said:

The new release of Jocular is now available. 

This is a major new version which supports ASI cameras natively across the 3 major operating systems. ROI/binning are supported for ASI cams, and the calibration library applies automatically to ROI-ed images. There are changes to the GUI to incorporate material design widgets, silent logging, easier installation and a host of other small changes including being able to change the location of the watched directory, though nothing major in terms of processing or object databases. Oh, and it has (annoying) tooltips for which the most important information is probably how to switch them off (conf/Appearance...).

For further details, see:

https://transpy.eu.pythonanywhere.com/jocular/

The download (~29M) now comes bundled with a couple of example captures.

I've developed and tested this on OSX using an ASI 290MM. I haven't been able to test on Windows but in theory it ought to work.

Please let me know (I'm sure you will!) of any issues 🙂

Martin

PS There is also (highly) experimental native support for the SX Ultrastar which I haven't been able to test, so any feedback from Ultrastar users on Macs would be appreciated (pretty sure this won't work on Windows but who's to say). Similarly untested is ASCOM filterwheel control. I'm planning to support ASI filterwheels in the near-ish future.

Really looking forward to trying this out!  Tooltips are a good idea for new users.  I think I’ll use the native ASI camera support for my calibration frames, though not for the lights for now as I’d rather sit inside (I.e. wireless) with my laptop during the winter.  Related question:  if I do use the native support for calibration frames, but with lights taken separately and going to a watched folder, can I still use the ROI feature?

[Martin Meredith]

Hi Steve

Not at present but if I can implement it I will. One problem is that for various reasons I'm determining ROI during the framing stage, so those captures would need to be done in Jocular. What I want to do is further rationalise the whole binning/ROI stuff so that it is camera-independent (at which point it will also apply to 'watched' captures).

By the way, the watched folder functionality has changed a little in ways that I will try to document before the month is out. 

I've only been able to properly test the darks so far so it will be interesting to see how you get on with flats. There is nascent support for choosing between bias and flat-darks, and a placeholder for using a constant a la Siril too. I tend not to use flats myself but when I do I use twilight flats, and I just haven't been around at twilight recently... 

Good luck!

Martin

 

 

[Steve in Boulder]

23 minutes ago, Martin Meredith said:

Hi Steve

Not at present but if I can implement it I will. One problem is that for various reasons I'm determining ROI during the framing stage, so those captures would need to be done in Jocular. What I want to do is further rationalise the whole binning/ROI stuff so that it is camera-independent (at which point it will also apply to 'watched' captures).

By the way, the watched folder functionality has changed a little in ways that I will try to document before the month is out. 

I've only been able to properly test the darks so far so it will be interesting to see how you get on with flats. There is nascent support for choosing between bias and flat-darks, and a placeholder for using a constant a la Siril too. I tend not to use flats myself but when I do I use twilight flats, and I just haven't been around at twilight recently... 

Good luck!

Martin

 

 

Thanks, Martin!  That makes sense.  A related question: If I am going to bin 2 on my lights, do I have to bin 2 when I take the calibration frames?

[Martin Meredith]

At the moment, yes, but the plan is to remove that restriction. I didn't want to do it in this release without thinking through any possible downsides. I think if hot pixels are properly dealt with then it will be OK to do this.

Since getting a CMOS camera I have been somewhat overwhelmed by the potential need for multiple calibration frames, so anything that can be done to redeploy them will be done!

What's your feeling about offset differences between calibration frames and lights if all the other parameters match up?

[Steve in Boulder]

11 minutes ago, Martin Meredith said:

At the moment, yes, but the plan is to remove that restriction. I didn't want to do it in this release without thinking through any possible downsides. I think if hot pixels are properly dealt with then it will be OK to do this.

Since getting a CMOS camera I have been somewhat overwhelmed by the potential need for multiple calibration frames, so anything that can be done to redeploy them will be done!

What's your feeling about offset differences between calibration frames and lights if all the other parameters match up?

I don't have anything useful  or intelligent to say on that -- I've only used the ASIAir for imaging so far, and it hides offset settings.  The Live view seems to work well with the calibration frames I take using the ASIAir, but I don't know what offsets are used. 

[Steve in Boulder]

3 hours ago, Martin Meredith said:

The new release of Jocular is now available. 

This is a major new version which supports ASI cameras natively across the 3 major operating systems. ROI/binning are supported for ASI cams, and the calibration library applies automatically to ROI-ed images. There are changes to the GUI to incorporate material design widgets, silent logging, easier installation and a host of other small changes including being able to change the location of the watched directory, though nothing major in terms of processing or object databases. Oh, and it has (annoying) tooltips for which the most important information is probably how to switch them off (conf/Appearance...).

For further details, see:

https://transpy.eu.pythonanywhere.com/jocular/

The download (~29M) now comes bundled with a couple of example captures.

I've developed and tested this on OSX using an ASI 290MM. I haven't been able to test on Windows but in theory it ought to work.

Please let me know (I'm sure you will!) of any issues 🙂

Martin

PS There is also (highly) experimental native support for the SX Ultrastar which I haven't been able to test, so any feedback from Ultrastar users on Macs would be appreciated (pretty sure this won't work on Windows but who's to say). Similarly untested is ASCOM filterwheel control. I'm planning to support ASI filterwheels in the near-ish future.

How should I install the ASI Camera 2 Mac library to work with this?  I believe this is the library that is in the ASI Camera SDK, is that correct?

[Martin Meredith]

It comes with the distribution so you don't have to do anything (I hope!)

Specifically, it is this one that gets selected if you're on a Mac: libASICamera2.dylib.1.18

Edited January 27, 2022 by Martin Meredith

[Steve in Boulder]

3 minutes ago, Martin Meredith said:

It comes with the distribution so you don't have to do anything (I hope!)

When I try to connect to ASI cameras, I get "Cannot find ASI library in resources."  The Jocular resources directory just has .dlls, true type fonts, and JSON files. 

[Martin Meredith]

You are quite right. It looks like it didn't package those files, presumably as I didn't specify their extensions in the manifest. Give me a few mins to sort it.

[Martin Meredith]

Some updates after initial user testing:

* Windows users wishing to use ASI cams need to download the ASI drivers from here

* Mac users need to ensure that the libusb-1.0.0.dylib is present and findable on their system. I may bundle this in the next release.

* anyone monitoring directories on another physical device (eg rPi) will find that the FITs are copied rather than moved, so the same FITs will get sent over and over again.  This will be fixed soon.

These and a few other minor issues are now mentioned in the installation instructions on the website.

 

[AKB]

Martin

Great to hear this!  Re. Flats, they’re rather important for my Hyperstar.  In fact, I just took some new T-shirt flats during dusk today, and both dust bunnies and vignetting are dealt with really well.  This, of course, is using the watched folder for both flats, darks, and lights, so I’m hoping this is all OK.

My experiments with OSC and binning are currently on hold, and I suspect this will be affected by your planned changes. I use both Mac and Ultrastar, but PC for the capture on the scope.  However, when I swap out the camera from scope I’ll give the native Mac capture a go.

Onwards and upwards for Jocular!

Tony

[Martin Meredith]

11 hours ago, AKB said:

Martin

Great to hear this!  Re. Flats, they’re rather important for my Hyperstar.  In fact, I just took some new T-shirt flats during dusk today, and both dust bunnies and vignetting are dealt with really well.  This, of course, is using the watched folder for both flats, darks, and lights, so I’m hoping this is all OK.

My experiments with OSC and binning are currently on hold, and I suspect this will be affected by your planned changes. I use both Mac and Ultrastar, but PC for the capture on the scope.  However, when I swap out the camera from scope I’ll give the native Mac capture a go.

Onwards and upwards for Jocular!

Tony

Thanks Tony. I hope the watched functionality is still all OK too though it is challenging to cope with all the various FITs that are being thrown at it...

I don't expect the native Ultrastar to work first time but it will be interesting to hear of your experiences (and if you don't already have libusb you might need to build it using 'brew install libusb').

Re binning/OSC/ROI etc, the plan is to rationalise things so that these options can be applied to all captures ie in a uniform camera-independent way, which is not the case at present where, for example ROI is only available for ASI cams, and debayering is only in place for watched captures... It needs a little refactoring but is coming soon.

[AKB]

2 hours ago, Martin Meredith said:

I hope the watched functionality is still all OK

I used this command to upgrade...

 pip install --upgrade jocular==0.5.3

...not sure is that's correct?  Actually, I struggled to find the syntax to do this, it's not available on the help text?

Initial findings are:

• Flats don't seem to work – at least, what worked yesterday (which was new flats applying to previous day's lights) doesn't make any difference – calibrator says "none suitable"
• Nothing in the config about age of flats to allow?
• Ring dimming controls gone??
• I was slightly thrown by the lack of a CLOSE button on the configuration, but moving the cursor out of the ring does the trick
• imstats: would it be possible to add image size in pixels?  I find this useful when dealing with different cameras

I've been working on a computationally simple, but effective, post-stack-and-stretch, noise reduction method, with very forgiving parameters.  I'd like to test this on any images you might think are appropriate.   I wonder if there's any chance of a discussion on this?

Thanks for all your hard work.  Life (or, at least, observing sessions) just wouldn't be the same without Jocular.

Tony

PS: LOVE the pop-up message on saving an image.  Real confirmation that something has actually happened!

 

Edited January 29, 2022 by AKB

[Martin Meredith]

44 minutes ago, AKB said:

I used this command to upgrade...

 pip install --upgrade jocular==0.5.3

...not sure is that's correct?  Actually, I struggled to find the syntax to do this, it's not available on the help text?

Initial findings are:

• Flats don't seem to work – at least, what worked yesterday (which was new flats applying to previous day's lights) doesn't make any difference – calibrator says "none suitable"
• Nothing in the config about age of flats to allow?
• Ring dimming controls gone??
• I was slightly thrown by the lack of a CLOSE button on the configuration, but moving the cursor out of the ring does the trick
• imstats: would it be possible to add image size in pixels?  I find this useful when dealing with different cameras

I've been working on a computationally simple, but effective, post-stack-and-stretch, noise reduction method, with very forgiving parameters.  I'd like to test this on any images you might think are appropriate.   I wonder if there's any chance of a discussion on this?

Thanks for all your hard work.  Life (or, at least, observing sessions) just wouldn't be the same without Jocular.

Tony

PS: LOVE the pop-up message on saving an image.  Real confirmation that something has actually happened!

 

I'll add the upgrade to the web page but I had a feeling just doing pip install jocular might be enough to achieve that.

Could you PM me the log that you'll find in joculardata/logs please? That will help identify any issues with the flats. I don't use max age any longer because I don't think it is necessary -- what it does is find the closest in age if there is a choice of suitable flats. I use closest because you might be reloading images captured years ago... (in which case keep the flats...).

I moved the ring dimming control (and renamed it transparency) to conf/Appearance. If inconvenient it could be re-added, but I was cleaning up the main eyepiece based on my own usage...

The plan is to add rather more information in the imstats, including FITs headers (for subs/masters) and that will give you the pixel dims. By the way, in principle annotation now makes use of any pixel height and binning info in the FITs header so you don't need to provide anything other than focal length to the platesolver. For those with multiple cameras (or with a colelction of previous observations taken with different cameras), this saves messing about with pixel height.

Yes, definitely interested in discussing your noise reduction method (and happy to incorporate it into the code if appropriate).

One other (potentially temporary) simplification I made was to bad pixel mapping. This is now done on each sub and is really just hot pixel removal (no mapping). The BPM started to get rather cumbersome with multiple sensors and ROI. There is a new outlier rejection threshold (in sigmas) that controls what is considered a hot pixel. Let's see how this works and if necc. I will go back to building maps. (the key difference is that a hot pixel formerly had to be present on multiple consecutive subs to count as one).

Martin

 

[Martin Meredith]

I've uploaded v0.5.4 now. To upgrade, type

pip install --upgrade jocular

Main changes:

[1] I've updated the ASI drivers to the latest version and included both 32 and 64-bit Windows versions that are automatically selected (no need for the user to choose)

[2] Fixed an issue with cross-device watched directories (eg monitoring a directory being written to by a remote RPi)

[3] Subs are now represented internally as float32 rather than float64. This will mainly help those with limited RAM/no SSD.

[4] Fixed a few other minor bugs

BTW If you can't select the S or W controls or find any other kind of overlapped widgets, it is almost certainly due to the widgets in the DSO panel (top left) overlapping with these controls. This can occur on small screens. The solution is to close the DSO panel when selecting those widgets. I'll look into more creative placement options.

Thanks to all for your feedback so far!

[AKB]

Definite improvement!  Now I am able to load and save bias by dropping in the watched folder...

...BUT

the processed image doean't use the flats...

[Martin Meredith]

What do you see when you click on library? Has it created the masters? That's the first step to thing to check for.

[AKB]

Yes, I checked that.  Here's what's there...

[AKB]

I'm having trouble trying to connect to an ASI 120 on my Mac.  I installed oaCapture, and that works just fine.  But in Jocular, when selecting equip > Camera > ASI and clicking connect... I get the error cannot initialise camera.

Any advice on how to diagnose?

Otherwise, the latest version is working well for me!

Tony

[Martin Meredith]

Does the log specify anything (like missing libusb?). As mentioned here, it appears that some Mac users have to build libusb.

Martin

[AKB]

Ah yes, quite so...

 7Feb22 10:31:12.261 | ERROR    | jocular.cameras.asicamera | connect                    |   91 | problem initialising ASI (dlopen(/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/jocular/resources/mac/libASICamera2.dylib.1.21, 6): Library not loaded: @loader_path/libusb-1.0.0.dylib

However, I had previously followed instructions to build libusb.  But i've just redone them:

(base) LFB-iMac:jocular lfb$ brew install libusb
Running `brew update --preinstall`...
==> Auto-updated Homebrew!
Updated 2 taps (homebrew/core and homebrew/cask).
==> New Formulae
asyncapi                   esphome                    inotify-tools              rure                       tfschema
canfigger                  ffmpeg@4                   linode-cli                 terminalimageviewer        usbutils
==> Updated Formulae
Updated 440 formulae.
==> New Casks
arctype               imagex                inkstitch             nozbe-personal        polypane              trex
==> Updated Casks
Updated 327 casks.
==> Deleted Casks
jsui                       liya                       nozbe                      powerword                  telavox-flow

libusb 1.0.24 is already installed but outdated (so it will be upgraded).
==> Downloading https://ghcr.io/v2/homebrew/core/libusb/manifests/1.0.25
######################################################################## 100.0%
==> Downloading https://ghcr.io/v2/homebrew/core/libusb/blobs/sha256:742a3d523988790f967df5c944b802a8d8f536a99fab123e823acbd6b1ce4fde
==> Downloading from https://pkg-containers.githubusercontent.com/ghcr1/blobs/sha256:742a3d523988790f967df5c944b802a8d8f536a99fab123e
######################################################################## 100.0%
==> Upgrading libusb
  1.0.24 -> 1.0.25 

==> Pouring libusb--1.0.25.big_sur.bottle.tar.gz
🍺  /usr/local/Cellar/libusb/1.0.25: 22 files, 525.8KB
==> Running `brew cleanup libusb`...
Disable this behaviour by setting HOMEBREW_NO_INSTALL_CLEANUP.
Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).
Removing: /usr/local/Cellar/libusb/1.0.24... (22 files, 516.7KB)
Removing: /Users/lfb/Library/Caches/Homebrew/libusb--1.0.24... (159.7KB)
(base) LFB-iMac:jocular lfb$ 

and still see this in the log...

OSError: dlopen(/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/jocular/resources/mac/libASICamera2.dylib.1.21, 6): Library not loaded: @loader_path/libusb-1.0.0.dylib
  Referenced from: /Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/jocular/resources/mac/libASICamera2.dylib.1.21
  Reason: image not found
 7Feb22 10:38:18.129 | DEBUG    | jocular.objectio          | check_save                 |  146 | 
 7Feb22 10:38:18.129 | INFO     | jocular.jocular           | on_stop                    |  165 | normal close down
 7Feb22 10:38:18.130 | DEBUG    | jocular.component         | close                      |   81 | closedown
 7Feb22 10:38:18.130 | ERROR    | kivy.app                  | stop                       |  971 | An error has been caught in function 'stop', process 'MainProcess' (3598), thread 'MainThread' (4618382848):

 

Do I have to close the shell and log in again?

 

[AKB]

@Martin Meredith

Having very recently swapped EEVA cameras, I've run into a couple of problems with Jocular:

1. wrong-sized flats (from old camera) being applied to new images
2. plate solving fails for new images

This all seems to come down to image size / number of pixels, so for reference the cameras in question are:

• OLD: Ultrastar uncooled mono CCD, 6.4µm pixels, 1392 x 1040, 11.2mm diagonal
• NEW: ASI 294MM cooled mono CMOS, 2.3µm pixels, 8288 x 5644,23.2mm diagonal

Now, obviously, there's a huge difference here, and for 'normal' EEVA use, I'm binning the new camera x2 and using an ROI of 2048 x 1536, for a diagonal of 11.8mm.  But it is still twice as many pixels (which was the whole point, to improve sampling.)

Flats:

Seeing as the image pixel sizes are difference, I was very surprised to fins that somehow the flats from my old camera (which was recognisable as being fairly dusty) were being applied to the new image.  How is this happening?  I haven't yet tried to make flats for the new camera, but it's actually very clean.  It would only be to reduce vignetting.

Plate Solving:

I'm presuming that there's some sort of limit that I've reached here.  I experimented by cropping a new image slightly and dropping that into Jocular, and it solves just fine.  This is after having adjusted the pixel size and binning factor in the configuration menu.

 

I'm most concerned about the flats, although this may be solved by simply taking some new ones, but it's also an issue for plate solving and identifying small targets.  If necessary, I could perhaps work around this by reducing the ROI even further. I should just add that the ROI is done in the capture software (ASIImage) outside of Jocular.

Thanks for any suggestions.

Tony

[Martin Meredith]

Hi Tony

Ugh... of course, this shouldn't be happening. Pending a deeper look here's what I think is going on: since implementing ROI, when looking for appropriate calibration frames I no longer enforce a match in image dimensions but instead use a match in the 'camera' property (and check that the light is a subregion of the calib frame). However, the camera property is only set for ASI cams at present, so as a fallback I do allow matches for cameras (such as the Ultrastar) where the camera property is not set. Let me think about a fix...

Could you say more about the platesolving issue? Do you think the FOV is too large or too small?

Have you retired the poor Ultrastar?

Martin

[AKB]

Thanks for hugely fast response!

I've PM'd a log.

For platesolving, I think it just seems to be 'too many pixels' !  Although it's only about 3 million vs. the 1.3 of the Ultrastar.  Cutting it down to 1600 x 1200 seemed to work, although maybe it's an issue of centering the target in the FOV too?

The Ultrastar will live on, I'm sure, it's such a great camera, and so easy to use (no power lead is important when using a Schmidt configuration.)  But I was just curious to see how cooled CMOS compares to uncooled CCD, and even from  just last night with Saharan dust and poor transparency, I have to say that the noise levels seem to be spectacularly low when cooled to -10ºC.

Happy to continue this issue by PM.

[Steve in Boulder]

Martin, does LRGB mode work with the watched folder, and if so is there a naming convention for the files?