Starlightlive dark frame troubles

[jimthompson]

Hi all,

It is entirely possible I am not doing something correctly, but presently I am having trouble using dark frames in StarlightLive.  I can record the frames and save them just fine.  When applied they work very well for removing hot and warm pixels, but only when the software is set to "linear" under Image Modifiers.  When I try to use dark frames with one of the other two choices ( arcsinh & x^0.25), the colours in the image become all messed up.  I have reproduced this problem with both a Lodestar X2c and Ultrastar C.  Am I missing something?  I really like x^0.25 for objects with wide dynamic range so it would be nice to be able to use it with the dark frames.

Thanks,

Jim T.

[HiloDon]

Jim,

I and others have had problems with restored dark frames.  Paul is aware of it and is working on a fix, probably in v3.2.  The work around is to only use the dark frames before they are saved.  There is also a problem with green stars appearing from the defective pixel removal feature, especially with the Lodestar.  Solution right now is to use dark frames or stick with mono.

I'm sure Paul will have this resolved soon.

Don

[Paul81]

I will be looking into the saved dark frame issue upon my return from holidays and get a fix ready to go in V3.2 which will be available once the problem is fixed. Hopefully nothing too serious and is a quick fix :-)

Paul

[jimthompson]

Hi Don,

I am not restoring from a saved dark, I am taking fresh darks, saving it but then continuing to use the darks I just collected.  The resulting image is great when set to "linear" but not any of the other non-linear settings.  I can switch back and forth from linear to one of the others and what I see is consistent; linear works fine with darkframes but the others don't.  Is this the bug that you also observed?  I have not tried restoring a previously saved dark yet.  I've attached some sample images from a couple nights ago that illustrates what I am observing.  Image description is in the filenames, but basically the top one is a stack with no darks with X^0.5 scaling, the middle one is with about 30 darks and X^0.5 scaling, and the bottom one is with same darks and linear scaling.

cheers,

Jim T.

[HiloDon]

Hi Jim,

If I use fresh darks, I have not had a problem other than the ones I described above.  Are you using the defective pixel removal?

I would suggest that the next time out, save a fits file and post it here or PM it to me.  I can test it to see if I can determine what's going on.  If you happen to have a fits for the one above, please send it to me.  Make sure it's a stacked fits.

Did the color change as the image stacked?  I have had that happen on occasion for certain objects.  M8 for example.  I didn't get that increase in noise though.

Don

[jimthompson]

Hi Don,

I was using the DPR, but I have seen the same behavior without it on.  I will attempt to reproduce the problem again and save as FITS as you suggest (I have only PNG for the images posted above).  The colours in the mixed up image did not change as more were stacked.  We have clouds here for a couple days so I will perhaps try to reproduce indoors and will post when I have something.

Cheers,

Jim T.

[jimthompson]

I did some more testing a couple nights ago and was able to confirm that indeed the problem arises when one saves the darks.  I collected my darks and then simply continued observing and I was able to use the x^0.25 tone adjustment curve with no issues.  In the other instances where I observed the weird behaviour it was when I collected the darks then saved them before continuing to observe.

I eagerly await Paul's update.

Cheers,

Jim T.

[HiloDon]

Thanks for the update, Jim.

Did you just save them or did you save and restore them?  It might be important for Paul to know so he can troubleshoot the code.

Don

[Paul81]

Hopefully should be a simple one to resolve. I'm back from holiday this weekend so am planning on working on SL next week :-)

If I need any data will let you know - sounds like its an easy one to replicate though which is always a massive help!

[Paul81]

Hi Gents,

At the moment I am having trouble replicating this issue. Here is my understanding of what causes the problem:

1. Start the application and switch to dark frame acquisition mode.
2. Take a sequence of dark frames (8).
3. Save the master dark (default file name).
4. Reset the current master dark.
5. Restore the master dark from the saved file.
6. Switch to image acquisition, stack images (SUM, 5) and x^0.25 processing mode does not work correctly.

My initial thought was that something with the process of saving and restoring the dark corrupts the dark data so when the dark is subtracted from the light pixel data you get incorrect values which are then put forward to the image processing stage.

Using my test mode (which uses raw FITS files in lieu of a real camera so I can debug during the day) I tried the following a couple of times:

1. I have run the above sequence and then saved the resulting live stack to a FITS file. I have then repeated the sequence above but without steps 3,4 and 5 and saved the stacked FITS file. Apart from the timestamp the saved stacked FITS are identical. I have done this for CYGM (lodestar-C) data. I visually checked the linear, arcsinh and x^0.25 and they seemed to be working OK.

If the save / restore corrupted the dark data I would have expected the FITS files to show different pixel values.

 

To help diagnose the issue please can you:

1. Write down the exact steps you take which yield the issue (including the number of exposures etc). I will basically follow your instructions to try and replicate the issue so the more detail the better.
2. When doing #1, can you enable the export to raw FITS files in the image export tab such that all your darks and lights are saved to FITS. Once complete please can you send me the FITS data as I will use that when I follow your instructions.
3. Does the problem happen every time or is it only sometimes?

Thank you for the help :-)

Paul

[alphatripleplus]

Hi Paul,

I mentioned a very similar problem using restored darks in my post on H-alpha captures with the Lodestar X2 mono.

These are the  steps that I follow which have got me into trouble with restored darks - hopefully this might help. I won't be able to send you FITS data at the moment though, as I haven' t saved any. (Sorry):

1. Start application, set exposure to 15sec, for example, and switch to dark frame acquisition mode.

2. Take a sequence of dark frames (20 usually).

3. Save the master dark, but save and specify a specific file name.

4. Switch to image acquisition/start light exposures, stack images use mean stacking,  linear processing, adjust white level, black level, contrast sliders and (usually) export image as .png after stopping exposure.

5. Change exposure to 30sec - not sure if I always do this step and start taking lights, or just move on to step 6

6. Restore a previously named and saved 30sec dark as the new master dark, usually saved in a previous session or earlier in the current session.

7. Repeat step 4 - image acquisition with 30sec subs, and the contrast level slider will not widen the histogram, which stays a narrow spike. The black level,  white level and brightness sliders work.

 

I nearly always use linear processing mode, and have the problems mentioned in step 7. The main difference I see between my process and yours is that I explicitly change the exposure length between taking darks and using them, and I always name my saved dark files. Hope this helps.

 

 

 

[jimthompson]

Sorry I am late responding.  Paul, in your list of steps above, I only had to save the master dark for the strange behavior to happen.  I did not need to read the saved dark back in again.  So if I simply acquired the darks (usually 5 to 10) and switched back into image acquisition mode there was no problem, but if I saved the dark after acquiring it things are messed up.  It seems to be repeatable.  Note I am using the program under Windows 8.1 with my Lodestar X2c.

Regards,

Jim T.

[Paul81]

Thanks for the extra info chaps - I haven't bee able to replicate the issue as of yet. I have decided to release V3.2 as is and then carry on investigating in the mean time.

[Ain Soph Aur]

I am having the same issue as Jim, just the act of saving a Dark file causes the issue. I believe I had the same issue in 6.2 last night during my initial test, although I will try again this evening to be positive which version I was using.

[Paul81]

Hmmm.. Something weird is going on here, might be something timing related. I couldn't replicate the bug in the test mode (where I use FITS files instead of a real camera) so i'll try again with a real camera connected as that will change the timing and processor loading which will be different to the test mode (the test mode is awesome for testing image processing logic etc - I naturally thought this was something in the dark frame code but nothing stands out or shows up under test mode).

[Ain Soph Aur]

Definitely something weird  Unfortunately, last evening's session (with my remote setup) went so perfectly that I didn't have the heart to try saving any darks with 6.2 and screwing SLL session with an (un-saved) stack of 60s darks up.

[Ain Soph Aur]

I am stlll slacking on testing this. My remote setup went over 3 hours perfectly last night, added over 20 new RGB captures for the AL Planetary Nebula Program. I am in a habit of running a stack of 60s darks (without saving) and use it on all 60s and sub-60s exposures, so not being able to save darks is not too annoying,

On a side note, the FWHM stacking feature is amazing.

[Paul81]

Glad everything is working well! I should have some time this weekend to take a deeper look at the dark frames :-)