Darks and Temperature

[AlistairW]

Hello,

Not sure of the way round this, but after 2.5 hours of Lights imaging, I then start to take my Darks. But as time has passed so the temperature has dropped, so the temps of the Lights and Darks are different. Also should I be having a digital thermometer to be exact about this, or is near enough on the temperature good enough ?

Thanks

Alistair

[michael.h.f.wilkinson]

Don't the EXIF data of your image contain sensor temperature? No need for an external thermometer, I would say. I seem to remember Deep Sky Stacker could handle variations in temperature in the darks

[Knight of Clear Skies]

That's a difficult question to answer. Some users have reported good noise reduction simply taking the darks later while others have had problems. Here's a recent thread with an example of it working well if you read down to post #20. Darks become more important with longer sub times and at higher sensor temperatures.

Personally, I use dithering, dark bias frames and kapp-sigma clipping when stacking to reduce noise. I'm going to experiment with taking darks at some point.

[Knight of Clear Skies]

Don't the EXIF data of your image contain sensor temperature? No need for an external thermometer, I would say. I seem to remember Deep Sky Stacker could handle variations in temperature in the darks

I'm not sure, but I've heard some people say the temperature in the EXIF data can be unreliable.

[michael.h.f.wilkinson]

I'm not sure, but I've heard some people say the temperature in the EXIF data can be unreliable.

Quite possible. I am not sure a separate digital thermometer would be better. After all the sensor is in an enclosure with all sorts of electronics circuitry generating heat

[IanL]

Have a read of this:

http://www.blackwaterskies.co.uk/2015/02/pixinsight-dslr-workflow-part-2a-dark.html

Sorry I haven't got as far as writing part 3 yet, but this article gives you an accurate method of determining whether you can rely on the EXIF temperature:

- I'd certainly measure it for a given camera model rather than relying on guesswork. If you find a good correlation between noise in the darks and the EXIF temperature, then you can use EXIFs to match dark frames together to make a master dark from individual darks of the same exposure and similar temperatures.  If you don't find a correlation, then measure the noise in all your darks and use that to match darks together to build a good master dark.

- The main problem for Canon users is that the RAW image is processed on-camera, causing the median brightness of the image to reduce as the dark temperature/exposure length goes up, which is the opposite of what should happen.  If you measure the noise in the dark, the correct relationship is found (noise increases with temperature and/or exposure length).  This means that you can't use EXIF temperature to match dark frames to light frames (and noise measurements from darks and lights aren't generally comparable due to being unable to separate the noise from the signal in the lights).

- For Canon cameras, if you use a simplistic approach to stacking (e.g. DSS) that relies on exposure lengths or temperatures, if you get lucky your darks improve your lights, and if you are unlucky you'll actually end up adding more noise!  Note this isn't the case for CCDs (and possibly not for other DSLRs but I have no info on that) as the CCD produces an unprocessed image straight from the sensor.

- The solution is to use PixInsight's ImageIntegration process.  This uses purely numerical methods to try to determine a scaling factor to apply to a master dark frame in order to make it match a light frame.  If the dark can't be scaled to match then it won't be used (so the image isn't made any worse), or if it can be scaled you can be sure that you are getting the best result possible with that combination of dark and light.  (In which case your only problem is to make the best possible dark in the first place as described above).

[michael.h.f.wilkinson]

Very interesting. Thanks!

[AlistairW]

Thanks for that. Having a read now - vey interesting.

[glowingturnip]

The solution is to use PixInsight's ImageIntegration process.  This uses purely numerical methods to try to determine a scaling factor to apply to a master dark frame in order to make it match a light frame.

does that mean that in the batch preprocess script, I should have 'optimise dark frames' ticked on ?

[IanL]

does that mean that in the batch preprocess script, I should have 'optimise dark frames' ticked on ?

Yes definitely - if unticked the master dark will always be applied as is with no scaling regardless of whether that does more harm or good.  If ticked then the optimisation process will try to scale the dark up or down, and won't use it if there is no match (that is what the "Warning: No correlation between the master dark and target frames" message on the console means: you can use the resulting calibrated frames, just that no dark has been subtracted because it would make things worse).

[glowingturnip]

interesting. 

I'd run into problems before with the batch preprocess script when I was loading up all the bias, darks, flats and lights and let it chug along all in one go - I found in the end result I was getting really very noisy lights (much noisier than any single component with lots of hot single-colour pixels).  After a long process of elimination, leaving out various steps, I found that what was happening was that when I ran the darks and flats together, the dark noise was getting subtracted from the flats to produce a master flat and of course that dark noise isn't in the flats, so once that dark-subtracted flat is then multiplicatively removed from the lights, that noise pops out horribly.

I currently do my preprocess as two stages - run just the bias and flats together, so the flats are only bias subtracted, then run those two masters with the darks and lights, and the end results are much better.

I think at some stage in my process of elimination I'd decided that the 'optimise dark frames' was exacerbating the problem, but I'll try it again now, with my two-stage process.

Cheers

[IanL]

I don't use the BPP script myself, I find it easier to just run each process manually so I don't know how you'd apply or not apply darks to flats, etc. but agree you should only use a bias with flats assuming they are relatively short exposures.  Bear in mind that bias frames are also be pre-processed on camera but I haven't measured the effects since there would be little by way of dark current noise or dark current to start with.  It might be an idea to experiment with using the bias as a master dark for the flats with optimisation on to see if you get a cleaner result that just a simple bias subtraction.

[glowingturnip]

Interesting articles Ian, looking forward to the next instalments.

I did a quick experiment on some of my darks, and get a clearly increasing relationship between EXIF temperature and MAD, but also interestingly an increasing relationship between temperature and Median (a Canon 1100d).  Too small a sample though, I need to redo on a bigger set (I delete my darks for space after I've made a master)

(MAD's on rhs axis)

Yes, my most recent set is at a blistering 34 degrees !

I also made myself a superbias following your instructions:

bias

superbias

difference

Looks pretty good - I actually used a custom blend between the 6 and 7 multiscale layers - 6 brought out the lighter top edge better but showed more of the noise, so used a combination between the two.

How often would you say I should redo this superbias ?

Cheers

[IanL]

Interesting articles Ian, looking forward to the next instalments.

I did a quick experiment on some of my darks, and get a clearly increasing relationship between EXIF temperature and MAD, but also interestingly an increasing relationship between temperature and Median (a Canon 1100d).  Too small a sample though, I need to redo on a bigger set (I delete my darks for space after I've made a master)

temp_vs_median_vs_MAD.JPG

(MAD's on rhs axis)

Yes, my most recent set is at a blistering 34 degrees !

I also made myself a superbias following your instructions:

bias.JPG

bias

superbias.JPG

superbias

diff.JPG

difference

Looks pretty good - I actually used a custom blend between the 6 and 7 multiscale layers - 6 brought out the lighter top edge better but showed more of the noise, so used a combination between the two.

How often would you say I should redo this superbias ?

Cheers

If you read Craig Stark's stuff, he found there was a point where his camera "turned the corner" and the median brightness started increasing again with longer exposures (assume it would be the same for temperatures). Maybe that is what you are seeing? I haven't fount that with my tests but then I was trying to keep temperatures down to what one typically gets during night time imaging in winter.

With regard to redoing library files, I don't honestly know as I've not tested it. I wouldn't have thought that it was necessary to do them frequently but someone may have better evidence.