Dark Frames for DSLR

[PhotoGav]

I have been wondering: Does it really matter what temperature the darks are shot at for use with a set of DSLR subs?

The thing is, the warmer the sensor the more noise is generated. So, could it be true that if you use darks shot at a higher temperature than the subs, more noise will be 'removed' from the subs, but at least as much noise as is in the subs anyway. Would removing more noise than might be in a sub significantly affect the quality of the sub?

My ideal scenario is that: I would build up a library of room temperature darks shot at the different sub time lengths and ISOs to then call upon for each future project, thereby dispensing with the need to shoot a set of darks with each and every target.

Has anyone done any tests on this?

I look forward to hearing opinion and hopefully hearing that subtracting more noise than is in a sub is not a problem.......

No doubt I will have to dream on!

[PhotoGav]

Here's another idea:

The ambient temperature of those lovely winter nights for imaging appears to range somewhere between about -2C and 7C. My kitchen fridge is set to run at 5C, so would it be possible (or even sensible) to put the camera (an eos 60d) in an airtight container in the fridge and expose subs in there at around 5C?

Thoughts please!

[Bizibilder]

That's the best way to do it - build up a library of subs.  I did mine on cloudy nights by putting the camera in the garage and, using a timer, taking 30+ subs of each of the regular exposure times that I use (30sec, 2min and 5min - all at ISO 800).  As the year passes it does not take that much effort to take subs every 5°C from around -5° to +20°.  These will need repeating as your camera sensor will change slightly over time - mine get replaced about once a year and so far have worked very well.  Some folk do use the fridge but I have always found the garage adequate.

When you have a set just look at them and you will soon see that they really are essential - especially at the "warmer" temperatures!!

[PhotoGav]

Thanks for that Roger. Absolutely, darks are essential with a dslr!

Good idea with shooting them on cloudy nights across a range of ambient outdoor temps. At least something useful can then be done on the all too frequent cloudy nights.

Cheers,

Gav.

[Steve 1962]

Hi Gav

A library of darks is the way to go.

....and it is a problem subtracting more noise than exists in your image. This is because you would actually introduce noise into the over corrected image.

Steve

[PhotoGav]

Thanks Steve. The library has just been officially opened!

My answer from Jerry Lodriguss was:

"If you use darks that are hotter than the lights, and you do a straight subtraction, then you will be left with black holes where the hot pixels were because the hotter ones in the darks will be brighter than the ones in the lights, and will leave dark holes afterwards."

And we don't want black holes...!

[Stardust]

I was wondering how critical the temperature is as I set up for 2 hours worth this week and the temperature changed by 2 degrees in that time, the same would have applied to the lights.

I think the only thing I'm any good at so far is a dark frame library   

[majapala]

I always learn something useful here. Thanks guys.

[RikM]

I shot a couple of big sets of darks with the camera in the fridge. Although the fridge temperature was 2-8°C, the chip temperature never got bellow 17°C. Sat outside imaging at -3°C the chip temp was 11°C and on a summer evening can be 20°C+. The fridge darks still worked okay. I think if you aim to get within ±5°C of your light frame chip temp it will be no problem at all.

If you use a master dark with very much more noise than your light frames, when you subtract it you end up adding noise to your light frame. It is all a bit odd really.

[PhotoGav]

Thanks Rik. I think I will try the fridge method for my 5°C dark library and use cloudy nights for the other temps. I started the 10°C library this evening, but rain has stopped play...

I think the only thing I'm any good at so far is a dark frame library

Hee hee, that made me chuckle!

[IanL]

I am in the process of performing some reasonably comprehensive tests since every time I post an image with darks subtracted there are lots of differing opinions as to what is the most effective strategy. I haven't finished the work yet, but am writing a blog post on the subject which will bare all.  Some things I am fairly certain of already:

- The number of darks is more critical than anything else in terms of reducing noise.  I have made temperature matched sets of 'fridge darks' at temps between 9C and 13C.  A stack of 100+ darks of all these temps introduces less noise than making smaller stacks of 20-30 darks of each temp and subtracting from matching temp light frames.  This isn't really surprising since the noise component should be reasonably similar at these different temps, and the more you stack the better the signal (dark current) to noise (dark current noise) ratio you get.  To be honest the difference is fairly marginal though, you can see it through statistical analysis of the final light frame, but visually there is not much to choose.

- It was suggested to me that using a master bias would be just as good as a master dark for DSLR images.  I am satisfied that this is not true.  The master bias frame shows a clear pattern on the scale of one to two pixels, whereas the master dark has a pattern on a larger scale of 4 pixels plus.  You can definitely see the difference in the final light frame, as there are many more 'blobs' of chroma noise (mostly red, but some green and blue 'islands' of colour).

- It was then suggested that using a defect map / cosmetic correction process or similar might also be a substitute for a master dark.  Actually this does work quite well and again you can only really see the difference in the statistics, whereas visually the images look almost identical.  The main challenge was creating the defect map in the first place, as using PixInsight the best way to do it is to create one from a master dark.  On the up-side you then don't have to worry about matching temperatures to get repeatable results.

- Using the 'big dark' works best in PixInsight by enabling the dark scaling routine.  Unlike most stacking packages, this process doesn't try to match darks by exposure time or temperature, it simply takes a master dark  and determines a scaling factor which produces the least noise in the resulting calibrated sub.  You can create a master dark at whatever temperature you like, and of a reasonable exposure length (say five or ten minutes) and then just apply it to all your subs regardless of their temperature and exposure length.  This works pretty well and also seems to overcome the other DSLR problem which is the on-camera processing that reduces the apparent dark current (but not the dark noise).

Finally it is important to appreciate that you cannot subtract noise from an image.  Noise is (by definition) random, and if you have a light frame containing noise, any time you subtract a bias, dark or flat frame that also contains noise, you will always add the extra noise to the image, not subtract it.  For the three types of calibration frame:

- Bias - you are removing a fixed pattern which appears in all frames due to the sensor.  For my Canon 500D this is a clear pattern of broken vertical stripes on a scale of 1 pixel, with the brightness increasing noticeably towards the bottom of the frame.  There is also a component of random noise (mostly generated by the readout electronics).

- Darks - you are removing dark current, i.e. any signal which is arise from thermal processes in the camera.  As noted above, for the 500D there is a clear pattern of 'blobs' on a scale of 4 pixels plus across the image.  Again there is a component of dark current noise from electrons being displaced in to the wells in a random manner due to heat in the sensor.  Darks also contain the bias pattern plus the readout noise.

- Flats - you are adjusting for differences in sensitivity in the imaging train (vignetting from the scope/lens, shadows from dust and fibres, differences in transmission through the individual bayer-matrix filter elements, differences in sensitivity from the actual sensor elements).  The flats will contain all of fixed pattern, readout noise, dark current and dark current noise.  In this case you apply a division process rather than subtraction, but it still adds the noise to the calibrated sub.

The trouble is that people erroneously refer to fixed pattern and dark current as 'noise' when they are not.  They are unwanted signal, since they are repeatable and measurable, whereas noise is (by definition) random.  The problem for DSLR users and darks is that it is hard to create identical conditions so that you can properly measure that unwanted signal (the dark current) and subtract it.

In all cases the trick is to stack multiple frames as this increases the signal (whatever it is you are trying to remove) more than the noise as follows:

ΔSNR = √ n - √ (n -1)

Where n is the number of frames stacked and ΔSNR is the improvement in signal to noise ratio.  You can see how this plots out in my blog post here:

http://www.blackwaterskies.co.uk/2013/09/pixinsight-dslr-workflow-part-1-bias.html

Most of the improvement is to be had as you go from one frame to 50 frames in a stack.  Typically I aim for about 100 frames if it is something that I can do with little effort and time, but 30 would be a reasonable stopping point if you don't have time.

[PhotoGav]

Ian, that's superb, thank you for all that info, just what I was hoping somebody would come up with! Some interesting conclusions that point towards it being possible to obtain acceptable DSLR images using a library of darks rather than session specific darks.

I wonder what the acceptable latitude of temperature difference between light and dark is - would a five degree difference be acceptable? So a light shot at ambient temp of 0C could use a large stack of darks shot at 5C?

On a related but seperate matter - I presume that it is fine to use one large master stack of bias frames for all lights... These are not temperature dependent, is that correct?

So, the goal of shooting just lights and flats for a target to produce an acceptable final image might be acheivable without the expense of a CCD camera?!

[RikM]

Smashing post Ian. It also supports my thoughts on my own DSLR images, that I get noticeably better results from stacks of 30-35 than from 20's. I use a 1000D and find that 35x 8min, and the same number of 'fridge darks' normally gives an image I am happy with. I do also flats and bias and tend to use 80x each since they are so quick to take.

[PhotoGav]

Rik - what temp difference between lights and darks do you find you can get away with? What temp are the fridge darks shot at? 5C? Also, can you use a single master stack of bias frames for all subs?

[Steve 1962]

 So, the goal of shooting just lights and flats for a target to produce an acceptable final image might be acheivable without the expense of a CCD camera?!

That's an excellent post by Ian.

Something which I think you also need to factor in are the benefits of "dithering" your images. This involves moving the scope a couple of pixels in a random direct between each sub (which most dedicated AP software will do automatically for you). Once the subs are registered or aligned, this has the effect of moving the "unwanted signal" (thanks for the distinction Ian!) which arises in the camera from being fixed to being more random. If you then stack the subs using a Standard Deviation function or Sigma Clip, it'll remove a lot of the random unwanted signal from the image.

Well worth experimenting with.

Steve

[IanL]

You certainly can use the same master bias for all frames.  I use ISO400 and made one bias of 330 frames (as per the blog post) which I always use.  I don't know if changing ISOs makes any difference to bias, as I have never needed to test it.  Realistically you are not going to see any visible difference once you go over perhaps 100 bias frames, but I use 330 since I made it during the testing and there is no reason not to.   I guess it might be an idea to make a new master bias once in a while if the sensor or camera electronics degrade somehow over time and change the pattern.

I haven't got any conclusions to offer on temperature differences for non-PixInsight users.  I guess it would be more critical to match temperatures between darks and lights if the software is just doing a subtraction (whereas I have found the PI dark scaling process means you really don't have to care too much as it is mathematically finding the best scaling factor before subtracting).

I concur with Steve regarding dithering.  I always use the maximum dither but this is a bit too small to be honest.  It has been suggested that DSLR users should use a really big dither (maybe 10's of pixels each time) to help with clipping out the bad stuff.  Maybe it is something we can as the PHD2 developers to look at adding, or failing that perhaps a slight re-pointing of the scope between batches of images.  More subs will help clipping too (though my last image only had a handful of long subs and a simple percentile clip worked perfectly well).  Whether you should artificially shoot shorter subs so you have more of them to assist in clipping or go for longer ones is another debate.  Personally I go for the longest subs I can without saturating but others will not agree with this.

[RikM]

Rik - what temp difference between lights and darks do you find you can get away with? What temp are the fridge darks shot at? 5C? Also, can you use a single master stack of bias frames for all subs?

For DSLR, I shoot tethered using APT, so it displays the chip temp rather than 'air' temp. My fridge darks were at a chip temp of 17°C. I don't know what the actual temperature inside the fridge was as I just chucked the camera body in the salad drawer and shut the door on the cable.

I do match the ISO of my calibration files to my light frames. I don't know if it is necessary, but I see no reason not to do it, just in case.

I shot sets of 3min, 5min and 8min darks and then reused them for every image from then on irrespective of chip temp but they don't tend to vary much beyond 5-6°C either way. I also dither guide by about 5-8pix (randomised) and then used sigma clipping at the stacking stage.

I find there are many things which make a statistical difference, but there is so much wrong with my images that these things make little practical difference to the final image.

[PhotoGav]

Great replies and advice there, thank you all.

Yes, I do dither my subs - essential for averaging. I will however look at the amount of dither used and ensure it is generous.

I'm using Nebulosity3 for calibration and stacking of subs. Not sure if this does the same 'dark scaling' as PI or not?

Whatever, I will be shooting a goodly number of darks in the fridge very soon!!!

[PhotoGav]

I find there are many things which make a statistical difference, but there is so much wrong with my images that these things make little practical difference to the final image.

Ha - that made me smile - I know the feeling!

[PhotoGav]

Fridge Darks - is it ok to put the camera straight in the fridge just as is or is some kind of 'protection' advisable?!

This is sounding like a silly question, but I hope a sensible answer will be forthcoming!

[Steve 1962]

Hi Gav

I'd be happy using an unprotected camera outside into the garden on a freezing night. However I'm always wary of condensation when bringing cold things into the warm house from the cold garden, so I'd put the camera into a tupperware box or something similar to minimise condensation when you bring it out of the fridge.

Steve

[narrowbandpaul]

Hi Ian,

Great to see some testing going on, it's the only way to learn. I would like to throw my tuppence in to this discussion if I may. In your discussion of noise being random, that may be true but certainly with regard to image sensor there are 2 components to noise. The random bit we all know and love and the fixed pattern component which you regard as signal. Considering the 'bright' fixed pattern noise, you are certainly correct when you say that it removes all those effects from vignetting to pixel level sensitivity variations, and I don't think many realise that. However, I don't agree that it is a signal, at least not how I would define it. I'd you look at the SNR equation you will see that there is a term for fixed pattern noise in the noise part of the equation. It is defined by FPN=PRNUxS, where PRNU (photoresponse nonuniformity) is the scale of the fixed pattern variations and S is the signal. We see it scales linearly and consequently limits the SNR to a value of 1/P. Clearly flats are required to improve SNR. So the fixed pattern component of the total noise is very much a noise.

Random noise comes from several sources, again you hit the nail on the head. There is random noise from the read out electronics and a fixed pattern component (it's fun to take lots of bias to reduce this to the sub electron level to really see the pattern). There is random shot noise from the object and the sky background, there is a fixed pattern noise associated which was discussed above. There is a random noise generated through the emission of electrons: Dark shot noise. There is also a fixed component to this too. It is the conventional hot and cold pixels that we all hate. Again this is described by similar formula. DFPN=DSNU*D, where DSNU is the dark signal nonuniformity and D is the number of dark electrons generated. So dark noise has a random component and a fixed component.

What does the dark subtraction do? I find this is commonly misunderstood. Phrases like remove the dark current are ambiguous at best. Removing the random dark noise too is a factually incorrect statement.

First thing to realise is that you cannot subtract out random noise, when you subtract two images with random noise, the resultant random noise will be higher. The only way to reduce random noise is to take many frames and average them. This is why we take many light frames, flat frames, bias frames, and dark frames. It is solely to reduce the random noise in these images to zero. Taking many bias frames leaves only the fixed component of the noise (or pixel to pixel variations), taking many flats leaves only the fixed component of noise and taking many darks leaves just the fixed component: hot and cold pixels.

So when we do image calibration we do NOT subtract the random noise, we subtract the fixed pattern components. You can ONLY remove fixed stuff via subtraction.

The SNR equation certainly treats all the things listed above as noise. The only thing that gets counted as Signal is the signal from the object itself. It doesn't include the sky background even as it merely adds an offset value that is easily removed, however the random noise and fixed pattern noise it injects is most certainly treated as noise.

I think on the points where we differ it is perhaps just a question of definition, you regard noise and only the random stuff and I see it as a pixel to pixel variation that can be both random and fixed (or spatially invariant).

Anyway, I just wanted to clarify a few things. It is good to see people running experiments, understanding how noise works really is essential for the imager and can only make your images better.

The SNR equation that I am referring to can be written (with random and fixed components) as:

So/sqrt (R^2+So+Sb+(PRNU*S)^2+D+(DSNU*D)^2)

Where So is the signal from the object. Sb is the signal from the background. R is the read noise. PRNU is the photoresponse nonuniformity, D is the dark electrons generated (dark current*exp time) and DSNU is the dark signal nonuniformity.

All these properties can be deduced using just excel. The method is called Photon Transfer Analysis.

Hope that was useful,

Cheers

Paul

[PhotoGav]

Wow, I actually think that all of this is sort of starting to make sense. My brain has been slightly bamboozled by that impressive equation Paul, but your explanations (and from everyone else on the thread) is fascinating, enlightening and most definitely useful. I thank you all for taking the time to share your knowledge & experience.

I never thought that the journey I was embarking on when I decided that I would like to take some pretty pictures of the night sky would require some post-grad level learning!

I will be putting everything into practice (good tip about camera removal from fridge caution, Steve) just as soon as the moon hides and the clouds part.

A final question: is it acceptable, at a family level, to start imaging on Christmas Day?!?!!!

[Russe]

I did my dark library by putting my Canon in the freezer, wrapped in aluminium foil and a towel in a plastic bag.

[PhotoGav]

I did my dark library by putting my Canon in the freezer, wrapped in aluminium foil and a towel in a plastic bag.

Mmm, that must have been pretty chilly! Do you know what temps - freezer & chip?