What ratio of darks to lights?

[Space Cowboy]

Looks clear for tonight so will be having another crack at M42 with 15 sec subs (longest my camera will go).

Hoping to get around 100 subs so how many darks should I take?

[antimorris]

As many as possible, generally the rule is have as many darks as you do lights. The more you have the better your master dark will be.

[Space Cowboy]

Thanks Anna!

[Digz]

Good advice Anna, but Im sure I have read somewhere that there is a limit on darks with around 20 being the optimum amount (I may be wrong). Have a search on the forum here there has been a lot of debate on this.

[Zakalwe]

I thought that there was an upper limit, after which more darks become ineffective? Otherwise you might as well have Noise Reduction enabled in a DSLR and take a dark with every light?

[dph1nm]

I thought that there was an upper limit, after which more darks become ineffective? Otherwise you might as well have Noise Reduction enabled in a DSLR and take a dark with every light?

No, theoretically, the more darks the better the master dark will be. However, the improvement only goes up as the square root of the number, so eventually you will loose the will to live!

The trouble with in-camera darks is that you lose valuable observing time on clear sky. It is implicitly assumed that if you take separate darks you do not do it while the sky is clear.

NigelM

[jgs001]

I tend to find the maximum number of darks I take is limited by my patience... it was easy when I was shooting 30 to 60 seconds, to grab 20 to 30 darks... when you get up to 5 or 10 minutes... and you're wanting to stack your lights, getting more than 1 hours worth is worse than watching paint dry... I guess it would be better if I had an obsy... but I need to keep an eye on stuff...

[Psychobilly]

I used to grab some whilst waiting for the sky to become truly dark, some mid sequence darks if there was passing cloud and more at the end whilst closing up......

Billy...

[rfdesigner]

I've just noticed this thread. There's another aspect not discussed here.

Darks are used to reduce noise.

To understand what you're doing it's worth analysing the noise... as an RF engineer by day I do this for radios most weeks, in one way or another.

Primary noise sources:

1. Dark current noise

2. Pattern noise

3. Sky glow noise

4. Readout noise.

When thinking about Darks, it's the first two that should be concidered. Skyglow we can't do much about with dark frames, and readout noise is noise source that stops us just using real time video.

Darks:

Dark Current noise in purely cumulative, if you wait for 100 electrons to acculate you wil have a nosie of 10 electrons or 10e.

If your images are 10 miniutes each and you have 10 of them and your dark current is 10e per minute then you will have 10minutes*10images*10electrons/minute noise or 1000electrons. The variation in that noise will be the square root of it or ~30e.

By taking dark images and removing them you should have exactly the same dark signal in the dark image as in the light image. But by taking say 4 dark images of 10 minutes each you will have 1/SQRT(4) noise in the darks or about half the noise. If you add this to the noise in the sub-fram it will raise it by sqrt(1^2+0.5^2) = 1.118, so a 12% increase or 0.12 magnitudes, 8 darks would give you 0.06 magnitudes increase over perfection.

If we have perfect tracking on our telescope then we run into trouble. Because of pattern noise.

Pattern Noise.

If all our subs are so well tracked that they simply don't need aligning because they are pixel by pixel aligned then when we remove the dark frame from each subframe all the dark frames will also perfectly align and suddenly any pixel to pixels variations in the dark frame will dominate.

However if we move, by at least one pixel, each subframe relative to all the other ones, by shifting our autoguider during the imaging session, then when we remove the dark frames from the subs, and stack the subs, the effect of the dark frames will be shifted over many pixels and so the noise level resulting from the dark frames will be as if we have taken many many more dark frames.

So. If you keep perfect alignment you'd need ~4x as many darks as lights. If however you nudge the tracking between each shot just 4 darks would arguably be quite sufficient.

Hope this helps

Derek

[roundycat]

Lots of advice above, all you need to do is pick out the relevant bits!

There is no law says you should not take darks on the cloudy nights that everyone is always moaning about.

The so-called rule about as many darks as lights is nonsense. Do the sums, three darks is no good to anyone.

Darks are used to subtract out the thermal noise, the rest of the noise in the frame you are stuck with. That's why stacking many lights is such a good idea.

When you subtract the master dark you leave a trace of the noise or measurement uncertainty behind. So, the master dark should be made from many frames to reduce this added noise. One dark will leave behind 41.4% of it's noise, ten darks leave 4.9% and twenty leaves 2.5%. Go up to 100 darks and the figure drops to 0.5% but you are really in the land of diminishing returns here. Once you get below the noise in the frame that you cannot get rid of there is no point in shooting more darks.

You could suffer from a lot of sky noise if you do not have a dark site and you will always suffer from readout noise as you cannot get rid of it. Shoot enough lights at a long enough exposure that they overwhelm the readout noise and the read noise tends to become insignificant - but it is still there.

Mostly pattern noise is taken care of with good flats.

I would recommend 20 darks but they must be at the same temperature as the lights or they will be invalid. They will then under or over correct.

Dennis

[rfdesigner]

RE: I would recommend 20 darks but they must be at the same temperature as the lights or they will be invalid. They will then under or over correct.

why 20?

when you say they should be the same temperature as the lights, I'm assuming by 'lights' you mean the actual image subs? in which case I would wholeheartedly agree.

Also I notice no one has mentioned bias frames.. does anyone else use them?

Derek

[Space Cowboy]

Thanks for the comprehensive advice guys! Taking darks during cloudy nights sounds very practical as long as the temperature is the same.

Unfortunately DSS is telling me that my camera's jpeg format means it cannot calibrate the darks.....

[roundycat]

Derek, I have been using 20 darks for masters for a long time. The added noise is only 2.5% and that is generally below the read noise so is effectively 'hidden' in the calibrated frames.

The only real-world use for Bias frames is if you don't take Flat Darks to calibrate your Flats. Then you must subtract the Bias offset or the flats will not work properly.

Dennis

[rfdesigner]

ah!, ok what you call Flat-darks I call bias frames, yes Flats need to have Flat-darks/bias taken off otherwise they're of limited value.

Clearly I am more tolerant of the residual noise, hence I am happy with 4 darks, which gives me 12% extra noise over perfect conditions.

Derek

[dph1nm]

Derek, I have been using 20 darks for masters for a long time. The added noise is only 2.5% and that is generally below the read noise so is effectively 'hidden' in the calibrated frames.

Don't understand this. The ratio of read noise to shot noise in the dark remains unchanged no matter how many darks you average. They both go down by sqrt(number of frames).

Also if you have 20 lights (say) then the random dark noise contribution to the averaged lights will be the same as in your master dark - by subracting your master dark you then increase this contribution by a factor sqrt(2). Now this may or may not be significant in the overall scheme of things, but it is certainly not optimal.

NigelM

[roundycat]

first, I meant the read noise in the lights.

second, the master dark is subtracted from each light, not the averaged total.

Dennis

[rfdesigner]

Don't understand this. The ratio of read noise to shot noise in the dark remains unchanged no matter how many darks you average. They both go down by sqrt(number of frames).

Also if you have 20 lights (say) then the random dark noise contribution to the averaged lights will be the same as in your master dark - by subracting your master dark you then increase this contribution by a factor sqrt(2). Now this may or may not be significant in the overall scheme of things, but it is certainly not optimal.

NigelM

Perhaps it's easier to understand if I put down the process I use

Say I use 20 'subs' each at 10m, 5 'darks' each at 10m, 20 'flats' each at 1second and 20 'dark-flats' or 'biases' each at 1second.

subs are sub1..20

darks are dark1..5

flats are flat1..20

biases are bias1..20

I use a program called IRIS but other programmes will do virtually identical things.

I use a command available to me called 'composit' which looks at the data statistically, I set it up to discard all data in each pixel outside 3 standard deviations from the rest of the data for that pixel, and I get it to loop through the calculation twice before taking the average of the remaining data. This nicely filters out any cosmic rays I may get in my dark frames.

: dark_master = 'composit' (dark1..5)

Then I take the dark_master away from each sub

: sub_dark_corrected1..20 = sub1..20 - dark_master

Now I work out my flat in the same way as the dark

: flat_composit = 'composit' (flat1..20)

but this doesn't take into account any readout offsets in the signal.

: bias_composit = 'composit'(bias1..20)

now we can take away the bias from the flat to reveal the pixel by pixel sensitivity of the CCD.

: flat_master = flat_composit - bias_composit

Now we correct the subs for sensitivity variation, I choose "gain" (just a constant) to ensure all the subs are not quite saturated on any pixel.

: sub_corrected1..20 = gain * sub_dark_corrected1..20 / flat_master

Now I do a bit of sky correction.. but we'll skip that for this.

Then comes the other important bit for noise performance.. Aligning.

: sub_registered1..20 = 'coregister' (sub_corrected1..20)

What this allows you to do is to 'smear' the effect of the dark_master you've taken off each frame. If all your subs stack without having to align them then any minor error in the dark master will come straight out in the final shot. In this hypotheitcal case this noise will of course be far worse than the noise in the subs, but by smearing you can average adjacent pixels from the dark_master in the final image and thus make it perform like you have 20 different dark_masters. To get the same performance without dithering the subs you'd need 100 darks.

Now just add them all together.

: first_basic_image = 'composit' (sub_registered1..20)

Then you can play with the image in 1001 ways.

This is what I do.. everyone will do things differently, but I put this in as an illustration.

I hope this is of some use.

Derek

[roundycat]

A read of Astronomical Image Processing for Windows (AIP for Win) will help to clarify what image calibration is all about. I think it is also helpful to be consistent with the terminology.

The idea of smearing anything in the image hardly sits well with the idea of making images as sharp as possible. I have certainly never allowed anything to 'smear' any of my images (or darks or lights come to that).

I am almost convinced that the best approach to dark subtraction is to apply a bad pixel map to each frame and then combine using SD Mask after dithering images at the mount. Thermal pixels are taken care of and the reduction in background noise with a sigma reject routine has to be seen to be believed.

Dennis

[rfdesigner]

The idea of smearing anything in the image hardly sits well with the idea of making images as sharp as possible. I have certainly never allowed anything to 'smear' any of my images (or darks or lights come to that)

No wanted information is getting smeared.. and from your description you are doing more or less exactly what I'm doing.

Derek

[dph1nm]

second, the master dark is subtracted from each light, not the averaged total.

It is worth realising that, mathematically, if you have equal numbers of lights and darks, then subtracting the master dark from each light and averaging the result is exactly the same as subtracting a different single dark from each light and averaging the result.

So you are in effect adding one darks worth of noise to each light.

NigelM