How many darks required???

[derrickf]

It's precisely what Merlin and I were saying if you had bothered to look at the links we gave Since the noise in any signal (and in this case the signal is the dark current) is equal to SQRT of the signal and because signal adds arithmetically whilst noise adds quadratically the improvement in SNR for n dark frames is SQRT(n)

so SQRT(15) is approx. 3.87 etc.

No magic or guesswork just simple maths.

Pointers to potential solutions to your noisy background can also be found in the links given above.

Right!

I am back from the French enemy line!

so, And the answer is:

15 darks, signal to noise ratio is 3.87

20 darks, signal to noise ratio is 4.47

50 darks, signal to noise ratio is ... 7

So, the winner is... Earl!

the more darks the better....

so, a good average would be 20 darks, regardless of the numbe of subs.

So I have to look elsewhere for my noisy background.

Rgds

Serge

[rfdesigner]

no one seems to be mentioning dithering (adjusting the guide position fractionally between subs)

One reason this helps is that any hot pixels will get 'smeared' and then a good sigma clipping algorithum will take them out.

The other reason is that as the master dark is removed from the frames before alignment, if there are any errors in the master dark (there will be.. it's called noise) then those errors will be 'smeared' in exactly the same way as the hot pixels during the alignment process.

So you only have to think about a single light and what's required to not add significant noise to that while removing pixel to pixel variations in the dark current which give a kind of fixed pattern noise.

Doing the maths a single frame will have 0.06 magnitudes more noise by removing a master dark frame made from 8 darks, you could go further but 0.06 magnitudes is fairly small. I've used 10 in the past and couldn't tell any difference compared to using 20.

If you don't dither (or have other errors), and all your lights line up exactly.. then you would need many more darks than lights.

Hope this helps

Derek

EDIT: of course if the temp of your CCD/CMOS sensor is off a bit between the darks and lights then that can be a substantial hit on the noise performance due to the substantial dark current sensitivty to temperature mentioned earlier.

EDIT2: I use IRIS, and learnt about processing from the author of that. (via tutorials etc. not personally)

[Earl]

I have started to use Dither guiding, and it has made an improvement

[perfrej]

15 flats, 30 bias. Period.

/p

[Merlin66]

I never got an answer.....

Seriously what do you do with those BIAS frames????????

If you have a good dark master it already contains a BIAS signal, the Light also has the BIAS, if you have a Flat (with it's own dark) also has the BIAS - then when you reduce the image, the bias looks after itself???

[ollypenrice]

I'd have thought that diminishing returns come in at around 40 myself and that's my standard dark and flat shoot. I find that they last a long time, at least 6 months, and I image a lot.

However, Serge, let me offer an alternative answer to 'How many darks?' None!

Yves and Harry Page between them have converted me, certainly on the rather difficult Starlight H36 camera. Darks work fine for me on on my Atik 4000s (or so it has always seemed) but they do not work at long exposure on the H36. They clip the black point not a little but a lot. Here's what I do in AstroArt 5;

Dither guide (but in fact it doesn't seem to matter if you don't. I do take a lot of data, though, often on several nights so a shift is inevitable. Also on long runs polar alignment can never be perfect to one pixel...can it??)

Make one master dark. I used 15 mins amd my usual 40 subs, but I doubt you need that many. One sub might even be enough, I don't know.

In Arithmetic-Clip-Minimum clip 2000 off the bottom and leave the white point alone. This removes all the usual dark frame noise apart from the hot pixels. Save this as a Defect Map.

Now, when stacking/calibrating in AA5 I put a master bias in the darks box instead of a regular dark and I put the Defect Map in the Defect Map box (which I think is new to AA5) and enable it.

It is vital to stack in Sigma and to have at least a dozen for it to work properly.

I caibrate my flats (usually about 30 to 40) with the same master bias. Individual time-consistent darks for flats are simply not needed).

I activate AA5's Hot Pixel Filter at default and that's that. The result is way, way, way better on the H36 than using darks. I have yet to try on on my 4000s but I think darks attack your faint signal. When I say 'way better' I mean thata glance at the two images is all you need to see it. A glance at the linear histogram confirms the clipping by darks, too.

Harry has been good enough not to say 'I told you so!' I highly recommend this system. The result images have better faint data and a fraction of the residual noise.

Olly

[Gina]

I think I shall go for around 15+ darks and DSLR cooling for the warmer weather.

[rfdesigner]

However, Serge, let me offer an alternative answer to 'How many darks?' None!

Yves and Harry Page between them have converted me, certainly on the rather difficult Starlight H36 camera. Darks work fine for me on on my Atik 4000s (or so it has always seemed) but they do not work at long exposure on the H36. They clip the black point not a little but a lot. Here's what I do in AstroArt 5;

Olly

Interesting, so does the software you use not allow for negative values?

In IRIS, it works on signed 16 bit numbers. In fact I've been experimenting with normalising images prior to stacking such that the sky is forced to average zero. Then when images don't line up and need registering, the blank area outside of one images view doesn't bias the result from the other images so I don't get black borders, I get slightly more field of view. Then of course I put the offset back in after stacking.

Derek

[rfdesigner]

I never got an answer.....

Seriously what do you do with those BIAS frames????????

If you have a good dark master it already contains a BIAS signal, the Light also has the BIAS, if you have a Flat (with it's own dark) also has the BIAS - then when you reduce the image, the bias looks after itself???

Bias frames allow you to make a dark frame that theoretically includes ONLY dark current. Becasue it only contains the dark current, if you take a 20 minute image and have only a 40 minute dark you can simply divide the dark by 2 before removing it. You don't need dark frames for every exposure length you use. Also you need them for flats, if you don't remove the bias from the flat frames then your flats won't QUITE work correctly, very nearly right but not precisely.

Derek

[Merlin66]

Understood.

But how many members actually use a 40 min dark to produce a 20 min "dark"??

I gather most of them take dark exposures to match the lights???

Just intrigued ......

[sergeC14]

a pixel becomes noisy 2 years later?

It changes every week when you're imaging say 3 times a week...

I hink it's natural radiation killing of pixels. Which is why I'll never buy a second hand camera.

Serge

[JamesF]

I'm slowly working my way through the maths at the start of the "Handbook of Astronomical Image Processing" at the moment. It backs up Derrick, Serge and Merlin regarding the theory and, if I've understood it properly, shows that whilst taking multiple darks is a good thing because the noise can be averaged out, the return diminishes as more are added.

So, it's up to the imager to know how muchj noise their camera generates and (perhaps more importantly) to decide what factor they'd like to reduce it by to determine the required number of darks. Back in the real world however, twenty would appear to be a more than reasonable starting point for experimentation.

James

[alpal]

a pixel becomes noisy 2 years later?

It changes every week when you're imaging say 3 times a week...

I hink it's natural radiation killing of pixels. Which is why I'll never buy a second hand camera.

Serge

I don't know what causes the pixels to change over time.

You certainly can't keep a dark library for years.

If you have an important picture then take fresh darks.

[rfdesigner]

I don't know what causes the pixels to change over time.

Cosmic 'Rays'. Cosmic ray - Wikipedia, the free encyclopedia

We don't get many at the earths surface but over months and years they do damage CCDs.

Derek

[themos]

The answer is always "a dozen". But dithering makes a huge difference to your need for smooth darks.

[alpal]

Cosmic 'Rays'. Cosmic ray - Wikipedia, the free encyclopedia

We don't get many at the earths surface but over months and years they do damage CCDs.

Derek

Thanks Drrek,

I'm know cosmic rays cause damage to CCD & DSLR sensors

but there may also be a natural aging process.

Maybe some pixels are just not properly formed at the factory

& die soon after to become dark, or become hot pixels?

We need a semiconductor physicist.

[daz]

Moved to discussion board

[rfdesigner]

Thanks Drrek,

I'm know cosmic rays cause damage to CCD & DSLR sensors

but there may also be a natural aging process.

Maybe some pixels are just not properly formed at the factory

& die soon after to become dark, or become hot pixels?

We need a semiconductor physicist.

I'm in the Semiconductor Business.

Semiconductors do age but generally not much at room temperatures, We are currently running an HTOL test on our latest chip (ultra low leakage, so similar to CCD in that sense) After 1000 hours at 125C we will test to see if it's still working within spec, if it is then we'll call it fit for purpose. (that and various other compliance tests) This translates to a certain acceptable failure rate at room temperature after 10 years.

Generally silicon doesn't like high temperatures, but loves low ones. If things fail at low temperatures that is usually due to package failures or a chunk of analogue circuitry that has gone out of bias or similar.

DSLR sensors are less susceptable to damage becuase they work on a subtly differenet principle, they are simply more rugged.

Yes your CCD will slowly degrade, but it shouldn't be dropping off a cliff.

Derek

[alpal]

Thanks Derek,

So the sensors do age.

There must be fabrication errors in manufacture which don't reveal themselves

until after a long time.

Therefore:

I am right to assume that not only cosmic rays cause the degradation we see.

Would anyone have a graph showing degradation of pixels -

( whether they go hot or cold ) - versus time in years?

[dph1nm]

Some facts to note. Your light frames contain random noise due to the dark signal. Your dark frames also contain random noise due to the dark signal. If you use a ratio of one dark per light then you WILL increase the contribution this random component makes to your final image by sqrt(2) when you subtract your master dark.

However - this contribution may be totally irrelevant compared with other sources of random noise ( e.g. sky), or, indeed, compared with non-random noise caused by the dark signal. Hence there is no single answer as to how few darks you can get away with - all you can say is that if you use substantially more darks than lights then you will be doing the very best you can.

NigelM

[rfdesigner]

I found this: http://www.stsci.edu/instruments/wfpc2/Wfpc2_isr/wfpc2_isr0501.pdf

Which is a detailed analysis of the Hubble WFPC2.

some nice graphs, but of course a top notch CCD, so most if not all degradation will have been from cosmic rays.

Derek

[JamesF]

all you can say is that if you use substantially more darks than lights then you will be doing the very best you can.

Whilst I don't think that's incorrect, my understanding of the theory suggests to me that it's not the full picture.

My reading of the maths as presented in "HoAIP" is that by combining darks you're attempting to remove the non-dark noise in the dark images themselves in order to recover the "dark signal" so that may be subtracted from the raw image data leaving the "image signal" plus whatever other noise is contained in the raw data.

Each dark added improves the quality of the dark signal, but by increasingly smaller amounts until there's either no perceptible or no useful improvement. How many images that takes is completely unrelated to the number of lights.

Whilst I did a quite stupendous amount of maths for O- and A-level (and places in between), it was twenty-five years ago now though, and stats was probably the area I struggled with most, so I freely admit that I may have entirely the wrong end of the stick here.

James

[alpal]

I found this: http://www.stsci.edu/instruments/wfpc2/Wfpc2_isr/wfpc2_isr0501.pdf

Which is a detailed analysis of the Hubble WFPC2.

some nice graphs, but of course a top notch CCD, so most if not all degradation will have been from cosmic rays.

Derek

Thanks - it sure shows that a new camera is worth a lot more than an old one.

[rfdesigner]

Whilst I don't think that's incorrect, my understanding of the theory suggests to me that it's not the full picture.

My reading of the maths as presented in "HoAIP" is that by combining darks you're attempting to remove the non-dark noise in the dark images themselves in order to recover the "dark signal" so that may be subtracted from the raw image data leaving the "image signal" plus whatever other noise is contained in the raw data.

Each dark added improves the quality of the dark signal, but by increasingly smaller amounts until there's either no perceptible or no useful improvement. How many images that takes is completely unrelated to the number of lights.

Whilst I did a quite stupendous amount of maths for O- and A-level (and places in between), it was twenty-five years ago now though, and stats was probably the area I struggled with most, so I freely admit that I may have entirely the wrong end of the stick here.

James

What you're trying to remove is variations in the dark current from pixel to pixel.

so if Pixel A has 0.1 e/s and pixel B has 0.05 e/s then you need to remove twice the offset from A than you do from B.

As you've grasped, taking more darks means you get a better measure of that pixel to pixel variation in dark current, what you can't do is remove the natural quantum variation in the dark current.

Derek

[dph1nm]

Each dark added improves the quality of the dark signal, but by increasingly smaller amounts until there's either no perceptible or no useful improvement. How many images that takes is completely unrelated to the number of lights.

Unfortunately not, because although, as you correctly state, by using more darks you beat down the random noise in the master dark, the same thing is happening to your lights. Each of them has some dark signal with its associated noise, and when you combine your lights together that random noise goes down as well. So if you have equal numbers of lights and darks you end up with a similar dark noise contribution in your master dark and in your combined lights. When you subtract one from the other, this contribution (in the lights) increases by sqrt(2). If you have fewer darks than lights then it will increase by more. It is the ratio of darks to lights which matters - not the absolute number.

However, in many cases both contributions may be so small anyway you don't really care!

NigelM