How many darks required???

[JamesF]

I see what you mean. There was something nagging at the back of my mind when I wrote that, but I couldn't place it. I think this might have been it

James

[derrickf]

Unless I have misunderstood, what you indicate below is that the darks should be combined into a master which is then subtracted from a similarly combined light master.

Standard image calibration has the master dark subtracted from each of the lights in turn which are then combined into a master, calibrated light (for the purposes of this discussion I'll ignore other calibration frames). I agree with James' (and every other source I have consulted) assertion that there is no correlation between between the number of darks and lights when considering the noise contributed to the final image by a master dark.

Unfortunately not, because although, as you correctly state, by using more darks you beat down the random noise in the master dark, the same thing is happening to your lights. Each of them has some dark signal with its associated noise, and when you combine your lights together that random noise goes down as well. So if you have equal numbers of lights and darks you end up with a similar dark noise contribution in your master dark and in your combined lights. When you subtract one from the other, this contribution (in the lights) increases by sqrt(2). If you have fewer darks than lights then it will increase by more. It is the ratio of darks to lights which matters - not the absolute number.

However, in many cases both contributions may be so small anyway you don't really care!

NigelM

[Ibbo!]

For DSLR I do 16

I read somewhere about diminishing returns etc and they suggested 16

for the CCD I don't do darks as it makes things worse

I use bias as its an SXH9

Oh and I do dither and combine using SD mask

I've just upped the no of flats from 20 to 40 for no logical reason at all

Steve

[JamesF]

If I've understood Nigel correctly I believe what he's saying is that your darks will contain the dark signal, plus other noise which is reduced (to a point) as the number of darks increases. The light signal contains the image signal, plus the dark signal, plus the dark noise, plus other noise.

By matching the number of darks to the number of lights, one might perhaps expect to get the "best match" between the dark signal + remaining noise in the master dark and the dark signal + dark noise in the combined light and therefore get the best result when subtracting one from the other.

This is definitely what was nagging at me when I wrote my earlier post, but whilst it appears to make sense intuitively I'm not sure it's actually statistically valid. I might have to go back to the stats and work my way through it. Unfortunately as I've already said, stats is something I struggled with at the time I was supposed to be learning it, let alone twenty-five years later. It's going to be the only way I can convince myself though. Appeals to common sense are too often wrong.

James

[derrickf]

One of the most proficient statisticians I know who is also an avid imager is Stan Moore (author of CCD Stack software); over the last 20 years or so I have followed (and partcipated) in many discussions on the subject of image calibration in which many other accomplished imagers have been involved and in which Stan has played a major role, often providing the rigorous mathematical treatment of the various arguments. Based on what I have learned I have to say that I can see no statistical validity for the assertion that there is any correlation between the number of darks and the number of lights which would determine the dark noise in the final image (unless the flawed calibration process alluded to earlier is followed).

Clearly the number of calibrated light images combined to create the master light has a direct effect on the image SNR but this is not directly linked to any correlation between light and dark frame quantities. As others have stated: the SNR of the master dark is a function of the number of darks and is easily calculated.

I had thought that a satisfactory response to the original poster's question had been reached several days ago when there seemed to be general acceptance that the empirically determined rule of thumb that 15- 20 darks was sufficient for most circumstances was supported by the underlying maths

[ollypenrice]

I image frequently for more than three nights per week. Sometimes it's seven. I don't find darks deteriorate quickly at all. I can still use darks that are nearly a year old, in fact.

I'm intrigued by the silence greeting my recommendation not to bother with darks at all. I was similarly silent when Harry first suggested I drop darks. All I can say is, Give it a try. You might be pleasantly surprised and save yourself a lot of faffing about.

Olly

[JamesF]

I had thought that a satisfactory response to the original poster's question had been reached several days ago when there seemed to be general acceptance that the empirically determined rule of thumb that 15- 20 darks was sufficient for most circumstances was supported by the underlying maths

Yes, it probably was I just wanted to explore the subject a little further for my own understanding.

James

[derrickf]

Olly,

does this work well with your Atik4000 cameras? I remember trying this some years ago with an ST-10 (KAF 3200 based camera) with little success although I know a number of people who had excellent results with Sony chipped cameras.

Derrick

I image frequently for more than three nights per week. Sometimes it's seven. I don't find darks deteriorate quickly at all. I can still use darks that are nearly a year old, in fact.

I'm intrigued by the silence greeting my recommendation not to bother with darks at all. I was similarly silent when Harry first suggested I drop darks. All I can say is, Give it a try. You might be pleasantly surprised and save yourself a lot of faffing about.

Olly

[derrickf]

Yes, it probably was I just wanted to explore the subject a little further for my own understanding.

James

You may have seen these already, but if not they make good reading:

http://www.hiddenloft.com/notes/SubExposures.pdf

http://www.hiddenloft.com/notes/DarkSubExp.pdf

http://www.hiddenloft.com/notes/SNR.txt

and some early work on the issue (mentioned elsewhere by Olly) of whether darkframes are needed at all - as you can see from the date, this concept is not a new one:

Do you need dark frames

HTH

Derrick

[JamesF]

They're on my reading list Derrick. I just haven't got that far yet Thanks very much for the pointers though.

James

[ollypenrice]

Olly,

does this work well with your Atik4000 cameras? I remember trying this some years ago with an ST-10 (KAF 3200 based camera) with little success although I know a number of people who had excellent results with Sony chipped cameras.

Derrick

No, I have yet to try it on my 4000s because I've always felt happy with the usual old darks routine. However, using Yves' SX H36 that way just wasn't working. It has a huge number of hot pixels and the darks clipped the image. (They really did clip it and not by just a little in the case of the longest ones I tried, which were 30 mins.)

There is absolutely no doubt whatever that its results using dither/bias/defect map/sigma are far cleaner than the same data calibrated with darks.

I really think Serge should try this routine. I know the Atik 11000 is good but noisy and this could be the answer. By the way, the Sigma routine in AA5 is vastly improved over AA4. Sat trails that remained in AA4 stacks are removed in AA5. I had dozens in my Witchead raws but they all vanished.

Olly

[ollypenrice]

Sorry, duplicate. Please delete.

[derrickf]

No, I have yet to try it on my 4000s because I've always felt happy with the usual old darks routine.

Olly

Most people who are experienced enough to appreciate what you are suggesting probably do have a satisfactory image calibration process that works well and that probably explains the deafening silence that has greeted your posts advocating that people try the "no-dark" calibration method.

I may try it again when I buy a new, large pixel-count camera at the end of the summer twilight. BTW the final link in my last post, addresed to James, has some quantitative comparisons between the two methods.

Derrick

[ollypenrice]

Most people who are experienced enough to appreciate what you are suggesting probably do have a satisfactory image calibration process that works well and that probably explains the deafening silence that has greeted your posts advocating that people try the "no-dark" calibration method.

I may try it again when I buy a new, large pixel-count camera at the end of the summer twilight. BTW the final link in my last post, addresed to James, has some quantitative comparisons between the two methods.

Derrick

Thanks for that link, Derrick. An interesting read. It might also be worth thinking about the fact that darks have to be temperature matched to work reliably. If the chip really does remain at the right temperature throughout the run both of darks and lights, then fine. But what if it doesn't? In that case the advantage slews heavily, I would have thought, towards the no-darks option.

I would like to know why darks proved so overly aggressive on my 30 minute subs with the H36, though. There are lots of mismatches between theory and practice in imaging, I seem to think, and since I make my living from it, in part, I always listen first to practice!

Olly

[Jessun]

I read an article in Astronomy (the April 2012 issue) by Tony Hallas. I can not just copy and paste here obviously but he argues that the gain is good up to 16 dark frames and no further gain made after 25 - reaching the 'asymptote boundary'. He relates this to S/N ratio where the improvement of signal to noise equals the square root of the number of frames combined. 16 frames have a four fold increase and 25 frames a five fold. In effect the article suggests taking the longest exposures possible for your sky and equipment.

Even in heavy light pollution I have noticed a great improvement from doubling my subs to 10 minutes - but beyond I reckon I'm getting close to the wash out.

I have images with over 80 5 min subs.... I wish I did it differently now.

As to darks btw I take 0. Just 20 Bias.

[rfdesigner]

It is the ratio of darks to lights which matters - not the absolute number.

NigelM

Only if you have perfect alignment such that you don't need to align images after dark subtraction.

If you have to align each sub a different lateral offset from the final image, then any errors you introduce from the dark master are placed on different pixels in the final image, so these errors do not stack on top of one another.

The key is that alignment is done after dark subtraction

This means the important factor is the number of darks, not the ratio of darks to lights.

Not immediately intuitive but it does work out.

Derek

[dph1nm]

Unless I have misunderstood, what you indicate below is that the darks should be combined into a master which is then subtracted from a similarly combined light master.

Standard image calibration has the master dark subtracted from each of the lights in turn which are then combined into a master, calibrated light

Oh, sorry, I sort of mis-worded my comments. You do subtract the master dark from each light. However, this doesn't help because each master dark is the same, so the random noise you add is 100% correlated between the lights. Statistically (assuming we have equal numbers of lights and darks), subtracting a master dark from each light, then combining the lights, is the same as subtracting one dark from each light, then combining the lights.

Please remember I am only talking about the random noise contribution from the darks here.

NigelM

[dph1nm]

Just to be clear why people are getting this wrong. If you use a master dark on each of 10 lights you will get 10 times the random noise in the master dark added to your lights. If you use it 100 times you will get 100 times the noise. There is no sqrt gain from using the master dark more than once because the random noise pattern is identical. On the other hand, the random noise in the lights from the dark signal is different each time, so you do get a sqrt(n) gain - hence the more lights you use the smaller the relative contribution this sort of noise makes. Eventually (when n lights > n darks), it will be smaller than the contribution from the master dark. You are now degrading your data using the master dark.

This is why the ratio of lights to darks may matter (I re-iterate again that other forms of noise may swamp this dark contribution and so make the argument irrelevant).

NigelM

[rfdesigner]

This is why the ratio of lights to darks may matter (I re-iterate again that other forms of noise may swamp this dark contribution and so make the argument irrelevant).

NigelM

NigelM

This is going to be my final word on this. I do stats as part of my living, I've even built a CCD camera from scratch (not a kit), what you are saying is wrong and I don't want others falling into the same logical hole you've fallen into.

let's agree the process is this or something very similar, note dark subtraction comes before align or stack:

1. dark subtraction

2. flat field

3. align

4. stack

5. post process

Please take time to think about this:

Concider a row of pixels of the master dark frame. Each pixel is made by averaging say 20 dark frames. Each pixel in the final dark master is statistically independant from its neighbours, this is what you're building on.

Due to dithering during imaging (or just plain alignment movement) the stars that are being imaged are shifted by one or more pixels between each sub, so the stars do not align with the same dark master pixels on each sub.

So when you align and stack the subs after having taken off the dark master the final image contains the noise from several different pixels in the dark master, a different pixel for each light frame. it's as if you'd generated as many seperate dark master frames as light frames.

THIS IS ONLY TRUE IF YOU DITHER.. You are absolutely correct in your assertion if your images are so well aligned that you do not need to align the images after dark subtraction, because then when you stack you are placing the error from a single pixel in the dark frame repeatedly onto the same pixel in the image frame.

please please do concider the impact of dithering, it is quite profound.

Derek

[ollypenrice]

...and, while you can guide at sub pixel level, can you polar align at what will remain sub pixel level on a set of long exposures? I doubt it. There will be 'unintentional dither' in any deep stack.

As Derrick says, the effect of dither is prodigious.

Olly

[Gina]

Well... I've been ploughing through this conversation/debate and not really seeing any real result coming out of it though the bias seems to be for no darks or about 15.

Last time we had a clear night (whenever that was!) I took two sets of 100 lights plus 15 darks of M42 at different exposures. Taking the best set of lights with its corresponding darks gave a disappointing result IMO. So I decided to go out on a limb and try all 200 lights and no darks. The result was much better. Does that prove anything? Was the improvement due to using more lights or no darks or a combination of both? Who knows

[rfdesigner]

Gina

I've found with temperature controlled CCD darks work very well.

But for DSLR and non-cooled CCD I've had problems, I believe because they simply do not hold the same temperature between the dark and the light frame so the variations in the dark current from pixel to pixel are not correctly moddeled in the dark master frame. In which case there is an argument for just making a hot pixel map and forgetting dark frames.

Derek

[Psychobilly]

With the 1000D i didn't bother with darks...

I would typically get around 0.1-0.2 pixel drift between frames (PA slightly offset). So with a combination of enough frames over a couple of nights, with a couple of pixels shift between runs and then using the cosmetic hot/cold pixel removal tools in DSS I was happy with the results...

Peter...

[dph1nm]

Due to dithering during imaging (or just plain alignment movement) the stars that are being imaged are shifted by one or more pixels between each sub, so the stars do not align with the same dark master pixels on each sub.

So when you align and stack the subs after having taken off the dark master the final image contains the noise from several different pixels in the dark master, a different pixel for each light frame. it's as if you'd generated as many seperate dark master frames as light frames.

THIS IS ONLY TRUE IF YOU DITHER.. You are absolutely correct in your assertion if your images are so well aligned that you do not need to align the images after dark subtraction, because then when you stack you are placing the error from a single pixel in the dark frame repeatedly onto the same pixel in the image frame.

Sorry - this is incorrect. Dithering on the sky has no effect on the noise due to the dark current, as this is a property of the chip. Think about it - the dark current in pixel x,y is still the same whether you move the telescope around on the sky or not. It doesn't know, or care, anything about the incoming light. So I am afraid one master dark is always just that - you can't get a free statistical lunch by dithering the scope around!

Dithering is useful of course - it helps to remove non-random noise.

NigelM

[Jessun]

Good ol' thread this!

I am having a bit of a lazy moment and have no real data to check on so I simply wonder if anyone has two stretched or fully processed stacks - one with darks and one without to post here for a comparison?

Apart from hot/dead pixels and that sort of very obvious oddities - errors that dithering or software can get rid of - what is the actual gain of darks at all? I mean does it make a difference worth the effort past the simple BIAS frames? Are they useful for any sky condition or does LP by far outgun that tiny dark current in the chip?

Sorry, lazy this - not experimenting myself, but I don't take darks at present and my DSLR past is best forgotten since I was rubbish at it...