Questions about darks and also exposure time for stars

[acharris77]

Hi all I have 2 questions that are probably silly but here goes lol. There are suppose to be clear skies tonight for a bit, so I might try some imaging again. My first question is how many darks should be taken in a session? or is there like a ratio to light frames?  Also if I wanted to build up a collection of darks which I will take on the cloudy nights in various temperatures how many darks should I take one night.

My 2nd question is if I want to image stars like Albeiro would I need long exposures or would a collection of  say 10 secs be more than plenty for stars?

Thanks for the help in advance.

[nmoushon]

In regard to darks - Your camera will heat up the longer you use it. So even if the outside temp stay the same all night your camera sensor wont. (This is more a worry for long exposure imaging than very short exposures) So I always did 1/3 to 1/2 of my darks at the beginning of imaging and then the rest at the end. This would give me a pretty good average of the night. For the number of darks I always tried to aim for half as many darks as lights. So 8 hrs of lights would equal 4 hrs of darks. If you are doing long exposure you don't want to take up your entire night doing darks so no need to do them all in one night either. You can divy up the time according to your imaging session that night. And don't be afraid to spend several night collecting darks and lights. That is the most constant way to get darks. But I know weather is not always friendly so if you dont think you'll have time to get all the lights you want then skip the darks and just do lights. You can always go out during another cloudy night that is the same temp and collect darks. Its not the best way but you have to work with what mother nature gives you.

In regards to stars - If you are imaging individual stars or clusters it is best to do short exposures. But it varies depending on the object and your skies. So you will have to play with the exposures a bit to make sure you don't blow them out and lose star color. But it usually runs between 10-30sec. And sometimes requires multiple exposure lengths and some photoshopping to get it looking right.

[acharris77]

Thanks for that really great advice, it makes a lot of sense and something I will definitely give a try. Thanks again.

[ollypenrice]

I cannot see any reason to believe that there is a relationship between the number of darks and the number of lights, I'm afraid. The more lights you have the less noise you have to contend with. If I had only a few lights I'd want the best possible darks. If I had a huge set of lights, especially if they were dithered, I'd have far less noise for the darks to deal with and might not use them at all. So in a sense I'm arguing the opposite point of view to nmoushon.

What I really think is that darks with DSLRs may not even be a great idea at all. It is very hard to know what the real sensor temperature was at the time of capture and it will also change during a single exposure. Darks at the wrong temperature will do more harm than good. What I would do if using a DSLR (and I don't use them at the moment) is take a very large number of subs, dithered on a large scale of around 12 pixels (based on a convincing demo by Tony Hallas) and subtract a master bias as a dark.

(Even using set point cooled CCD, where darks are taken at the right temperature, I don't use them any more. I prefer bias subtraction, bad pixel mapping and hot pixel filtration.)

The best tool of all for you will be large scale dither and bias subtraction. Darks at 'guess that temperature' probably won't help at all.

Olly

[nmoushon]

I can't really argue with you there Olly because I havent heard of your theory before let alone try it. It does make since though...most of it. Though I can't see how darks would hurt more than help, especially with a DSLR. CCD I could see it more your way. I just see the DSLR creating so much noise from the heat that you would need darks to remove all the noise that the combine light totals wont get rid of. I'm also thinking only long exposures as short exposures don't heat the sensor up as much. Also I think that darks within a degree or two wouldnt be that horrible. I know exact is preferred but is not always possible. Now 5 or more degrees would be doing some damage. I'm having a hard time seeing it from my own experience. With 8 hrs of lights on M42 I saw massive noise in my subs and saw a big improvement with more and more darks I added. I will say I didnt try it without darks as your suggestion so I could've ended up similar. Just hard to see.

Do you mind linking where you saw that demo if its on the web? Would love to read up on this technique before I really pick a side to defend. I really hope it has some good evidence and practice behind it. I would LOVE not have to darks and keep them....they are such a PITA. 

 

[ollypenrice]

I can't really argue with you there Olly because I havent heard of your theory before let alone try it. It does make since though...most of it. Though I can't see how darks would hurt more than help, especially with a DSLR. CCD I could see it more your way. I just see the DSLR creating so much noise from the heat that you would need darks to remove all the noise that the combine light totals wont get rid of. I'm also thinking only long exposures as short exposures don't heat the sensor up as much. Also I think that darks within a degree or two wouldnt be that horrible. I know exact is preferred but is not always possible. Now 5 or more degrees would be doing some damage. I'm having a hard time seeing it from my own experience. With 8 hrs of lights on M42 I saw massive noise in my subs and saw a big improvement with more and more darks I added. I will say I didnt try it without darks as your suggestion so I could've ended up similar. Just hard to see.

Do you mind linking where you saw that demo if its on the web? Would love to read up on this technique before I really pick a side to defend. I really hope it has some good evidence and practice behind it. I would LOVE not have to darks and keep them....they are such a PITA. 

Darks contain a pattern of noise. This pattern varies with temperature. If the temperature doesn't match the lights the pattern which is subtracted does not subtract the noise in the lights but subtracts ADU values arbitrarily, so, in effect, adding noise. I've seen darks make mattters worse plenty of times which is why I don't use them any more.

I saw the Tony Hallas video via a link posted on here maybe a month ago. I have to rush but it should be searchable.

Olly

[ollypenrice]

Here you go. 

https://www.youtube.com/watch?v=PZoCJBLAYEs#t=39

What's significant about the large scale dither is that it reduces what Tony Hallas calls background sky 'colour mottle' which is, indeed, the bane of DSLRs. Normal noise can be dealt with by a much smaller dither.

Olly

[Psychobilly]

I have a feeling  of deja vu  - I remember a good few years ago when I was actively imaging mentioning that I didn't use darks with DSLR's - provided they didn't have large scale issues like ampglow...  This was a an "experimental" result rather than one buried under pages of "theory"... 

Peter...

[wxsatuser]

I understand that modern highend Canon dslrs have on sensor dark current suppression and low fixed pattern.

With these cameras darks and bias are not needed and if camera lenses are used, no flats are needed with lens profiles.

Hot, dead and stuck pixels are ignored by modern RAW convertors if sensor clean is invoked before each session.

[nmoushon]

Thanks for the link Olly. It took me a while but finally got to watching it. Thats some very convincing speech. The only thing I wish he had actually gone over was long exposure DSO imaging. I still think there could be some discrepancies....could be. He only showed examples of short exposures and a low number of frame with a very wide FL. I wondering how that translates over to long exposures and hundreds of subs at over a meter of FL. Do you use his technique of only dithering with your cameras (both DSLR & CCD)?  

[sharkmelley]

Darks contain a pattern of noise. This pattern varies with temperature. If the temperature doesn't match the lights the pattern which is subtracted does not subtract the noise in the lights but subtracts ADU values arbitrarily, so, in effect, adding noise. I've seen darks make mattters worse plenty of times which is why I don't use them any more.

I can see what you are saying but I slightly disagree.  Darks contain both read noise and a coherent thermal pattern of noise.  This thermal pattern is caused by "hot", "warm", "normal", "cool" and "cold" pixels and is totally unique to every individual sensor.  If sufficient darks are combined into a master dark then the read noise in the master dark can be reduced to arbitrarily low levels, leaving only a characterisation of this thermal pattern.  The thermal pattern itself is not temperature dependent but its intensity (or brightness) does increase with temperature - typically the intensity of the pattern doubles for each 6C increase in sensor temperature.  If you don't have set point cooling then the master dark must be scaled differently for each sub-exposure that it is subtracted from.  Also if the master dark is not constructed from sufficient darks then some read noise will be added to the subs during the subtraction.  So the question raised by the Original Post is definitely a pertinent one - the more lights you have, the more darks you need for your master dark.  But it is very difficult to come up with a hard and fast formula.

The Hallas method always raises a lot of intense controversy.  Since it relies on using Adobe Camera Raw to process the raw data there is no possibility of using darks.  But ACR is very powerful in performing lens corrections, which is very useful if you are using lenses.  So dithering is essential in order to remove both the thermal pattern and the so-called colour mottle.  ACR is also excellent at preserving the correct colour balance - something which many folk struggle with when using Photoshop "curves" (since the "curves" operation whitewashes the correct RGB ratios of the brighter data).

The other weakness of the "Hallas Method" is that it uses lens vignetting correction instead of using flats.  So if you have too much dust on your sensor then it cannot be corrected for.

That being said, my own opinion is that there is a lot of value in the "Hallas Method".  I am slightly worried that faint stars might be erased or have their brightness reduced by the noise reduction step - properly scaled darks don't have this effect.  However, I'm open minded enough to want to pick the best of the processing methods that are available to us - so to that end I'm planning to give the "Hallas Method" a proper try in the near future for comparison purpose with my current processing pipeline but first I'll have to do some serious acquisition with a much greater level of dithering than I currently use.

Mark

[ollypenrice]

Mark, points well made. A small thing but can dust on the sensor be corrected by flats? I'd have thought that anything on the sensor would be totally opaque meaning no information behind it. Flats work by brightening or dimming the underlying information but if there isn't any...?

This doesn't affect me because I use shutterless CCD cameras and simply don't get dust on the sensor. (Touch wood!)

I think we agree that experiment beats any other method.

Olly

[sharkmelley]

Olly,

Generally the dust sits on the internal filter, a few millimetres in front of the sensor.  If those filters have been removed then the dust will sit on the sensor itself but there is still the thickness of the cover glass (at least) between the dust and the photosites.  So at least some of the cone of light will reach the photosites directly behind the dust unless imaging at very high F-ratios e.g. F/10, F/20.   So it could affect planetary imaging.

Regarding darks, there is an interesting article here written by a professional astronomer who takes imaging (in his spare time) quite seriously: http://www.asiaa.sinica.edu.tw/~whwang/gallery/random_notes/dark_tests/index.html  

He comes to the conclusion that with good dithering during acquisition then sigma clipping as just as effective as taking darks, at least for a fairly modern DSLR.

Mark

[Psychobilly]

Interesting article thanks for the link...

I used to advocate people getting to know their own personal cameras performance.

Whenever I used to get a new DSLR I have had quite a few (Owned Nikon D50,D200 Canon 350D,1000D,450D,500D,7D,5DIII and 7DII and had other models pass through my hands on loan) I always used to  carry out a series of tests to see how noise levels were effected by exposure time, ISO setting, time between exposures, Sensor temp as a function of number of subs etc

Towards the end of my active imaging exploits (using a modified 1000D with an external PSU and battery eliminator) I only used to shoot lights, slats and bias, I made sure I had a large number of lights , used an outlier stacking method (Kappa-sigma in DSS)  and the frames were naturally dithered (on a small scale) by a slight misalignment of the PA and by an offset  each night spent on a particular target...... This was before inter sub "dithering" became an established method  plus my mount (a CPC 800 on a wedge) wasn't computer controlled I just used to use the handset..

The older cameras in particular the Nikon D200 I started with and the Canon 350D which was my first "modified" camera definitely needed darks and a quite long inter sub pause to allow the sensor temp to become relatively stable...

Peter...

[ollypenrice]

Olly,

Generally the dust sits on the internal filter, a few millimetres in front of the sensor.  If those filters have been removed then the dust will sit on the sensor itself but there is still the thickness of the cover glass (at least) between the dust and the photosites.  So at least some of the cone of light will reach the photosites directly behind the dust unless imaging at very high F-ratios e.g. F/10, F/20.   So it could affect planetary imaging.

Regarding darks, there is an interesting article here written by a professional astronomer who takes imaging (in his spare time) quite seriously: http://www.asiaa.sinica.edu.tw/~whwang/gallery/random_notes/dark_tests/index.html  

He comes to the conclusion that with good dithering during acquisition then sigma clipping as just as effective as taking darks, at least for a fairly modern DSLR.

Mark

I see, yes. I've been influenced buy the large number of modded DSLR images I've seen or been invited to look at in which dust on the sensor itself seems to be a fairly common problem. I guess not all mods are carried out in perfect conditions, even by some of those who do it professionally.

Olly

[wxsatuser]

I have tried with and without darks and bias using the 60D but was hard to tell the difference.

All I do now is lots of subs, at least 30+ and Kappa Sigma when stacking.

[Jokehoba]

That being said, my own opinion is that there is a lot of value in the "Hallas Method".  I am slightly worried that faint stars might be erased or have their brightness reduced by the noise reduction step - properly scaled darks don't have this effect.  However, I'm open minded enough to want to pick the best of the processing methods that are available to us - so to that end I'm planning to give the "Hallas Method" a proper try in the near future for comparison purpose with my current processing pipeline but first I'll have to do some serious acquisition with a much greater level of dithering than I currently use.

 As you said, the "Hallas Method" raises controversy and I for one was quite intrigued by it. But Ivo Jager, developer of StarTools, was not quite so impressed (http://forum.startools.org/viewtopic.php?f=4&t=634). Though I am no expert by any stretch, I think Ivo has a point (several, in fact) and his last comment "..(Tony Hallas) is doing a lot of budding astrophotographers a great disservice here" is a warning to those starting out in processing to stick to mainstream methods and master those before trying others.

If it was simple to process out the noise in PhotoShop (or other such software) then we would not be using the likes of PixInsight, StarTools, and others. In any case, I thought processing individual sub-frames prior to stacking was accepted as bad practice...for precisely the reasons Ivo Jager argues?

That said, I must admit that it was through watching Tony hallas' video that the value of dithering was brought to my attention. I thought I had been dithering my lights for the last 2 years (my capture software stated that it could do dithering!) only to discover I had never turned on the feature.

We live and learn.

[orion25]

Great discussion! I'm still learning this imaging game and it's great to read threads like this to consider different points of view 

Reggie

[nmoushon]

 As you said, the "Hallas Method" raises controversy and I for one was quite intrigued by it. But Ivo Jager, developer of StarTools, was not quite so impressed (http://forum.startools.org/viewtopic.php?f=4&t=634). Though I am no expert by any stretch, I think Ivo has a point (several, in fact) and his last comment "..(Tony Hallas) is doing a lot of budding astrophotographers a great disservice here" is a warning to those starting out in processing to stick to mainstream methods and master those before trying others.

If it was simple to process out the noise in PhotoShop (or other such software) then we would not be using the likes of PixInsight, StarTools, and others. In any case, I thought processing individual sub-frames prior to stacking was accepted as bad practice...for precisely the reasons Ivo Jager argues?

Thanks for this link. I had not seen it before but it has agreed with some of my suspicions...at least on the PS and processing side of the video. I always thought it strange how he did his techniques in PS. Especially processing before stacking.....that I never found to make sense.

Though I do think that the biggest thing that Tony has going for him (and that Ivor I think is missing) is that he is using a very specific type of AP images for his examples. All his examples are very wide field images and even nightscape images. These images are only using relatively short exposures that are not guided either. So very high ISO with short exposures and very few total subs. The way this type of AP imaging is both collected and processed VERY differently than how most AP imagers practice the hobby. 

Now dithering is still very new to me so I'm still hesitant on trying to speak on it either way. But from my limited knowledge of it I do still feel that dithering does have an advantage. Now Ivor brings up some good points that its similar to the methods of using Drizzle. Again though I think that in a typical AP image at FL of 400mm+ Ivor might be right. But with wide angle AP in the 14mm-100mm I'm not convinced that it would have the same effect. The detail and resolution isn't there from the beginning. Again I've never used either do my perspective of both could be wrong.