Help needed with Darks/Temperature with DSLR

[Rico]

I'm a tad confused with the use of darks in my work flow. 

I use an unmodded/uncooled DSLR. I tend to take lights and then follow that up with the same number of darks (same ISO and exposure setting). The other night I didn't manage to get the darks going and so I decided to proceed with them the following night (was hoping the temperature would be roughly the same, which it was ... give or take. I went ahead and used the darks in the DSS and the difference it then makes seems questionable . 

I had a close look at the EXIF camera temperature and they seem to vary quite a bit between images, but on average stay around 16c.

My questions are

1) how accurately do I need to match the temperatures?

2) If the temperatures varies during a session, do I need the same combination of temperatures in the darks?

3) How accurate does the temperature have to match?

4) Does the EXIF temperature measure the temperature at the start or at the end of the capture, in which case what if the temperature varies during a long exposure?

5) convert all my RAW files to TIFF before I use them in DSS. I've noticed that there is a 'blacks' slider which defaults at 5 in camera raw. Do I need to change this for the lights and/or the darks and do I have to be consistent between the two?

6) Do I even really need DARKS?

I have attached crops of a 8x600s image  (processed in PS) -  i've tried to match colours as much as I can.

Without Darks

With Darks

Thanks

[Bizibilder]

I have found that with my DSLR (A modded Canon 1000D) darks just add to the noise so I have started processing without them.  I seem to get much better results.

When I did use them I built up a library of darks taken (usually on cloudy nights) at about 5°C intervals and used the nearest to process my pictures.

[Kropster]

As far as point 5 goes, it must surely be better to leave the files as RAW....the original in camera format.....nothing added by processing or conversion.

[Rico]

I have found that with my DSLR (A modded Canon 1000D) darks just add to the noise so I have started processing without them.  I seem to get much better results.

When I did use them I built up a library of darks taken (usually on cloudy nights) at about 5°C intervals and used the nearest to process my pictures.

I 'm starting to build up a library as well ... although I'm not sure if i'll use them at this rate. 

As far as point 5 goes, it must surely be better to leave the files as RAW....the original in camera format.....nothing added by processing or conversion.

I think I tried using the raw files once in DSS and it choked. I'll try it again and see if its compatible. Thanks

[Kropster]

I always use RAW in DSS and never have any problem.

Maybe you should re install it.

[Rico]

With respect to point 5, I just processed it in DSS using the RAW files and its seems to have done some funny things to image. I'm really struggling to stretch the image properly. I'm far happier with the results I get using TIFF files ( which aren't compressed by the way)

[AndyKeogh]

I'm a tad confused with the use of darks in my work flow. 

I use an unmodded/uncooled DSLR. I tend to take lights and then follow that up with the same number of darks (same ISO and exposure setting). The other night I didn't manage to get the darks going and so I decided to proceed with them the following night (was hoping the temperature would be roughly the same, which it was ... give or take. I went ahead and used the darks in the DSS and the difference it then makes seems questionable . 

I had a close look at the EXIF camera temperature and they seem to vary quite a bit between images, but on average stay around 16c.

My questions are

1) how accurately do I need to match the temperatures?

2) If the temperatures varies during a session, do I need the same combination of temperatures in the darks?

3) How accurate does the temperature have to match?

4) Does the EXIF temperature measure the temperature at the start or at the end of the capture, in which case what if the temperature varies during a long exposure?

5) convert all my RAW files to TIFF before I use them in DSS. I've noticed that there is a 'blacks' slider which defaults at 5 in camera raw. Do I need to change this for the lights and/or the darks and do I have to be consistent between the two?

6) Do I even really need DARKS?

Thanks

You do not say what camera you are using  so let me give you my view on your issues.

1) always use RAW

2)  process in DSS with and without darks.  Have a look at the sky background figure for both runs.  you should find a considerable improvement when you use darks  unless of course you are imaging from a really dark site.

3) Do not stretch in DSS use a proper package like CS or whatever else you have that allows you stretch each channel independently.

4)  As you have an unmodified DSLR then at this time of year  temperature does not change much.  I shoot my darks , flats and dark flats around 6am  and are generally complete by the time I am washed dressed and fed   How do you take your darks?  I have some lens covers which are not completely lightproof.  Are you using a scope or camera lens?

5) Use the native format of your camera.  I converted some Canon Raw to Tiff files when DSS did not process my image size correctly  and that process added a lot of noise.

HTH

Andy

[AndyKeogh]

As a further thought please detail what equipment you are using and what exposure ties you are using.

Andy

[Rico]

I'm using a Canon 7D mounted on a SW ED80. Exposure lengths vary, but have recently been using 10 min subs.

This image is stacked using TIFF (converted from CR2 in Camera RAW) in DSS and processed in Photoshop (8x600s). No Darks have been included in DSS. I'm really struggling to get the same dynamic range if i use the raw files in DSS. I think you're right about it adding noise to image, but this seems to be outweighed by the improvement in the way it deals with this file format ... from my initial impression.

Stupid question ... how do i check the sky background figures?

[AndyKeogh]

I'm using a Canon 7D mounted on a SW ED80. Exposure lengths vary, but have recently been using 10 min subs.

Stupid question ... how do i check the sky background figures?

It is the second last column in the DSS processing screen.  Immediately left of the number of stars in each image.

At 10 min subs, how do you guide?

How do you take your darks?

Have you looked at the histogram of your darks in PS ?

[Rico]

Sky background reads about 40% using the TIFF files and 29% if use the CR2 files directly. 

I have a guider working from PHD.

I take my darks by adding the telescope lens cap. The histogram for my darks are very heavily shifted to left ... the black slider has to be set to 0 to see any sort of peaks in the RGB channels. I don't know what exactly it should look like anyhow.

[AndyKeogh]

Sky background reads about 40% using the TIFF files and 29% if use the CR2 files directly. 

I have a guider working from PHD.

I take my darks by adding the telescope lens cap. The histogram for my darks are very heavily shifted to left ... the black slider has to be set to 0 to see any sort of peaks in the RGB channels. I don't know what exactly it should look like anyhow.

The 40% figure seems high to me.  Suggests you might have over exposed your images.

What happens to the sky background when you process your images with your darks?

Have you set your black point to 0 when using darks?

I have just checked some 5 min subs I took with my Megrez 90 , a Canon 50D at ISO 1600  and an IDAS-D1 LP filter.  My sky background is 17% and drops to 5.8% with my 31 darks added.

Andy

[lensman57]

Forget about using darks for now. What you have to learn is that with DSLR AP, dithering is essential and negates the use of the troublesome darks. Set your capture software (I use APT and Nebulasity 3 for capture and both can dither with PHD guiding) to very high or even extreme dither during capture. If you PA is not spot on you may even enjoy what is called natural dithering.  The 2nd point to learn is that you need at least 100 and preferably 200 bias frames with a DSLR to make as noiseless a master bias as possible. Take about 100 flats for the same reason and good flats are essential if you want to extract detail from long exposure AP, so learn to take  them correctly. They are a pain to do but as I said flats are essential. Now with a large number of subs well dithered and good calibration frames you would not need darks as these are extremely temperature sensitive and to do them correctly you need set point cooling such as the ones used in CCDs. I know some people leave the camera in the fridge and so on and my self on occasions have left the camera in the garage taking darks at night but these will not give good darks.  Hope this helps.

A.G

[Rico]

Andy ... if I use the RAW files and use the darks during stacking it drops the Sky background to 8%. Black point has been set to 0. What I didn't realise is that camera raw adds some sharpening, brightening, noise reduction etc before it converts to TIFF. To some extent this made it easier for me to process the image later in PS. Having now had a better look at the output file from DSS using the RAW files ... I can start to see how I can best process the image ... however, its does seem to require a hell of a lot more steps than before. Thanks for the advice.

Lensman ... thank you for your input!! I may have to give the garage a go as well!! Great idea!! Flats and bias frames are not some thing I've done yet. I've only been doing this for 2-3 months. A lot to take in! Thanks for the advice on dithering I'll see if I can incorporate that into my imaging. Do bias frames need to be at the same temp?

Thanks

[lensman57]

Andy ... if I use the RAW files and use the darks during stacking it drops the Sky background to 8%. Black point has been set to 0. What I didn't realise is that camera raw adds some sharpening, brightening, noise reduction etc before it converts to TIFF. To some extent this made it easier for me to process the image later in PS. Having now had a better look at the output file from DSS using the RAW files ... I can start to see how I can best process the image ... however, its does seem to require a hell of a lot more steps than before. Thanks for the advice.

Lensman ... thank you for your input!! I may have to give the garage a go as well!! Great idea!! Flats and bias frames are not some thing I've done yet. I've only been doing this for 2-3 months. A lot to take in! Thanks for the advice on dithering I'll see if I can incorporate that into my imaging. Do bias frames need to be at the same temp?

Thanks

Strictly speaking yes but these are easiest of all the calibration frames to take so do them at the end of the session or during cloud cover and then once stacked delete the subs to save space. Again I stress flats are essential so is dithering with a DSLR but not darks.

A.G

[Jammy]

With respect to point 5, I just processed it in DSS using the RAW files and its seems to have done some funny things to image. I'm really struggling to stretch the image properly. I'm far happier with the results I get using TIFF files ( which aren't compressed by the way)

I tend to use TIFF files.  I've had problems with RAW files in DSS too.

[Rico]

I tend to use TIFF files.  I've had problems with RAW files in DSS too.

I solved this by making sure that under the 'RAW settings' in DSS the black point is set to 0. As I said before, converting it to TIFF does some processing along the way, so if you use the RAW files in DSS, more steps will have to be taken afterwards to improve sharpness, contrast, etc... It's a little more time consuming, but I'm trying to figure it out how much I gain from this. 

[dph1nm]

process in DSS with and without darks.  Have a look at the sky background figure for both runs.  you should find a considerable improvement when you use darks  unless of course you are imaging from a really dark site.

All darks do is remove the bias level - which is irrelevant to the true sky background.

NigelM

[Kropster]

All darks do is remove the bias level - which is irrelevant to the true sky background.

NigelM

I thought Darks also got rid of hot pixels, or do Bias and Flats do that as well?

[lensman57]

I thought Darks also got rid of hot pixels, or do Bias and Flats do that as well?

No, Bias and Flats do not get rid of hot pixels. Any decent stacking software has a cosmetic correction tool, this is the one that in conjunction with dithering and sigma clipping gets rid of the imperfections.

With a DSLR there is no short cuts I am afraid, you do need a lot of subs and a large number of calibration frames to get good clean noise free masters otherwise they just induce more noise into the equation. So with a DSLR it is better to do 25~30 300s rather than 10 or 12 600s  even though there may not be enough signal to pull the faint stuff out but at least you can have a good go at stretching what you have due to relative lack of noise. With regret this is where DSLR looses big time to a cooled CCD with set point cooling.

A.G

[Kropster]

No, Bias and Flats do not get rid of hot pixels. Any decent stacking software has a cosmetic correction tool, this is the one that in conjunction with dithering and sigma clipping gets rid of the imperfections.

With a DSLR there is no short cuts I am afraid, you do need a lot of subs and a large number of calibration frames to get good clean noise free masters otherwise they just induce more noise into the equation. So with a DSLR it is better to do 25~30 300s rather than 10 or 12 600s  even though there may not be enough signal to pull the faint stuff out but at least you can have a good go at stretching what you have due to relative lack of noise. With regret this is where DSLR looses big time to a cooled CCD with set point cooling.

A.G

So, just to clarify, darks play no part in getting rid of hot pixels....they just help with black levels?

[lensman57]

So, just to clarify, darks play no part in getting rid of hot pixels....they just help with black levels?

They can help clear them but they are more trouble than they are worth. A bad pixel map will do a much better job without the hassle. Unless your darks are at very close temp to the lights avoid using them.

A.G

[Kropster]

I seem to get a lot of hot pixels in the RAW files.

Are these taken out in DSS processing? (sorry about the endless questions!)

[sharkmelley]

The subject of darks is one that tends to generate a lot of confusion - especially when related to DSLRs - because of the lack of sensor cooling and especially the lack of set point cooling.

Here are a few pointers from my own experience.

Hot pixels are pixels that are more or less saturated in every frame.  A hot pixel map is the best way to get rid of these

There are other pixels (I call them warm pixels) which are consistently brighter than their surroundings - this is what the darks get rid of.  It is what is known as coherent noise i.e. the component of thermal noise that is constant from frame to frame.

There are other pixels that look bright occasionally - this is just the random nature of photons/electrons.  Only stacking gets rid of these.

To work well, darks need to be taken at a sensor temperature of within 1C of the lights.  Ignore the temperature recorded in the DSLR EXIF file - this never corresponds well to the dark current in my experiments.  An additional complication is that the dark current is continually changing during an imaging session - during a typical session the dark current will increase by a factor of 4 as the sensor gradually warms up.  So the later exposures will have far more thermal noise than the earlier ones.  The conclusion is that you have no hope of manually matching darks to lights - even with a "darks library" - because you need a different temperature of dark for each exposure.

So use the dark optimisation, available in most processing packages.  For each light frame, it scales the master dark frame to best subtract out the coherent noise.  In DSS, make sure "Dark Optimisation" is switched on.  It calculates a multiplier (in the range 0 to 1) for the dark frame using "Entropy-based Dark Frame Subtraction". Further details and a link to a technical article are found here:

http://deepskystacke...h/technical.htm

All DSLR users should have this DSS option switched on.

There is one other important point to get this right. DSS scales the dark frame down (and not up) during dark optimisation. Therefore, if you are using this option, the darks used must be taken at the end of an imaging session on a warmish night and not during a coldish night.  In my case I have a single set of darks that I took at room temperature.  I shoot a new set a couple of times a year as the sensor characteristics evolve.

Mark

[Rico]

Very interesting Mark. I'll read the paper later. This would change things dramatically for me as darks can be very time consuming!!