Dark Frame Library

[Twisted Lip]

Hi all

Over the last few weeks I've seen more and more people refer to using a dark library. i've always avoided because I've wanted to get darks at exactly the same time (pretty much a best practice thing - nothing more).

Having said that I've struggled more and more with time recently and I'm coming around to the idea of creating a library. Given that I have a few questions:

1. how do people organise their libraries (in terms of ISO, exposure time, temp, etc)?
2. How likely is it the dark library frames will differ from darks taken at the time (in terms of dead/hot pixels)?
3. What is the general image impact of using a library?

Any help appreciated

Will

[davefrance]

Good question. I would think any library is somewhat of a compromise and not as good as real time dark frames. But what do I know, nothing probably. Will be interesting to see replies.

[ChrisLX200]

Standardise on one ISO for most objects (for extra high ISOs you don't use often then continue to take darks on the night). It's the temperature variation which is the real problem, the pixels on the chip don't change character (much) but the effect of temperature has a drastic effect. Using un-cooled cameras like DSLRs it becomes a real problem, and my collection of calibration files are only valid for a couple of ambient temperatures. If you use APT (for example) the camera temperature at the time of the exposure is included in the image file header so you can check which ones to use. It's also better if at all possible to use a battery elimininator for your caemra which keeps temperature and variation down a bit (the voltage regulator on the battery emits heat into the battery compartment which then heats the camera as a whole). Using the internal battery for long exposures will slowly heat the camera until it reaches equilibrium and then it cools down again between shots. This additional variation can amount to several degrees (3-5) which wrecks your objective of obtaining accurate calibrations.

ChrisH

[ollypenrice]

If you have a set point cooled CCD there is absolutely no reason to do them at the same time. It makes no difference whatever.

If using an uncooled DSLR, doing them at the same time is impossible because you are imaging at the time you need to take the darks! (Seriously, the chip temp varies even during an exposure and certainly during most nights. So 'same night' darks bring no guarantee.) If the temp varies much during the night then temp matched library darks might even be better. Anything you can do to stabilize your camera temperature will be an advantage. Cool box, etc.

Olly

[Twisted Lip]

If you have a set point cooled CCD there is absolutely no reason to do them at the same time. It makes no difference whatever.

If using an uncooled DSLR, doing them at the same time is impossible because you are imaging at the time you need to take the darks! (Seriously, the chip temp varies even during an exposure and certainly during most nights. So 'same night' darks bring no guarantee.) If the temp varies much during the night then temp matched library darks might even be better. Anything you can do to stabilize your camera temperature will be an advantage. Cool box, etc.

Olly

Thanks Olly, unfortunately I'm in option 2 and you can see the temperature variation even in the lights that I take! Its not so bad when the temp is around 0 outside but much above 4 degrees and the impact is markedly noticeable.

I've started collecting data with the darks when I take them and I'll see if I can build a library and experiment.

Cheers

Will

[Pete Presland]

So does the fact that you can not guarantee the dslr being in the exact orientation as when the lights were taken make any difference?

Sent from my GT-I9100 using Tapatalk 2

[ollypenrice]

So does the fact that you can not guarantee the dslr being in the exact orientation as when the lights were taken make any difference?

Sent from my GT-I9100 using Tapatalk 2

Absolutely none so far as darks are concerned. The chip is always orientated with the chip! It doesn't matter which way up the camera is.

It can matter with flats. If any vignetting is truly central then you might get away with turning the camera after shooting flats. Most of the bunny-creating dust is close to the chip, so it's on the filters which turn with the camera. However, the light cone may not be perfectly centred and this would be a problem. It's a case of 'try it' really. (Same with separate flats for each filter. When everything's clean I find this totally unnecessary and use the same flats on all filters. But sometimes you're unluck and have a filter specific bunny which needs its own flat.)

A little improvised cooling might help your DSLR. One of our guests used to have a few physio gel packs in the freezer and wrap them round the camera body, recycling them as they warmed up with cold ones from inside.

Olly

[IanL]

The difference between DSLR darks taken at the time and a library is negligible, assuming you either match the temperature well or use software to scale the darks.  I guess some hot/cold pixels may develop over time, but I haven't updated my dark library for 12 months and it works fine:

- Taking temperature matched darks is difficult to say the least. The sensor temperature from the probe is some guide, but it doesn't tell you very much about what's really happening during light or dark exposures (initially the temperature will climb with each frame, and then there will be a cycle of heating/cooling during/after each exposure which you just can't measure at all).  The best you can do is to try to match the ambient temperature and then use pauses between each dark to deal with any excessive increase from one to the next.  I put my camera in a mini-fridge with freezer packs as well, and use APT to take exposures with pre-programmed pauses (quite long ones of up to 20 mins).  By varying the pause length you can consistently get temps of between 9 and 13C.  Takes time though, you will usually end up with 20 or so good darks per day.

- You can further match the darks by measuring the noise in them.  Take the standard deviation of each dark (which is a reasonably good proxy for noise) and match them that way.  Should be much more accurate than the EXIF temperature since STDEV is proportional to the noise in the dark, whereas temperature is just the temperature at the start of the exposure, and doesn't necessarily reflect the temperature at the end or the rate of change of temperature during it.

- Stack as many darks as you can that are a close match - if using temperatures then maybe go for those that are +/- 1 degree of your desired temperature.  Try for at least 30 frames, and the more above that the better, up to perhaps 50 or 100 depending on your time and patience.

- If you're using simple software without dark scaling capabilities (e.g. DSS or similar) then you will also need to match the exposure lengths between darks and lights.

- I've not tried varying the ISO, but it seems sensible to use the same as your lights to avoid any other odd variations.

- You're still not going to get as good a result as you will with set-point cooling, especially since the DSLR also pre-processes darks and lights and reduces the appearance of dark current (but not dark current noise), see here: http://www.cloudynights.com/item.php?item_id=2786  I still think DSLR darks are useful though (see below for why).

- The best bet is to use processing software that supports dark frame scaling if you can.  PixInsight does it very well, and I understand that ImagesPlus can also do it (but haven't used the latter). Basically they calculate a multiplication factor to apply to the dark to make it match the light by checking how much noise there is left in the image after applying the dark. Least noise = correct scaling. Using this type of process, you can (within reason) ignore the temperature and the exposure length of the darks.  I currently use a master dark shot at ISO 400, 109 x 10 minute exposures with EXIF temps between 9 and 13 degrees C with all my images (all shot at ISO 400, but exposures of anywhere between 5 and 30 minutes and temperatures between 6 and 26 degrees C).

- People have suggested that DSLR darks are of no benefit and that we should do other things.  My tests have confirmed to my satisfaction that:

- Using just a master bias frame is not satisfactory, lots of hot pixels and bright/dark artefacts on a scale of 4-6 pixels remain.

- Using a master bias and a master dark deals with most of the hot pixels and the bright/dark artefacts.

- Using PixInsight's 'CostmeticCorrection' process actually does better than the master dark, as it removes the residual hot pixels as well, but you need to use your master dark as one of the inputs to the process to deal properly with the bright/dark artefacts.  If you just use the hot/cold sigma settings, you can kill the hot/dead pixels off, but many of the artefacts remain (or you can make the sigma values too restrictive to compensate for this and damage genuine image content).

[Pete Presland]

Absolutely none so far as darks are concerned. The chip is always orientated with the chip! It doesn't matter which way up the camera is.

It can matter with flats. If any vignetting is truly central then you might get away with turning the camera after shooting flats. Most of the bunny-creating dust is close to the chip, so it's on the filters which turn with the camera. However, the light cone may not be perfectly centred and this would be a problem. It's a case of 'try it' really. (Same with separate flats for each filter. When everything's clean I find this totally unnecessary and use the same flats on all filters. But sometimes you're unluck and have a filter specific bunny which needs its own flat.)

A little improvised cooling might help your DSLR. One of our guests used to have a few physio gel packs in the freezer and wrap them round the camera body, recycling them as they warmed up with cold ones from inside.

Olly

sounds like a "dark frame library" is the way to go, something else to sort out on the next not too distant cloudy night!

[PhotoGav]

Just reading this thread now... Thought I would add my experiences.

I have made a library of darks by putting the camera in the bottom compartment of the fridge (the meat section!), which is around 3C, and shooting 49 frames at 800ISO, my standard setting for my 60D, at all my usual exposure lengths. Laptop next to the fridge with BackyardEOS clicking through the exposures. I think I put a 30 sec delay between exposures - perhaps longer would be better? I have used the resulting master darks on a number of projects now and am very happy with the results. I still dream of a cooled CCD though! Check my flickr page for some of the pics - all the most recent targets, M45, M42, C49 & IC434, have used the fridge dark library. Maybe not the perfect pictures, but I'm certainly happy enough with the results given the equipment!

Good luck. I did get some strange looks from the rest of the family when I put the camera in the fridge though....!