Calibration frames; how many do you guys use, and is my understanding correct?

[Trippelforge]

I have been imagine for a few years, yet have never taken calibration frames. It's just something I never took the time to do, perhaps due to laziness or maybe because I was happy enough with my images. But now that my son has joined up in the hobby with me, we want to hit the limits of our equipment. So I think it's time for me to actually take the extra steps for improvement.

With that being said I am getting FLOODED with information / opinions on how to deal with calibration frames. Over on cloudy nights there have been some heated debates on the amount of frames overall. And there is a HUGE divide... some are saying 15-20 darks, and some 200+... All I want is a simple metric to start with.

First of all am I understanding the reasoning behind each type?

1. Darks help eliminate noise, are re-usable, and you simply cap off the scope to take them. Also they should be taken at the same sensor temperature (and ISO) as the lights. 

2. Flats are for vignetting and dust spots. I need to use a t-shirt to cover the scope, and the images need to peak at the middle of the histogram. The imaging train cannot be moved afterwards, hence These are not re-usable. Also I have to use an even light source, or the sky at dusk / dawn.
 

3. Bias frames simply help read-out noise. Cap the camera or scope and take them as fast as possible, you can re-use them. 
 

If I am correct; how many frames do I need? And how long should each exposure be for each type?  

Apologies on the dead horse topic, I am just getting so many different opinions and wanted to ask here. 

 

[tooth_dr]

11 minutes ago, Trippelforge said:

I am just getting so many different opinions and wanted to ask here. 

You are only going to get more opinions here too though.

You have also missed out dark flats. These are used to calibrate you flats, similar to a dark but same time as your flats.

I’m not able to see what equipment you are using but if your camera has set point cooling then just keep the lights the same temp all the time and take your calibration data as the same temp (same gain, same offset, etc).

I use 30 of each type for my calibration data.  That works for me but I suspect I wouldn’t see any difference with 25 or 100 frames.

Flats imo are the most important calibration frame, they make processing a lot easier if applied correctly.

17 minutes ago, Trippelforge said:

All I want is a simple metric to start with.

Take 30-50 of each.  Process your data and look at it.  Speaking from personal experience, the number of calibration frames won’t make any tangible difference, it’s what happens to the stacks afterwards where it all goes wrong (and occasionally right!)

[Trippelforge]

9 minutes ago, tooth_dr said:

You are only going to get more opinions here too though.

You have also missed out dark flats. These are used to calibrate you flats, similar to a dark but same time as your flats.

I’m not able to see what equipment you are using but if your camera has set point cooling then just keep the lights the same temp all the time and take your calibration data as the same temp (same gain, same offset, etc).

I use 30 of each type for my calibration data.  That works for me but I suspect I wouldn’t see any difference with 25 or 100 frames.

Flats imo are the most important calibration frame, they make processing a lot easier if applied correctly.

Take 30-50 of each.  Process your data and look at it.  Speaking from personal experience, the number of calibration frames won’t make any tangible difference, it’s what happens to the stacks afterwards where it all goes wrong (and occasionally right!)

Aren't dark flats most important with dedicated / cooled astro camera's (read that on a random blog)? I am shooting with a DSLR (Nikon D5300). 

Are you saying that the amount of frames really doesn't make that big of a difference? One question, how long of an exposure should each dark and flat be? I didn't see that mentioned anywhere. 

[tooth_dr]

2 minutes ago, Trippelforge said:

Are you saying that the amount of frames really doesn't make that big of a difference?

In my experience, you’re unlikely to see any difference in your image between 30 frames and 300 frames.  Very easy to check this theory though…..

7 minutes ago, Trippelforge said:

One question, how long of an exposure should each dark and flat be?

Darks are the same length as your lights. Flats are literally whatever exposure is required to get the histogram in the right place.  Could be 0.5s for example. 

 

4 minutes ago, Trippelforge said:

Aren't dark flats most important with dedicated / cooled astro camera's

Yes, but you hadn’t mentioned your camera in your post!

5 minutes ago, Trippelforge said:

I am shooting with a DSLR (Nikon D5300)

This changes your calibration pattern.  It’s been a while since I used my DSLR but I’d drop darks, make sure you dither and use bias and flats.

 

You could be building your calibration library now 

[WolfieGlos]

As a fellow DSLR user, this is what I do:

1) Ignore darks. They can make a difference though if you are not dithering. A lot of people say you can be within 5 degrees of the lights temperature, and I found this to be true. They actually masked walking noise that I was suffering from early on. I once used 30 degree darks with 15 degree lights….it was not a pretty sight!

2) Bias, set your camera to the same ISO, fastest shutter speed (mine is 4000th second), put the lens cap on and shoot away. I always use ISO 400, and I’ve used the same files for the past 12 months. I have 50 of them.

3) Flats - put your camera on Av mode, using the T-shirt, flat panel, etc and shoot away. I used to use 50, but have been using 35 lately and notice no difference. Although technically you aren’t supposed to use them more than once, I keep my scope and camera in the same setup and use them across a few sessions. Usually I change after 3 or 4 sessions. 

[vlaiv]

1 hour ago, Trippelforge said:

1. Darks help eliminate noise, are re-usable, and you simply cap off the scope to take them. Also they should be taken at the same sensor temperature (and ISO) as the lights. 

No, they don't eliminate the noise - they eliminate dark current signal. Noise remains in the image.

They are reusable only if have set point cooling and can reproduce temperature.

If you use them, bias files are not necessary as bias signal is contained in dark subs

1 hour ago, Trippelforge said:

2. Flats are for vignetting and dust spots. I need to use a t-shirt to cover the scope, and the images need to peak at the middle of the histogram. The imaging train cannot be moved afterwards, hence These are not re-usable. Also I have to use an even light source, or the sky at dusk / dawn.

That is pretty much correct - include flat darks for best results (darks that match flat exposure and which you subtract from flats).

Flats are reusable if you have permanent setup or have electronic filter wheel with good repeatability or OSC sensor and you don't dismantle your optical train. If you for example pack after each session but leave the scope and camera attached as a single unit - you can reuse flats

1 hour ago, Trippelforge said:

3. Bias frames simply help read-out noise. Cap the camera or scope and take them as fast as possible, you can re-use them. 

Calibration files don't remove noise - they remove signal. Bias files remove bias signal and they can be used if:

1. You have modern DSLR that has dark subtraction built in. This will remove dark current without bias so you manually have to remove bias afterwards

2. You plan on using dark scaling / dark optimization when calibrating subs

3. You use very short flat exposures then you can use bias files instead of flat darks

I advocate use of larger number of calibration subs. As much as you can shoot without too much inconvenience.

Calibration subs don't remove noise - but they do introduce new noise in the image. More calibration subs you have - less new noise you'll introduce into final image.

[Trippelforge]

17 minutes ago, WolfieGlos said:

As a fellow DSLR user, this is what I do:

1) Ignore darks. They can make a difference though if you are not dithering. A lot of people say you can be within 5 degrees of the lights temperature, and I found this to be true. They actually masked walking noise that I was suffering from early on. I once used 30 degree darks with 15 degree lights….it was not a pretty sight!

2) Bias, set your camera to the same ISO, fastest shutter speed (mine is 4000th second), put the lens cap on and shoot away. I always use ISO 400, and I’ve used the same files for the past 12 months. I have 50 of them.

3) Flats - put your camera on Av mode, using the T-shirt, flat panel, etc and shoot away. I used to use 50, but have been using 35 lately and notice no difference. Although technically you aren’t supposed to use them more than once, I keep my scope and camera in the same setup and use them across a few sessions. Usually I change after 3 or 4 sessions. 

I don't dither right now as I don't know how. I know what it is, but software wise I am unsure on how to control it. I have one program (by Celstron) to move the mount, and I use NINA to control everything else. I guess I need to dig more into that as Vlaiv pointed out that they aren't re-usable due to temperature fluctuations. 

AV mode? You shoot it in video?  

 

15 minutes ago, vlaiv said:

No, they don't eliminate the noise - they eliminate dark current signal. Noise remains in the image.

They are reusable only if have set point cooling and can reproduce temperature.

If you use them, bias files are not necessary as bias signal is contained in dark subs

That is pretty much correct - include flat darks for best results (darks that match flat exposure and which you subtract from flats).

Flats are reusable if you have permanent setup or have electronic filter wheel with good repeatability or OSC sensor and you don't dismantle your optical train. If you for example pack after each session but leave the scope and camera attached as a single unit - you can reuse flats

Calibration files don't remove noise - they remove signal. Bias files remove bias signal and they can be used if:

1. You have modern DSLR that has dark subtraction built in. This will remove dark current without bias so you manually have to remove bias afterwards

2. You plan on using dark scaling / dark optimization when calibrating subs

3. You use very short flat exposures then you can use bias files instead of flat darks

I advocate use of larger number of calibration subs. As much as you can shoot without too much inconvenience.

Calibration subs don't remove noise - but they do introduce new noise in the image. More calibration subs you have - less new noise you'll introduce into final image.

So is it possible to use solely flats to get rid of some dust spots I have? I will be tight on time tonight but would love to get rid of them. Ya this entire thing is seemingly more complicated than I thought. The dark re-use situation makes sense... and it kind of sucks due to having to re-do that every night. =(

 

 

[WolfieGlos]

4 minutes ago, Trippelforge said:

I don't dither right now as I don't know how. I know what it is, but software wise I am unsure on how to control it. I have one program (by Celstron) to move the mount, and I use NINA to control everything else. I guess I need to dig more into that as Vlaiv pointed out that they aren't re-usable due to temperature fluctuations. 

AV mode? You shoot it in video?  

Ah sorry, Av on a Canon is Aperture Priority. It will automatically set the shutter speed based on the ISO you have chosen.

I use PHD 2 for guiding and control my mount with Nina with dithering. If you are using the sequencer to control it, there is a tab on the far right to enable it. You will need to configure a setting for how much to move your scope, but that’s pretty much it.

[vlaiv]

15 minutes ago, Trippelforge said:

So is it possible to use solely flats to get rid of some dust spots I have?

Thing with flats is that they work properly when applied to light signal only. You can't have other signals present in the image or master flat in order for it to work.

Dust and vignetting reduce amount of light by some percentage and in order to get the right amount of light - you need to divide with that percentage again (first time it multiplies and second time you divide when calibrating and two cancel out).

But this whole thing works - only when there is no other signal present, otherwise, it won't work completely - or it will either over or under correct.

For example - say that you have 600 units of light and dust shadow only passes 75% so you end up with only 600 * 0.75 = 450 units of light hitting your sensor. But you want to correct that and you record master flat which records 0.75 (this is for purpose of demonstration - it records other values but this is how it essentially works).

Now you divide your image with flat and get 450 / 0.75 = 600

All is good, right?

But what happens if you have say some dark or bias signal that you have not removed from your image?

This means that instead of 450 you actually recorded 470. 450 is light signal and 20 is some other signal - be that dark signal or bias signal, it does not matter.

Now when you try to correct with flat - you get 470 / 0.75 = 626.6666

We have brighter image than we should be having - this is over correction by flats

There is another case that can happen - maybe you forgot to remove bias signal from your master flat

In that case you won't have 0.75 as your master flat but something like 0.77 - 0.75 being light part and 0.02 being bias part.

Now if we try to correct we have 450 / 0.77 = 584.41...

This value is smaller than 600 - we have under correction.

We can even have mix of the two - if you don't remove residual signals from both lights and flats,

Just using flats will correct things - but it won't correct fully and how much you'll have issue because of this residual signal - depends on how big that residual signal is compared to light signal and flat signal.

If you are using DSLR - you can just use bias as there is dark compensation thing happening in the camera (most modern sensors do this).

[Trippelforge]

3 minutes ago, vlaiv said:

 

If you are using DSLR - you can just use bias as there is dark compensation thing happening in the camera (most modern sensors do this).

So simply shoot bias frames, no darks or flats? Ugh, sorry I am getting confused as this entire thing is much more complicated than I at first thought. 😃

[vlaiv]

3 minutes ago, Trippelforge said:

So simply shoot bias frames, no darks or flats? Ugh, sorry I am getting confused as this entire thing is much more complicated than I at first thought. 😃

It really depends on what data are you trying to calibrate.

I'll assume the following:

- you have DSLR

- your DSLR has automatic dark current removal. You can test this by taking two darks. One very short (say one second) and another rather long - say 30 seconds or so. Both images need to be true darks. Try to avoid any light leak or even IR leak (infrared can penetrate plastics). On DSLRs, be sure to use viewfinder cover to block any light getting in that way.

Best to take subs in very dark room without any light.

Once you have your subs - open them in any software that loads raw files and gives you access to raw data and simply measure average ADU value of them. If both subs have same average ADU value - you have automatic dark current removal (otherwise average ADU value of longer sub should be higher as it has more dark current signal).

If above is all true - then calibrate as follows:

- shoot lights

- shoot bias (which are darks at minimum exposure length)

- shoot flats

- match ISO setting between all three. When shooting flats - avoid clipping. Histogram should show three nice looking peaks at the center or 2/3 to the right.

Stack bias to master bias

subtract master bias from every flat and every light

Stack flats (with bias removed) into master flat

Divide each light (with bias removed) with master flat.

Ideally, software that you are using should do above for you if you provide it with said files automatically.

[Trippelforge]

6 minutes ago, vlaiv said:

It really depends on what data are you trying to calibrate.

I'll assume the following:

- you have DSLR

- your DSLR has automatic dark current removal. You can test this by taking two darks. One very short (say one second) and another rather long - say 30 seconds or so. Both images need to be true darks. Try to avoid any light leak or even IR leak (infrared can penetrate plastics). On DSLRs, be sure to use viewfinder cover to block any light getting in that way.

Best to take subs in very dark room without any light.

Once you have your subs - open them in any software that loads raw files and gives you access to raw data and simply measure average ADU value of them. If both subs have same average ADU value - you have automatic dark current removal (otherwise average ADU value of longer sub should be higher as it has more dark current signal).

If above is all true - then calibrate as follows:

- shoot lights

- shoot bias (which are darks at minimum exposure length)

- shoot flats

- match ISO setting between all three. When shooting flats - avoid clipping. Histogram should show three nice looking peaks at the center or 2/3 to the right.

Stack bias to master bias

subtract master bias from every flat and every light

Stack flats (with bias removed) into master flat

Divide each light (with bias removed) with master flat.

Ideally, software that you are using should do above for you if you provide it with said files automatically.

Thank you! That really cleared things up for me. I haven't ever even seen anything say an ADU rating before. Is that in the RAW image properties itself? Or do I need to open it in specific software?

Anyhow I appreciate all of the help, and thanks for taking the time to break it all down. 

[vlaiv]

2 minutes ago, Trippelforge said:

Thank you! That really cleared things up for me. I haven't ever even seen anything say an ADU rating before. Is that in the RAW image properties itself? Or do I need to open it in specific software?

Anyhow I appreciate all of the help, and thanks for taking the time to break it all down. 

ADU just means pixel value in some "analog / digital units" - term used for measured pixel values that are not photon or electron count but some number value after gain has been applied (ISO setting) and after A/D conversion has been performed by camera.

In this context you should read it as "value in 0-65535 range" that we get when we examine raw image file pixels.