darks and bias library

[bomberbaz]

I am absolutely sure this has been asked before but given how systems and equipment evolve constantly, thought it worth asking again.

I am going to build a library of darks and bias to save time, seeing as the camera is cooled, it is a simple (although tedious) task to be done during the daytime. 

My queries are firstly how many would you do? I have read in several places that 30 of any averaged stills is generally enough and I am happy to take that as a maximum point.

However when I get down to the 3 and 5 minute darks, we are getting to a serious amount of data, in this case 30 subs give 1'30" at 3 minutes and over 2'30" at 5.  This seems like over kill to me so what is anyone's rule of thumb. (I realise people will have their own methods, I am trying to find my own starting point)

Next the master dark or bias created during first use. Is it ok to keep reusing these? I have also read these can degrade over time although I don't understand how this can happen but it is still what I have read.

Cheers all

Steve

[AstroNebulee]

I generally do 30 of each. My flats and flat darks are done at the end of every session and don't take long to do. I don't use bias frames. 

I have reused my master dark files I've built in a library since getting my camera in January so only used about 3 times 😂, regarding the degradation, is it because the osc can change a bit over a  long period of time, so say take darks once a year maybe? Someone with better knowledge will be along shortly. 

Cheers 

Lee 

[vlaiv]

I use a lot of subs.

There is no rule of thumb really - same thing applies to darks and bias as it does to lights - noise goes down by square root of stacked subs.

You inject back this noise into every light you calibrate so it is worth spending some more time to minimize this noise you put back in.

Good thing is that you don't have to shoot all your subs at one time.

You can start one day and do 4h of darks. You can add next batch of darks in few days when you have the time. I usually just leave it in basement where temperature is lower so it is easier to get to target temperature for 4h at a time while I'm doing something else.

I've used up to 256 of subs in my stacks (256 is "default" for flats and flat darks). I start with 64 dark subs and build from there.

By the way, if you don't scale darks and don't optimize them - then bias are really waste of time. You only need flats, flat darks and darks for complete / proper calibration.

[bomberbaz]

1 hour ago, AstroNebulee said:

I generally do 30 of each. My flats and flat darks are done at the end of every session and don't take long to do. I don't use bias frames. 

I have reused my master dark files I've built in a library since getting my camera in January so only used about 3 times 😂, regarding the degradation, is it because the osc can change a bit over a  long period of time, so say take darks once a year maybe? Someone with better knowledge will be along shortly. 

Cheers 

Lee 

Query this Lee as I know you use the ZWO ASI Air like me. The system allows taking of light, dark, bias and flats in the autorun section. Where is the button for (or how do you take) dark flats.

I understand them but still cannot fathom how to do it using the ZWO system as it calculates the exposure and gain for flats for each time used, how do you replicate or am I missing something?  (probably the latter)

cheers

steve

[AstroNebulee]

20 minutes ago, bomberbaz said:

Query this Lee as I know you use the ZWO ASI Air like me. The system allows taking of light, dark, bias and flats in the autorun section. Where is the button for (or how do you take) dark flats.

I understand them but still cannot fathom how to do it using the ZWO system as it calculates the exposure and gain for flats for each time used, how do you replicate or am I missing something?  (probably the latter)

cheers

steve

I just take my flat darks under the bias autorun section, keep the same exposure time and gain as my flats and put the cap on the scope. I don't use the asiair auto setting on my flats, I take 2 second flats and flat darks. This on my flat brings the hump on the histogram around the middle mark. 

I use a homemade flat tracing panel box and worked out how many sheets of paper I needed to use to get the histogram where it is based on 2 sec exposures. 

As long as they are the same exposure time and gain each all is good., 👍

Edited April 26, 2022 by AstroNebulee
Spelling

[bomberbaz]

Good idea and thanks for the explanation. I shall look into emulating this idea if it works for you and sack the bias files as seemingly they are of little use.

[AstroNebulee]

4 hours ago, bomberbaz said:

Good idea and thanks for the explanation. I shall look into emulating this idea if it works for you and sack the bias files as seemingly they are of little use.

Sorry, earlier I should of said I set my flat darks on the dark plan in autorun still 2 seconds long at same gsin as flats, but dfirsny make any difference. 

Cheers

Lee 

[bomberbaz]

On 26/04/2022 at 16:16, vlaiv said:

I use a lot of subs.

There is no rule of thumb really - same thing applies to darks and bias as it does to lights - noise goes down by square root of stacked subs.

You inject back this noise into every light you calibrate so it is worth spending some more time to minimize this noise you put back in.

Good thing is that you don't have to shoot all your subs at one time.

You can start one day and do 4h of darks. You can add next batch of darks in few days when you have the time. I usually just leave it in basement where temperature is lower so it is easier to get to target temperature for 4h at a time while I'm doing something else.

I've used up to 256 of subs in my stacks (256 is "default" for flats and flat darks). I start with 64 dark subs and build from there.

By the way, if you don't scale darks and don't optimize them - then bias are really waste of time. You only need flats, flat darks and darks for complete / proper calibration.

Forgot to ask about this, what is meant by scaling darks and why are bias not reallyneeded.

I ask because ASI air, SIRIL, DSS etc all ask for them as part of their programme/

[vlaiv]

25 minutes ago, bomberbaz said:

Forgot to ask about this, what is meant by scaling darks and why are bias not reallyneeded.

I ask because ASI air, SIRIL, DSS etc all ask for them as part of their programme/

Bias as such is not needed in calibration workflow because it cancels out. Let me show you (and explain why that is).

Workflow involving bias goes like this:

You create master bias by stacking bias subs - let's call that MB

You create master dark by stacking (D - MB) subs - or in words - you take each dark and remove bias by subtracting master bias from it and then stack those - lets call that MD

You calibrate each light sub by

1) removing bias

2) removing dark

3) doing flat calibration (this step is not important for this so I won't bother with it)

calibrated light = light - MB - MD

Right? But let's just expand that expression a bit

calibrated light = light - MB - average(D - MB)

If you have a constant in average, you can pull that constant out as average(D-MB) is just (D1-MB) + (D2-MB) + (D3-MB) + .... + (DN-MB) all divided with N or number of subs. If we rearrange brackets we end up with D1+D2+D3+ ... + DN - MB -MB -MB (this repeats N times) all divided with N

That is further (D1+D2+...+DN - N * MB)/N

And from this you can see that you can pull MB out of that as N/N is 1

so above expression

calibrated light = light - MB - average(D - MB)

transforms into

calibrated light = light - MB - (average(D) - MB) = light - MB + MB - average(D) = light - average(D)

There we go

calibrated light = light - average(D)

No need to use bias to get calibrated light.

Why is bias then used in some cases?

Two reasons really. Sometimes bias is used instead of flat dark. This is strictly speaking wrong - but in many cases, flat exposure is short and dark current is low - it can work, or rather error will be too small to notice.

Second reason is if you plan to scale darks or use dark optimization (which is really the same thing except computer trying to figure out proper scale factor).

Sometimes you build dark library of say 10 minute exposures, but for some reason use 5 minute exposures when shooting target - you can still calibrate your light subs with mismatched darks if you scale them.

Dark current signal builds up linearly with time - which means that in 10 minutes it will be twice as strong as in 5 minute exposure, or 5 times as strong as in 2 minute dark exposure.

We can exploit this fact if our camera behaves properly (CCDs tend to behave good for this and CMOS cameras behave poorly - so I would say that this is mostly available for CCDs).

Dark subs contain dark current signal and bias signal. If we remove bias signal we can then multiply dark current signal (bias does not depend on time so we must not multiply it - we must first subtract it).

This is the reason why there is bias subtraction in above expression - but it is only needed if you plan to scale your dark current. If you plan to scale dark current - you can no longer take bias in front of the brackets and it won't cancel out. It is needed for whole thing,

Makes sense?

[ollypenrice]

I like 30 minute subs for CCD narrowband. Occasionally I like them for luminance as well. You have a problem finding time for 5 minute darks???   😁

Fortunately, I found that darks did not deliver what they were supposed to deliver, even with an old-school and very noisy, set-point-cooled, Kodak CCD. Of course I know the theory. I've read it a thousand times. I just found it didn't do, for me, what it said on the tin.

To put it simply, there is no way in the world that you will notice a difference in your final image between calibrating with 20 darks and calibrating with 50. Of all the things standing between you (or any of us) and a good image, it ain't that. Put your energy into more productive lines of thought.

Olly

[Anthonyexmouth]

I recently watched this by Adam Block when i was looking into using WBPP for PI, but it actually gives quite a good and understandable explanation of calibration frames. Something that i'd never actually seen in such a simple way before. 

 

 

[bomberbaz]

On 21/05/2022 at 12:23, vlaiv said:

Bias as such is not needed in calibration workflow because it cancels out. Let me show you (and explain why that is).

Workflow involving bias goes like this:

You create master bias by stacking bias subs - let's call that MB

You create master dark by stacking (D - MB) subs - or in words - you take each dark and remove bias by subtracting master bias from it and then stack those - lets call that MD

You calibrate each light sub by

1) removing bias

2) removing dark

3) doing flat calibration (this step is not important for this so I won't bother with it)

calibrated light = light - MB - MD

Right? But let's just expand that expression a bit

calibrated light = light - MB - average(D - MB)

If you have a constant in average, you can pull that constant out as average(D-MB) is just (D1-MB) + (D2-MB) + (D3-MB) + .... + (DN-MB) all divided with N or number of subs. If we rearrange brackets we end up with D1+D2+D3+ ... + DN - MB -MB -MB (this repeats N times) all divided with N

That is further (D1+D2+...+DN - N * MB)/N

And from this you can see that you can pull MB out of that as N/N is 1

so above expression

calibrated light = light - MB - average(D - MB)

transforms into

calibrated light = light - MB - (average(D) - MB) = light - MB + MB - average(D) = light - average(D)

There we go

calibrated light = light - average(D)

No need to use bias to get calibrated light.

Why is bias then used in some cases?

Two reasons really. Sometimes bias is used instead of flat dark. This is strictly speaking wrong - but in many cases, flat exposure is short and dark current is low - it can work, or rather error will be too small to notice.

Second reason is if you plan to scale darks or use dark optimization (which is really the same thing except computer trying to figure out proper scale factor).

Sometimes you build dark library of say 10 minute exposures, but for some reason use 5 minute exposures when shooting target - you can still calibrate your light subs with mismatched darks if you scale them.

Dark current signal builds up linearly with time - which means that in 10 minutes it will be twice as strong as in 5 minute exposure, or 5 times as strong as in 2 minute dark exposure.

We can exploit this fact if our camera behaves properly (CCDs tend to behave good for this and CMOS cameras behave poorly - so I would say that this is mostly available for CCDs).

Dark subs contain dark current signal and bias signal. If we remove bias signal we can then multiply dark current signal (bias does not depend on time so we must not multiply it - we must first subtract it).

This is the reason why there is bias subtraction in above expression - but it is only needed if you plan to scale your dark current. If you plan to scale dark current - you can no longer take bias in front of the brackets and it won't cancel out. It is needed for whole thing,

Makes sense?

Thanks @vlaiv for the above explanation, I get it all until the last (underlined) equation. However rather than mash my brain trying to really get my head around it, I will accept that you are right and they are not really needed.

However I am one of these people who will go out now and run comparisons because I want to see for myself. I guess that is the scientist part in me. I can accept what someone tells me is right but I won't fully believe it until I prove it right. 

On 21/05/2022 at 19:16, Anthonyexmouth said:

I recently watched this by Adam Block when i was looking into using WBPP for PI, but it actually gives quite a good and understandable explanation of calibration frames. Something that i'd never actually seen in such a simple way before. 

 

Anthony, now I know what WBPP is (google is ace), I shall watch this as I really do want to get my head around calibration frames fully. As above vlaiv's explanation is very detailed but lost me at the last minute.  Maybe this video will help a few pennies drop.

Cheers guys.

Steve

[Anthonyexmouth]

1 hour ago, bomberbaz said:

Thanks @vlaiv for the above explanation, I get it all until the last (underlined) equation. However rather than mash my brain trying to really get my head around it, I will accept that you are right and they are not really needed.

However I am one of these people who will go out now and run comparisons because I want to see for myself. I guess that is the scientist part in me. I can accept what someone tells me is right but I won't fully believe it until I prove it right. 

Anthony, now I know what WBPP is (google is ace), I shall watch this as I really do want to get my head around calibration frames fully. As above vlaiv's explanation is very detailed but lost me at the last minute.  Maybe this video will help a few pennies drop.

Cheers guys.

Steve

The video, well a little series explains it really well with diagrams. The wbpp series is quite long but worth watching and there are 3 or 4 that cover calibration frames but some are about DSLR so may or may not be of any use. 

 

 

 

[bomberbaz]

On 23/05/2022 at 23:18, Anthonyexmouth said:

The video, well a little series explains it really well with diagrams. The wbpp series is quite long but worth watching and there are 3 or 4 that cover calibration frames but some are about DSLR so may or may not be of any use. 

 

Finally watched this, very interesting stuff and I get more now the rationale behind flat darks. 

think I will make a similar flat panel to @AstroNebulee has below and then I can set my flats/flat darks to always run at a set level, makes sense.

Thanks Gentlemen.

On 26/04/2022 at 17:18, AstroNebulee said:

 

 

[Xilman]

On 21/05/2022 at 12:23, vlaiv said:

Second reason is if you plan to scale darks or use dark optimization (which is really the same thing except computer trying to figure out proper scale factor).

Sometimes you build dark library of say 10 minute exposures, but for some reason use 5 minute exposures when shooting target - you can still calibrate your light subs with mismatched darks if you scale them.

Dark current signal builds up linearly with time - which means that in 10 minutes it will be twice as strong as in 5 minute exposure, or 5 times as strong as in 2 minute dark exposure.

We can exploit this fact if our camera behaves properly (CCDs tend to behave good for this and CMOS cameras behave poorly - so I would say that this is mostly available for CCDs).

Makes sense?

Makes perfect sense and is exactly the reason I take and use bias frames with my CCD cameras.

My interest is photometry of objects which could be as bright as V=5 or as faint as V=20 --- a million-fold range in intensity. Exposures can range from well under a second to several hours. Most of the time I use 30s subs as that is a good compromise between noise and sensitivity and having to discard images because of cosmic rays, tracking errors, passing clouds, being Starlinked, and so on. In my case, standardizing of 30-second calibration images and scaling appropriately is precisely what I need.