Whats a DARK FLAT FILE?

[Ant]

Just putting some files into DSS and I keep meaning to ask...

I know what a FLAT file is.

I know what a BIAS file is.

I know what a DARK file is.

But what's a DARK FLAT file?

Thanks

Ant

[Beyond_Vision]

It's an exposure the same length as the flat frame but with the lens/scope covered up. It helps to reduce the noise in the flat frames.

Regards

Kevin

[Ant]

Ahhh that makes sense

Thanks Kevin

Ant

[tmarkuk]

Every time I open DSS I ask myself this same question, I'm glad someone else thought to actually voice it!

[MartinB]

It's an exposure the same length as the flat frame but with the lens/scope covered up. It helps to reduce the noise in the flat frames.

Regards

Kevin

So what is the difference between that and a dark then?

[Ant]

The flat frame is of an even field, with the camera decided the exposure length (AV mode).

So the Dark will match the Light in duration, but the Dark Flat will match the Flat in duration.

If I've got that right.

Ant

[jgs001]

That's right Ant. One word of caution... If you are using Bias frames you don't need dark flats, and vice versa. If you use both then weird things start to happen (I think it's down to the removal of the bias current twice but I've no clue about that level of stuff)

I used to take a sequence of dark flats as well as the Bias frames, but you can save some time not taking the dark flats and use the stored Bias frames instead. Thanks to whoever, I think it was Martin, that pointed out the Bias frames.

[MartinB]

Ah so it is just a dark but timed for the flat. OK.

[roundycat]

pedantic, picky old git that I am all the reference works refer to them as flat darks.

Dennis

[narrowbandpaul]

interesting, since light images experience fixed pattern noise, which is removed with flats, and since dark frames have dark fixed pattern noise, could one flatten a dark to remove dark fixed pattern noise.

Its worth an experiment......

paul

[RichardHurst]

On 31/05/2009 at 11:50, Beyond_Vision said:

It's an exposure the same length as the flat frame but with the lens/scope covered up. It helps to reduce the noise in the flat frames.

Regards

Kevin

But isn't that a Bias frame??

 

[happy-kat]

No as bias frames are same iso, lens cap on and fastest possible exposure length

[tomato]

The experts say to use darks, flats and bias frames with CCD sensors and darks, flats and flat darks  with CMOS sensors, from my experience this would appear to be sound advice.

[ollypenrice]

15 hours ago, tomato said:

The experts say to use darks, flats and bias frames with CCD sensors and darks, flats and flat darks  with CMOS sensors, from my experience this would appear to be sound advice.

Yes, the convenient shortcut with CCD is that a master bias can perfectly well double as a flat dark because the thermal noise built up in very short flat exposures is insignificantly different from that found in a bias. This isn't true for CMOS cameras.

Olly

[Martin Meredith]

This got me thinking about whether there is ever a context where a bias is actually necessary (as opposed to using a flat dark). The only one I can think of is when using 'thermal' frames, which I believe are used to scale a master dark collected at a long exposure so that it can be used at arbitrary exposures without having to collect darks at that exposure. In this situation the bias has to be subtracted from the dark itself to remove fixed pattern noise. Does anyone use thermal frames?

Martin

 

[vlaiv]

4 minutes ago, Martin Meredith said:

This got me thinking about whether there is ever a context where a bias is actually necessary (as opposed to using a flat dark). The only one I can think of is when using 'thermal' frames, which I believe are used to scale a master dark collected at a long exposure so that it can be used at arbitrary exposures without having to collect darks at that exposure. In this situation the bias has to be subtracted from the dark itself to remove fixed pattern noise. Does anyone use thermal frames?

Martin

 

That approach is viable with CCD sensors and it is good idea to shoot only the longest darks if you use multiple exposures and scale them for shorter ones as longest darks will have the least overall thermal noise (although it seems that the longest dark will gather the most thermal current and hence associated noise - it will be scaled and noise will be scaled as well - signal scales with time but noise with square root of that).

With cmos sensors it usually does not work as there are issues with bias files. I've tried 5 different CMOS cameras so far and not one had usable bias subs (for dark scaling).

Another place bias can be used is if you don't have set point cooling on your camera. In that case, it is good idea to try dark scaling (algorithm that tries to scale master dark to match that of sub it is calibrating) - again bias needs to be removed prior to dark scaling as bias signal does not depend on temperature.

Interestingly enough, DSLR CMOS sensors don't seem to be suffering from bias issues like dedicated astro CMOS cameras (or I could be wrong there).

[kirkster501]

A sanity check then guys for the CMOS world....  Do you have to take flat darks with a CMOS? With a CCD when PixInsight calibration routine scales a dark to the flats  it kicks off a warning about the dark having no correlation to the flats (because of the radically different exposure time).  This implies that darks with flats on a CCD do not matter that much.  Are we saying it *does* matter with a CMOS camera?  Why so?  PI does not know the light/flat was taken with a CCD or a CMOS does it?

Also, with a CCD the "normal" dark of say 20 mins can be scaled to a light of 10 minutes by using "half" of the dark.  Is this still the case with CMOS?

[vlaiv]

2 minutes ago, kirkster501 said:

A sanity check then guys for the CMOS world....  Do you have to take flat darks with a CMOS? With a CCD when PixInsight calibration routine scales a dark to the flats  it kicks off a warning about the dark having no correlation to the flats (because of the radically different exposure time).  This implies that darks with flats on a CCD do not matter that much.  Are we saying it *does* matter with a CMOS camera?  Why so?  PI does not know the light/flat was taken with a CCD or a CMOS does it?

I'll need clarification on this.

Flat darks (or dark flats) are in principle always needed - regardless of CMOS/CCD sensor type technology.

Flats work when they contain only light signal.

When we take regular sub, it contains numerous signal sources: 1. Bias signal, 2. Dark signal, 3. Light signal. Same happens with flats - except light signal being uniformly lit flat panel. In order to have only light signal remaining in our master flat file we need to remove both bias signal and dark signal (however small it is for short flat exposure - if we want to be completely correct).

We can do that in couple of ways (right and "wrong" ones):

1. Subtract Flat darks that contain both bias and dark signal - this is what we are saying here

2. Remove bias with master bias and then remove remaining dark signal with some means - like dark scaling or similar

3. Remove bias signal only - wrong in principle but in practice it will work 99% of the time since flat exposures are really short and cooled cameras have low dark current, so overall dark signal in such short time is minimal.

CMOS sernsor have issues with bias and it is best not to use bias if one can with CMOS sensors - for this reason, I always recommend number 1. option - for both CMOS and CCD - simply because it works properly.

9 minutes ago, kirkster501 said:

Also, with a CCD the "normal" dark of say 20 mins can be scaled to a light of 10 minutes by using "half" of the dark.  Is this still the case with CMOS?

No. Yes. Well it depends.

In principle - same thing. If you can remove bias than you are all OK - you can scale dark exposure as dark current depends on time (and temperature but we assume temperature is fixed in this discussion).

Problem is that one should not expect bias to be well behaved with CMOS sensors.

I've found at least 2 or 3 different scenarios where bias is not what one would expect with CMOS sensors.

1. Internal bias calibration

This is present on smaller sensors - usually planetary type cameras / earlier models. Each power cycle, sensor would perform internal calibration and some sort of automatic offset. This means that one can't expect bias to be on the same level between power cycles as offset can be (and often is) different. Solution was to shoot everything in same session without shutting down camera.

2. Multimode operation

For some reason offset in very short exposures was different than offset in long exposures. I think this was driver issue rather than sensor issue, but could be both - driver just trying to deal with sensor firmware operation.

It means that bias for very short exposure (like under one second - explained by ASI as internal camera clock being used to time exposure as opposed to "bulb" mode where application is timing exposure for subs longer than 1s) was different than bias for long exposure and bias subs are inherently shortest exposure possible.

I measured mean value of bias sub to be larger than dark sub on several occasions (and that can be true if bias is correct because dark sub = bias signal + dark current signal).

 

[kirkster501]

Thankyou @vlaiv for your time in explaining this.

So, in practise in the real world, what would one do with a CMOS to calibrate lights and flats in terms of what calibration frames are required ?

[vlaiv]

9 minutes ago, kirkster501 said:

Thankyou @vlaiv for your time in explaining this.

So, in practise in the real world, what would one do with a CMOS to calibrate lights and flats in terms of what calibration frames are required ?

You take a set of flat darks (dark flats - or any other word order ) and you create master from those (stack them). You take your flats and you stack them and subtract master flat dark - this creates master flat.

You create master dark like you normally would (no dark scaling so you need to match exposure of your lights exactly).

Or in formulas:

master_flat = stack_of_flats - stack_of_flat_darks

(flat darks match flats in everything)

calibrated_light = (light - master_dark) / master_flat

(darks match lights in everything).

You need flats, flat darks, darks to do proper calibration (with any camera, and CMOS in particular).

[jif001]

This may help ...

 

[Astro_mark_c]

For flat darks, do you switch it back to manual mode and set the sutterspeed to the length of your flats? (lens cap one).

[drivera]

I'm jumping on this post.. I'm new to this. 

I'm using a DSLR (Sony A7iii). As an example I'm going to take some pictures tonight from say 11pm-1.30am.  Can I / should I take my dark frames prior to me kicking off my session? Instead of getting up at 1.30am to then put the lens cap on and run a session of say 10-20darks, could I do this at 10pm with my dark frames? Or will the temp be too far off my imaging and won't work?

Trying to think of preserving my sleep on these weeknights haha.

[vlaiv]

15 minutes ago, drivera said:

I'm jumping on this post.. I'm new to this. 

I'm using a DSLR (Sony A7iii). As an example I'm going to take some pictures tonight from say 11pm-1.30am.  Can I / should I take my dark frames prior to me kicking off my session? Instead of getting up at 1.30am to then put the lens cap on and run a session of say 10-20darks, could I do this at 10pm with my dark frames? Or will the temp be too far off my imaging and won't work?

Trying to think of preserving my sleep on these weeknights haha.

That really depends.

If you don't have set point cooling and don't use advanced algorithms - best course of action is to do half / half. Shoot half of your darks prior to lights and half after. That will sort of average out darks / temperature difference.

If your camera has stable bias - well, then you can use advanced algorithms like dark optimization. This algorithm will try to scale darks automatically to compensate for temperature difference. In this case - it does not make much difference when you shoot your darks - and you can do it prior to session. In fact - you can have sets of master darks taken at different temperatures and use one that is closest to shooting temperature. This will save you time - so you don't have to take darks every session.

[drivera]

6 minutes ago, vlaiv said:

That really depends.

If you don't have set point cooling and don't use advanced algorithms - best course of action is to do half / half. Shoot half of your darks prior to lights and half after. That will sort of average out darks / temperature difference.

If your camera has stable bias - well, then you can use advanced algorithms like dark optimization. This algorithm will try to scale darks automatically to compensate for temperature difference. In this case - it does not make much difference when you shoot your darks - and you can do it prior to session. In fact - you can have sets of master darks taken at different temperatures and use one that is closest to shooting temperature. This will save you time - so you don't have to take darks every session.

Thanks for that. that's quite interesting.  Sorry to sound so naive how would I learn and understand about stable bias for my camera and then look at dark optimisation with different darks? or would that be irrelevant on a DSLR?