Enough with darks flats, they are useless

[lock042]

Hello,

Behind this provocative title we have wrote a post about dark flats.

Indeed, more and more, I see the term master dark flat all over the forums and I would like to explain why you shouldn't do it.

This is here: https://siril.org/2021/12/enough-with-dark-flats/

[tooth_dr]

@lock042 Is a dark flat detrimental to the final image, or is it useless because the bias does the same thing and therefore it is redundant?

[lock042]

From a pure noise point of view, dark flat adds more noise as we show in the link. However, it is probably negligible.

But why make unnecessary frames when you can avoid it with synthetic biases?

[inFINNity Deck]

Hi Cyril,

thanks for this interesting read! When I take flats (SW Esprit 150ED with ZWO ASI1600 Pro Cool and ZWO filters) I always do it in one sequence. To avoid too short exposures (which will cause banding due to the flickering nature of the flat panel I use), the exposure for luminance starts around 1s, as a result of which Red is taken with an exposure time of 2.5s, followed by Green (4s), Blue (11.5s), H-alpha (34s), S-ii (59s) and ending with approximately 67s for O-iii. In your article you wrote "However, a flat is an image taken in full light, at times very short generally less than 5s." Clearly this is not the case in my workflow, while you imply that masterdarkflats are useful for longer exposures. So I would like to turn the question around in the hope you can elaborate on that:

Upwards from what exposure-time should we use masterdarkflats?

Nicolàs

PS: the reason for my workflow is that I have an observatory and thus can permit to 'fire-and-forget': when I come back after several hours all subs have been taken. Same for darks...

Edited December 15, 2021 by inFINNity Deck
typo

[inFINNity Deck]

47 minutes ago, tooth_dr said:

@lock042 Is a dark flat detrimental to the final image, or is it useless because the bias does the same thing and therefore it is redundant?

Hi @tooth_dr,

Yes and no: Yes, because they add noise (depending on the amount of darks taken), and no as they will remove amp-glow.

A few years ago I did a test on the amount of darks, compared images corrected with 25 and 500 of them, to find out that 25 sufficed. The write-up of that test is on a Dutch forum, but should translate well when opened in Chrome: https://www.starry-night.nl/forums/topic/ruis-in-de-cmos-cameras-metingen-en-berekeningenn/#post-21015

Nicolàs

Edited December 15, 2021 by inFINNity Deck

[tooth_dr]

37 minutes ago, lock042 said:

From a pure noise point of view, dark flat adds more noise as we show in the link. However, it is probably negligible.

But why make unnecessary frames when you can avoid it with synthetic biases?

I dont use Siril, is this possible in other software?

[lock042]

13 minutes ago, inFINNity Deck said:

Clearly this is not the case in my workflow, while you imply that masterdarkflats are useful for longer exposures. So I would like to turn the question around in the hope you can elaborate on that:

Upwards from what exposure-time should we use masterdarkflats?

In fact that depends more of your ampglow (do you see ampglow flash in your masterflat?). Take a look at our figure 3: even if you do have some thermal signal you don't care. Don't forget your flat is full of light and the thermal signal is not a problem like it could be with your light frames.

[vlaiv]

Issue that you mention - one of injecting more noise with flat darks - is non issue.

We control amount of noise we inject back by choosing number of subs we use to create our masters. Want less noise? Use more subs for masters.

Problem with bias only approach is that you will have incorrect flat calibration.

Using bias instead of flat darks can cause flats to over correct. If you don't remove dark signal - it will also be subject to flat calibration - but it was not subject to attenuation by optical train in the first place - so you'll end up with uneven offset signal (if your dark is uniform -and worse if it contains amp glow).

There is a reason why flat darks are used - because it is proper calibration of data.

 

[lock042]

11 minutes ago, vlaiv said:

Want less noise? Use more subs for masters.

Or just a constant. You can't beat it.

11 minutes ago, vlaiv said:

Problem with bias only approach is that you will have incorrect flat calibration.

Wrong. We wrote a tutorial here: https://siril.org/tutorials/synthetic-biases/#appendix-understanding-how-the-flats-correct-the-lights if you want to take a look.

The appendix details computation.

11 minutes ago, vlaiv said:

There is a reason why flat darks are used - because it is proper calibration of data.

Absolutely not. The most important in the process is the level you subtract to the image to have a proper calibration of data (and offset has same level than dark, you can check by yourself).

Just test it and you will see. And please, post the results here.

Edited December 15, 2021 by lock042

[vlaiv]

2 minutes ago, lock042 said:

Wrong. We wrote a tutorial here: https://siril.org/tutorials/synthetic-biases/#appendix-understanding-how-the-flats-correct-the-lights if you want to take a look.

The appendix details computation.

It is not wrong - it is basic math.

If you have flat without removed dark - you have light signal and dark signal. Only light signal determines percentage of light - dark signal must be removed.

Say you have 70% vignetting at one pixel and your flat at that pixel is 7010ADU - 7000ADU being light and 10ADU being dark that you did not remove. Peak flat is 10000ADU or 10010 ADU because of dark (let's assume dark current is uniform).

So your scaled flat in that pixel is 7010 / 10010 = ~0.7003

There you go - we just got wrong flat - for 70% vignetting our flat will correct with 70.03%

It might not look like much - but once you stretch your data - that will show as improper correction

7 minutes ago, lock042 said:

Absolutely not. The most important in the process is the level you subtract to the image to have a proper calibration of data (and offset has same level than dark, you can check by yourself).

I have - many times. All of my cameras have measurable amount of dark current - and yes it corresponds for published dark current for those sensors.

Offset level is not the same as level of dark in any of my cameras.

[lock042]

17 minutes ago, vlaiv said:

It is not wrong - it is basic math.

You're right, but in your example you take a flat underexposed and a very high dark current to best match what you are saying.

Anyway, I take the example of a basic non cooled camera I have: Canon EOS 100D.

level of bias: 2048 (2^11).

level of dark (60s) : 2044.

It is the same especially when you compare to the flat. 

17 minutes ago, vlaiv said:

If you have flat without removed dark - you have light signal and dark signal. Only light signal determines percentage of light - dark signal must be removed.

Say you have 70% vignetting at one pixel and your flat at that pixel is 7010ADU - 7000ADU being light and 10ADU being dark that you did not remove. Peak flat is 10000ADU or 10010 ADU because of dark (let's assume dark current is uniform).

So your scaled flat in that pixel is 7010 / 10010 = ~0.7003

Then in this case take a constant that correspond to the value you want. We encourage users to test it. I'm sorry but it works, and it works well. Saving process time and disk space. Because yes, this is not a issue of noise.

Did you read the second link I gave you?

Edited December 15, 2021 by lock042

[vlaiv]

3 minutes ago, lock042 said:

Then in this case take a constant that correspond to the value you want. We encourage users to test it. I'm sorry but it works, and it works well. Saving process time and disk space. Because yes, this is not a issue of noise.

Why would I take a constant to replace actual dark current that might not be constant - or rather never is?

Why should I forcefully push what is essentially wrong method of calibration over right one?

7 minutes ago, lock042 said:

Anyway, I take the example of a basic non cooled camera I have: Canon EOS 100D.

Did you try this on any dedicated astronomy camera with set point cooling?

[lock042]

1 minute ago, vlaiv said:

Did you try this on any dedicated astronomy camera with set point cooling?

Yes. Example in the post are done with 294MC.

And dark of 300s has same level than bias, except in the ampglow flash

 

[vlaiv]

3 minutes ago, lock042 said:

And dark of 300s has same level than bias, except in the ampglow flash

Can you post mean ADU of 300s, 150s and bias that you measured in 294MC camera?

[lock042]

7 minutes ago, vlaiv said:

Why should I forcefully push what is essentially wrong method of calibration over right one?

15 minutes ago, lock042 said:

This not because you will repeat it's wrong that is really wrong :).

[scotty38]

4 minutes ago, lock042 said:

Yes. Example in the post are done with 294MC.

And dark of 300s has same level than bias, except in the ampglow flash

 

Is that from a 294mc pro as it looks nothing like I've ever seen in mine, just curious?

[vlaiv]

Just now, lock042 said:

This not because you will repeat it's wrong that is really wrong :).

You are correct - it won't be wrong just because I say it is wrong.

I've shown you very simple math that shows that you can't have correct flat if you don't remove offset.

You are further claiming that bias offset is the same as dark current offset, and that contradicts my measurements on numerous occasions.

It also contradicts common knowledge about dark current - why do we even have a term if it does not exist?

[lock042]

1 minute ago, vlaiv said:

Can you post mean ADU of 300s, 150s and bias that you measured in 294MC camera?

Mean is a non robust estimator. Median is a better one.

Here you have what you want. The level of a master dark at 5sec you can compare with the previous image: 

Then in the the table here you can see median evolution:

[lock042]

3 minutes ago, scotty38 said:

Is that from a 294mc pro as it looks nothing like I've ever seen in mine, just curious?

yes

[vlaiv]

2 minutes ago, lock042 said:

Then in the the table here you can see median evolution:

You are saying that standard deviation is rising as exposure time but mean / median value stays the same?

What capture software are you using?

[scotty38]

1 minute ago, lock042 said:

yes

ok thank you, My gain is 121 and offset 8  but will have a play with your settings just to see......

[lock042]

5 minutes ago, vlaiv said:

It also contradicts common knowledge about dark current - why do we even have a term if it does not exist?

It contradicts common beliefs. But it is normal and this is why I encourage people to make tests!!

 

2 minutes ago, vlaiv said:

What capture software are you using?

The dark at 300sec come from the 294MC and NINA:

23:58:12: Reading FITS: file DARK_300s_G120_O30_T-10C_bin1.fit, 1 layer(s), 4144x2822 pixels
23:58:17: Running command: stat
23:58:17: B&W layer: Mean: 1922.3, Median: 1919.0, Sigma: 125.5, AvgDev: 4.9, Min: 1862.0, Max: 65532.0

Others example come from 294MM. In the post we did not show dark with longer exposure of 5s because it was not the purpose of the post.

 

[vlaiv]

Ok, so I took my ASI185 - and SharpCap and took 5 shots.

bias, 1s, 2s, 4s, 8s

here are reported mean and stddev values.

[vlaiv]

I can repeat this with SGP - using ASCOM driver as well instead of native - but result will be the same

[lock042]

@vlaiv: just to make sure.

We are talking about modern CMOS sensors. As we explain in the post it is not true for old CCD sensor where the bias could contain linear gradient