Enough with darks flats, they are useless

[lock042]

3 minutes ago, vlaiv said:

I can repeat this with SGP - using ASCOM driver as well instead of native - but result will be the same

Do you have your offset set to 0??? 

[vlaiv]

3 minutes ago, lock042 said:

@vlaiv: just to make sure.

We are talking about modern CMOS sensors. As we explain in the post it is not true for old CCD sensor where the bias could contain linear gradient

Yes, ASI185 is modern CMOS sensor 1944x1224 pixels from ZWO (Sony IMX185)

I also have ASI1600 with Panasonic CMOS sensor and ASI178 with again Sony sensor - all behaving the same.

Here, I did same test again in SGP:

you have exposure length in title and rest is self explanatory ...

[vlaiv]

Just now, lock042 said:

Do you have your offset set to 0??? 

No, in SGP I have offset set to 64 in this case (camera is 12bit so ADU values are multiplied by 16 - but that does not change anything 16*64 = 1024, so  ADU "offset" is around 1024 in this case - as you can see from above table Mean values start at around 1000 and raise - zwo offset is not very precise it often "misses" by dozen or so ADU).

[lock042]

It's look like you have some leak of light...

Here an example of 294MC non cooled, It is less stable but not like you:

Shot with sharpcap

Edited December 16, 2021 by lock042

[oymd]

🍿..🍿

🙂

[inFINNity Deck]

10 hours ago, lock042 said:

In fact that depends more of your ampglow (do you see ampglow flash in your masterflat?).

Yes, at the upper- and lower-right I do see ampglow in my masterdarkflats:

Above image is stretched and resized, the original fits is here. Recording with ZWO ASI1600MM Pro Cool @ -20°C, 67.1 seconds exposure.

Nicolàs

[vlaiv]

10 hours ago, lock042 said:

It's look like you have some leak of light...

Here an example of 294MC non cooled, It is less stable but not like you:

Shot with sharpcap

No, I don't have a light leak. Every camera is different and we are shooting in different settings.

ASI294 has quite low dark current, lower than ASI1600. ASI185 might not have such low dark current, and it depends on ambient temperature what that dark current will be (for non cooled cameras).

In any case - both measurements prove my point - there is dark signal that needs to be removed, otherwise you might end up with flawed flat calibration.

You are right in one point - people might get away with using bias instead of flat darks, and that is fine in some cases - like short subs with set point cooled camera where there is not enough dark signal to make visible difference, but please don't make that into general conclusion - you'll be only confusing people that don't have enough knowledge to distinguish different cases.

Flat darks work in all cases, they are correct way to do things - no shortcuts taken, and as such they should be advised as proper way to do things. If one knows what they are doing, then by all means - shoot in any way you want, but don't go advising others if you don't fully control their process as well.

[tomato]

I don’t have Siril, but I do have CCD and CMOS cameras, both with and without amp glow. I do have an automated system for taking flats and dark flats and plenty of disk storage, and I cannot see any flaws in @vlaiv’s logic, so I’m staying with dark flats.

[lock042]

In your case, you do use sensor not cooled and with a lot of glowing.

As we said in the post. We encourage users doing tests before as we are speaking about modern sensors.

For example, what I'm saying is TRUE for modern DSLR. Try to make a 60s dark and a bias with a DSLR. Level is the same.

Astrophotography evolves, sensors too.

Edited December 16, 2021 by lock042

[geeklee]

18 minutes ago, vlaiv said:

Flat darks work in all cases, they are correct way to do things - no shortcuts taken, and as such they should be advised as proper way to do things. If one knows what they are doing, then by all means - shoot in any way you want, but don't go advising others if you don't fully control their process as well.

As @tomato mentions and @vlaiv states above, why risk it?  I'll stick with dark flats for now.  I want to ensure I'm doing everything I can to make calibration successful -  even at the cost of extra time capturing these.  

[scotty38]

14 minutes ago, lock042 said:

In your case, you do use sensor not cooled and with a lot of glowing.

As we said in the post. We encourage users doing tests before as we are speaking about modern sensors.

For example, what I'm saying is TRUE for modern DSLR. Try to make a 60s dark and a bias with a DSLR. Level is the same.

Astrophotography evolves, sensors too.

So why such a sweeping opening statement and title?

Are you saying this is also TRUE for my 294MC Pro cooled camera?

[drjolo]

I have been doing biases when I imaged with CCD cameras. The only reason I did bias frames was, that CCD dark signal scaled well with time, and it was possible to "synthesize" dark signal for different dark time having bias frame and dark frame of the different time. At least Maxim did that in the reasonable way. But that was a workaround of course.

Since the time I switched to CMOS cameras I never did bias frames, I use only dark flats. For me it is the only way to have proper flat calibration of the frames. The higher background signal (LP) is and the longer exposure for flat frame is required, the more useless bias is for flat calibration. 

As vlaiv mentioned - this is a simple math, and to have proper flat signal to calibrate, it must be subtracted with dark flat. I prefer to fight with noise with more data captured, because removing of dust donuts caused by bad flat calibration is for me a much more pain. 

Edited December 16, 2021 by drjolo

[vlaiv]

I did not even mention unstable / unusable bias.

My ASI1600 has larger mean bias value than in short exposure. If I used bias - I would end up getting "negative dark current" in my flats.

[teoria_del_big_bang]

3 minutes ago, vlaiv said:

I did not even mention unstable / unusable bias.

My ASI1600 has larger mean bias value than in short exposure. If I used bias - I would end up getting "negative dark current" in my flats.

Even as a novice and not understanding all of these arguments this was a big worry of mine going back to using Bias for CMOS.
Using Bias for my ASI1600M just never seemed to work for me.

Steve

[lock042]

22 minutes ago, scotty38 said:

Are you saying this is also TRUE for my 294MC Pro cooled camera?

Yes and we have made tests with this camera. 

[lock042]

10 minutes ago, vlaiv said:

I did not even mention unstable / unusable bias.

My ASI1600 has larger mean bias value than in short exposure. If I used bias - I would end up getting "negative dark current" in my flats.

Not if you use constant.... 

[vlaiv]

9 minutes ago, lock042 said:

Not if you use constant.... 

If I use constant as a bias???

Why leave all that fixed pattern noise in the image?

[lock042]

21 minutes ago, vlaiv said:

Why leave all that fixed pattern noise in the image?

In the flat????

The bias only applies on flats.

And if you take a look at our plots. That is really not a big deal.

 

Of course master dark is very importants for lights. Because lights have a very low level in comparison of flats.

Edited December 16, 2021 by lock042

[drjolo]

What is really misleading in this thread and the linked article, is calling calibrating with dark flats "useless" - in my opinion it is 100% correct way of calibrating images. 

Synthetic bias or synthetic offset can be used, but in all cases it will be a workaround. How much inaccurate workaround? It only depends on the data from the camera. When you have ampglow - you should have dark flats. Many CMOS cameras have AG. Some of them uses kind of "AG reduction" that basically subtracts AG signal from the frame. It leads to another CMOS problem - dark signal is not linear with exposure time, and may decrease with increase of the exposure time. For my QHY163M we have 733ADU for 100ms dark, and 710ADU for 180s dark. At some point I could go below zero. 

Another thing is that data from CMOS depends on gain, offset, USB traffic, temperature and driver version. If anything changes, there is a good chance you need to update your calibration library, but also need to take care of determining new bias offsets. 

 

And how much noise is introduced into the final image by dark flat?

My 40x100ms dark flat has 3.8ADU noise. 

My 120s light frame has 168ADU noise.
With dark flat added it will be 168.04 ADU (noise does not add linearly, it is square root of sum of squares). I do not see a real benefit here. 

My 180s Ha frame has 29.7ADU noise.
With dark flat added it will be 29.9ADU. Still not too impressive difference, plus 180s is pretty short for narrowband and f/7 optics.

 

So, I would definitely avoid calling dark flat calibration useless. It is 100% correct, but introduces some noise. I would rather call master bias or bias offset calibration useful in some cases, and never 100% correct. 

Edited December 16, 2021 by drjolo

[lock042]

11 minutes ago, drjolo said:

When you have ampglow - you should have dark flats

In this image, prove me you cannot replace the darkflat by a constant (right figure) (of course darkflat plot has been y shifted to be visible on the flat plot): 

It is not because you see ampglow it will be a problem. Especially when you scale the darkflat to the flat....

11 minutes ago, drjolo said:

And how much noise is introduced into the final image by dark flat?

We agree it is not really a problem of noise.

Edited December 16, 2021 by lock042

[drjolo]

6 minutes ago, lock042 said:

In this image, prove me you cannot replace the darkflat by a constant (right figure): 

That is not correct by definition. AG is a area with different ADU values than other frame parts. And how do you want to replace value that changes over frame with some constant? It may be more or less accurate, may be not visible at some stretch level, or become visible, but will never be 100% covered.

[lock042]

You say than ampglow has different ADU values. But take a look at the figure we did: the plot is inside the AG area on purpose!!. These differences are negligible in comparison to the flat. They mean nothing.

Edited December 16, 2021 by lock042

[drjolo]

I totally agree for this camera and masterflat case the difference may be not visible in the final image. The same as the amount of noise introduced with dark flat may not be visible. My only concern is that fight against the dark flats as useless method of calibration. 

I know many amateurs who do not feel comfortable with ADU readings and plots. And they really prefer to shot dark flats that takes a few minutes, than taking bias frame and analyze that. That will not be useless at all, and the final outcome of both methods will also not be useless, but the same in most cases. 

Edited December 16, 2021 by drjolo

[clouzot]

Here are some measurements I made with my 294MM, bin2 mode (11 Mpix), gain 120, T=-10°C, offset set to 30 (default value).

I took a bunch of darks (or "bias", but real bias frames are a no-go with that camera) at various exposures, and measured some statistics (with no other software than Siril, of course, but ASTAP reports the same values)

First, the median ADU value. All exposures are in ms, so we have measurement points from 10ms up to 180s (180000ms). Despite the well-known ampglow with that sensor, the median ADU value stays pretty much the same.

 

What changes is the amount of noise, as measured by the dispersion. Here is a plot of the standard deviation (sigma) for the same frames.

The phenomenon at stake is mainly the gradual build-up of thermal noise (low on this sensor) and ampglow (substantial, as you all know).

 

So what does that mean from a calibration perspective?

- under 5 or 10s, the thermal/ampglow signal is pretty much negligible compared to the flat signal level (which should be bright enough to swamp any kind of noise there may be).

- still under 5 or 10s, the median ADU value of an offset (or "dark flat") frame stays stable around 1920

- If the 294 were behaving nicely with all exposures, that would mean that any flat exposure under 10s could be calibrated with a fixed, constant offset value, and that's pretty much what synthetic offsets are about.

- However, the 294 has timing issues that prevent the capturing of very short flat frames. That is exactly why people started capturing those long flat frames, and started taking dark flats to calibrate them, but in lots of cases it's imho worthless. From my experience, I can say the technique works pretty well with flats whose exposure stays between 0.5s and 10s.

- while the amount of noise that flatdarks bring back to the images may be negligible, one big advantage of that technique is that you have a single "master offset for flats" that you can reuse for any flat exposure between 0.5 and 10s. In most cases, even with restrictive filters, that should be enough. In my case, I can always keep my flat exposures under 10s, be it with a 7nm SII filter or the IR-UV cut I use for luminance. That's really the main point for me.

 

All of the above applies to my specific model (ASI294MM Pro), but that should also be applicable to more recent sensors (ASI533/2600 etc).

Edited December 19, 2021 by clouzot

[vlaiv]

2 hours ago, clouzot said:

So what does that mean from a calibration perspective?

I think that ZWO is again messing with drivers and trying to hide amp glow.

They are probably removing constant offset or something based on their measurement of dark current.

As far as calibration goes - I would just check if darks are now consistent. Something silly like removing constant offset can compromise darks if not done properly.

We want darks to be repeatable and well behaved - this means that if you take 10 darks at some temperature and settings - they should all have same mean value and they should all calibrate each other out perfectly (that means - if you take any two darks and subtract them you should get 0 mean image with pure noise - do FFT on it and check for any anomalies - FFT should also be just noise image).

If this is true - then you don't have to worry about silly median values you are getting, but it also means that you should do proper calibration - darks, flats and flat darks for best results.