Enough with darks flats, they are useless

[vlaiv]

Here is above "experiment" that I conducted on some old data that I found. Have a session from 2015 where I have bunch of nice bias files with camera that will show this effect nicely.

First row is single bias and its standard deviation.

Second row is that same bias with median value of 204 removed - of course nothing changes in terms of noise when you remove constant value.

Third row is original bias sub "calibrated" with master bias. Noise is now lower.

This is counterintuitive if we know that mathematical operations like addition and subtraction add noise rather than remove it. How can it then be lower?

Bias data consists out of two components - bias signal and read noise. If we assume bias signal is flat / constant - like removing constant value from our bias sub - we will make a mistake of not removing "detail" in bias signal - all the variations of bias signal itself. From data and image point of view - this is "noise" - although it is not random in nature - it is unwanted signal that we don't want polluting our data / image.

Here is what master bias looks like for this camera:

all those vertical lines are in fact unwanted signal / variation that shows up as StdDev in above measurement - and you don't want it in your image.

Proper calibration will remove it, while using synthetic bias - won't.

In any case - I think I have proved my point with numerous examples. Regular / proper calibration with dark flats - is guaranteed to work (if we have well behaved camera that can be calibrated), while using synthetic bias / flats and other - is not guaranteed to work. It might work in your particular case with your equipment - but it is wrong to conclude that it is proper / better way to do things.

[drjolo]

11 minutes ago, vlaiv said:

Here is above "experiment" that I conducted on some old data that I found. Have a session from 2015 where I have bunch of nice bias files with camera that will show this effect nicely.

Was it a CCD sensor? It looks similar to Kodak equipped CCD cameras I had. This fixed pattern noise was pretty significant, however it was 100% repetitive, so it was easy to remove. Repeatability and predictability it is something that I miss the most in CMOS cameras. 

[vlaiv]

17 minutes ago, drjolo said:

Was it a CCD sensor? It looks similar to Kodak equipped CCD cameras I had. This fixed pattern noise was pretty significant, however it was 100% repetitive, so it was easy to remove. Repeatability and predictability it is something that I miss the most in CMOS cameras. 

Actually it is CMOS sensor - one of "early models" - Aptina MT9M034 (found also in ASI120mc).

I do believe there is always this fixed pattern component present - it maybe less noticeable in some newer models, but it is always there.

[vlaiv]

Here is same thing again with DSLR (Canon 750d):

15 bias subs in total - used 14 to create master bias and remaining one to measure StdDev and then to calibrate out fixed pattern noise.

Again - stddev is smaller (and it would be smaller still had I used more subs).

[clouzot]

Here's my take on this technique. I can publicly share all subs (save the individual darks and offsets, which I don't have anymore) if you want to replicate the experiment

I had those 11 sub-exposures of NGC891 done on my C6 (those who have this little scope will recognize the well-known looping star flare!) and my ASI294MM Pro bin2. Gain was 120 (unity), exposure set to 120s, offset set to 30 (default). So there's 22 minutes worth of lights, which is admittedly not much by AP standards. Sadly, good weather is also a myth down here.

I also had 19 inidividual flats, with the following settings: exposure 5s, gain 120, offset 30, bin2, mean ADU around 32k (16 bits, so 65k max)

Those individual flats were then assembled with the following processing chain in Siril:

- calibration with either a true 5s master darkflat, or a synthetic offset (ADU = 1920)

- stacking and multiplicative normalization of said calibrated flats

...yielding two different master-flats.

 

Said master flats were then used to calibrate the subs, along with a 120s master dark. The two results were registered, stacked in Siril. No background extraction process was applied, hence the slight gradient

The renderings below show how the the final stack look like in false color and "histogram equalization" display mode that stretches thing a lot in order to see any substantial defect. I also added their respective image stats.

TRUE DARK FLATS

 

SYNTHETIC OFFSETS

To further explore things, I did a pixel-math subtraction of the two stacks, and while they're definitely not bit-equal, the difference is mainly in the... bright stars, oddly enough.

Anyway, no trace of the ampglow can be seen in that direct subtraction between the "synth offset"-calibrated stack and the "true dark flat"-calibrated one.

For further reference, here's what the true 5s master dark flat looks like for my camera. The ampglow of the 294 can already be clearly seen (star burst pattern of the right, and top and bottom rows as well)

So at this level of ampglow, synthetic offsets don't make a visible (at least, to me!) difference in the final product. Of course, things would perhaps be different if the ampglow were much more present, ie with 30s+ flat exposures. I'm yet to test that, perhaps with my upcoming 3nm Antlia filter which, I suspect, would need more exposure.

 

[vlaiv]

7 minutes ago, clouzot said:

I can publicly share all subs (save the individual darks and offsets, which I don't have anymore) if you want to replicate the experiment

I'd be interested only in two final linear fits - one with synthetic bias and one with flat darks (I assume this is luminance only) for inspection. Thanks

 

[clouzot]

2 hours ago, drjolo said:

This is not only ZWO, but also QHY case for sure.  Some value is subtracted  (probably not constant over frame, so it can minimize AG), that makes average ADU lower for longer exposures. As I mentioned before

That's really interesting. Is that what those two companies (falsely) called "ampglow reduction" then? Altair (and probably all the white label cameras that share the same hardware: RisingTech, etc) is in the same boat then, as my old 294c had the same behavior.

7 minutes ago, vlaiv said:

I'd be interested only in two final linear fits - one with synthetic bias and one with flat darks (I assume this is luminance only) for inspection. Thanks

Here you go: https://1drv.ms/u/s!AmemdJT-7GC6hB000TreEx9SZ1jT?e=dbQUil

This was luminance data only as you noted (captured during a live EAA session, hence the absence of RGB data and the short 120s exposures).

Please let me know what you find out, because I'm genuinely interested in what kind of devil could be hidden in that offset-VS-darkflat controversy.

[vlaiv]

29 minutes ago, clouzot said:

Please let me know what you find out, because I'm genuinely interested in what kind of devil could be hidden in that offset-VS-darkflat controversy.

Difference is exceptionally small between these two, but there is difference - and it is far stranger than I expected to be honest:

This is difference of the two images stretched to insanity

You can see that there is some small difference in signal in the central galaxy - that can be down to flat correction not being 100% correct, but I'm really puzzled by these lines in the image. If both images are processed completely the same except for bias removed from flats - then these have to do with flats and I'm really puzzled by look of that pattern. As if it is warped / stretched somehow. Not nearly what I would expect from regular grid pattern of pixels.

I think I also see amp glow in top right part (ImageJ loaded images vertically inverted in comparison to your presentation above).

Here is above image smaller and vertically inverted to match it

Well, this does serve to show that there is a difference, and in this particular case difference is very small (would not be shown in final image by any stretch) - but do bare in mind that with much more integration time - people tend to push their data stronger and even small imperfections start to show as these won't go away with number of stacked subs - they are fixed signal in nature.

By the way - I love this strange pattern in the image and can't even begin to fathom what it could be.

[lock042]

14 hours ago, vlaiv said:

If that is the case, can you run simple test for me with ASI294, just to confirm?

Take two bias subs.

On one of those, subtract offset and then analyze it for standard deviation (measure of noise). Then take again first sub and subtract second from it and again analyze it with standard deviation to compare which approach produces noisier result.

OK, what you were asking for before is not the test you have been doing, as nowhere in your procedure you were asking to subtract a master.

What I gather from your experiment is that:

• First line is the stats of a single bias

• Second line is the stats of same bias with a constant value subtracted

• Last line is the stats of same bias – a masterbias.

Fair enough, it makes “more sense”. At least if the discussion from the start had been about calibrating biases… which is not at all the point.

So here is the test that we have been talking about. Made 200 darkflats of 0.9s, matching the flats done for a session from last summer. ZWO ASI294MC, gain 120, offset 30, T -10C for reference.

Stacked 1, 10, 25, 50, 100, 150 and 200. Yes, of course, the std decrease with increasing subs…

Then checked the stats of a flat (1^st CFA channel only, it would not make sense on a color sensor to measure all channels altogether) from which I have substracted a synthetic offset of 1920 and masterdarkflats made with 50, 100, 200 stacked frames respectively.

And here-below the results:

I’m not even going to claim that there is significant reduction in noise in the calibrated flat, because the stds are probably within the uncertainties. But to be honest, this kind of measure is the basis for questioning the usefulness of using darkflats (or bias) to calibrate the flats.

I hope this will encourage others to test with their setup.

So now I think we've done enough testing (here and with the post). We're not forcing anyone to adopt synthetic bias, but we've shown in different ways that it's worth a try.

[clouzot]

Thanks @vlaiv for the thorough examination of that small dataset.

The difference between the two comes as no real surprise: this is obviously not a bit-exact process, by any means.  The presence of a residual ampglow may also have to do with the masterdark itself being slightly off (I'm always suspicious about my calibration frames, and sometimes rightly so to be honest).

28 minutes ago, vlaiv said:

By the way - I love this strange pattern in the image and can't even begin to fathom what it could be.

That skewed grid-like pattern is puzzling indeed! Surely it has to do with the little signal there is in each sub, which sometimes causes Siril (and some other stacking software I've used, including APP) to create those moire-like patterns after registration. I could hardly use more exposure that night, as stars were already saturating big time, though.

The differences you found may also be related to the (additive) normalization step that is applied by default at the end of stacking process. As the two stacks can't have have the exact same pixel values, additive normalization will probably behave a bit (pun intended) differently. And most probably every other calibration and staking step may be offset by the slight difference in pixel values found in the calibrated subs: registration, rejection stacking, you name it.

However, I'm not trying to shrug off your case, and intuitively speaking, I get your point about those small non-random differences building up as the integration time increases. That could a true concern. As I'm not in the 28-hours-per-channel league - yet - I'd truly appreciate if someone could test that approach with a more decent dataset.

Anyway, let's assume the synth-bias technique appears to be working correctly for casual stacks (I've only done up to 3 hours for the moment), and its further applicability to longer stacks remains to be proven or falsified/disproved... depending on how one sees it.

[inFINNity Deck]

10 hours ago, clouzot said:

So at this level of ampglow, synthetic offsets don't make a visible (at least, to me!) difference in the final product. Of course, things would perhaps be different if the ampglow were much more present, ie with 30s+ flat exposures. I'm yet to test that, perhaps with my upcoming 3nm Antlia filter which, I suspect, would need more exposure.

For me that would indeed be of interest. As I explained before my flats range from 1s to 67s, depending on the filter used. So I hope this experiment is repeated for longer exposures like 30s as @clouzot suggests, but also for 45s, 60s and 90s.

As my mount is in Italy since the first week of November for maintenance and not expected back until after Christmas, my observatory is completely stripped down, so I have no easy way to produce some results myself (and last night we had crisp clear skies... :-().

Looking forward to further tests!

Nicolàs

[drjolo]

So, correct me if I am wrong, now we agree that the problem exists, but the question is if it is visible or not, right? 

So the conclusion could be at that point, that when we switch from dark flats to synthetic bias method, we exchange dark flat random noise problem to synthetic bias uncorrected fixed pattern noise problem. Both can be visible or not, but in dark flat case we can minimize random noise by increasing the calibration frames number. FPN cannot be reduced by synthetic bias, because FPN information is lost in that method. 

[inFINNity Deck]

4 minutes ago, drjolo said:

So, correct me if I am wrong, now we agree that the problem exists, but the question is if it is visible or not, right? 

So the conclusion could be at that point, that when we switch from dark flats to synthetic bias method, we exchange dark flat random noise problem to synthetic bias uncorrected fixed pattern noise problem. Both can be visible or not, but in dark flat case we can minimize random noise by increasing the calibration frames number. FPN cannot be reduced by synthetic bias, because FPN information is lost in that method. 

Add to that: "for short darkflat exposures (i.e. <= 5s)", as indicated in the OP's article.

Nicolàs

[sharkmelley]

11 hours ago, vlaiv said:

By the way - I love this strange pattern in the image and can't even begin to fathom what it could be.

A "warped grid" like that is often an indication of geometric transformations during star alignment.

[vlaiv]

1 hour ago, sharkmelley said:

A "warped grid" like that is often an indication of geometric transformations during star alignment.

First time I've seen it like that. I saw regular square grid at an angle due to rotation and use of bilinear interpolation, but never this warped

[sharkmelley]

1 hour ago, vlaiv said:

First time I've seen it like that. I saw regular square grid at an angle due to rotation and use of bilinear interpolation, but never this warped

The same for me - I've never seen an example this warped either.

[iantaylor2uk]

I tend to use an ASI Air Pro and use auto exposure time for flats, using a light panel, and typically get exposure times around 40-50 ms with my ZWO 071 MC Pro camera (and UV/IR filter). I tend to use bias frames rather than flat darks as I'm not sure you can choose those exposure times with the ASI Air Pro.

[clouzot]

5 hours ago, iantaylor2uk said:

I tend to use an ASI Air Pro and use auto exposure time for flats, using a light panel, and typically get exposure times around 40-50 ms with my ZWO 071 MC Pro camera (and UV/IR filter). I tend to use bias frames rather than flat darks as I'm not sure you can choose those exposure times with the ASI Air Pro.

This blog entry explains how you can align your darkflat exposures with those that the AAP determines automatically for flats

I don’t know if the trick works for those short exposures though 

https://eastwindastro.blogspot.com/2021/03/asiair-autoflat-frame-exposures.html?m=0

[jjosefsen]

This has certainly been an interesting and educational read, so thank you for that.

I don't think you can say that synthetic bias works for all flats < 5 sec. I have an ASI1600 clone and I have amp glow in 1,8s darkflats.. 🤔

So it is also equipment specific.

 

[lock042]

9 hours ago, jjosefsen said:

I don't think you can say that synthetic bias works for all flats < 5 sec. I have an ASI1600 clone and I have amp glow in 1,8s darkflats.. 🤔

 

Be careful. Your ampglow can be negligeable at 1.8s, like with 294MM. If you look your image very stretched, of course you will see it, but statistics don't lie.

[vlaiv]

1 hour ago, lock042 said:

Be careful. Your ampglow can be negligeable at 1.8s, like with 294MM. If you look your image very stretched, of course you will see it, but statistics don't lie.

It actually depends on other factors if it will be significant or not.

I've measured amp glow to be 1-2e higher than surrounding signal. ASI1600 is 12 bit camera and people often use unity gain. Flats will therefore contain about 3000e of signal.

That is variation of about 1/1500 in intensity if not removed.

If one is shooting in very light polluted area - background sky signal can be quite high. You are correcting this background sky signal with variations in flats. It will create some variation in corrected background signal - more signal there is, greater variation will be in absolute value.

We end up removing that background - but that variation will remain.

[lock042]

40 minutes ago, vlaiv said:

If one is shooting in very light polluted area - background sky signal can be quite high. You are correcting this background sky signal with variations in flats. It will create some variation in corrected background signal - more signal there is, greater variation will be in absolute value.

We are talking about removing dark flat to flats. We don't care about light pollution.

This is my whole point. We are just talking about flats here.

Of course darks are very important to remove all unwanted signal to light frames.

Edited December 22, 2021 by lock042

[vlaiv]

1 minute ago, lock042 said:

We are talking about removing dark flat to flats. We don't care about light pollution.

This is my whole point. We are just talking about flats here.

Flats correct light pollution as well

Say you have 0.1% difference in flat due to amp glow - and it's possible given that amp glow can be up to 2e with flat signal being 2000e (12 bit camera, unit gain - ~3/4 histogram + some vignetting).

You have 1000e of light pollution which you correct with 80% and 80.1% of amp glow so that is 1000e / 0.8 =  1250 and 1000e / 0.801 = ~1248.44

That is ~1.5e of difference in background signal when you remove background (you can't remove that as it is neither linear nor 2-3 order polynomial).

Easily seen when data is stretched.

[lock042]

5 minutes ago, vlaiv said:

Flats correct light pollution as well

 

Ok. Sorry but I stop here. This assertion is so wrong that I couldn't go further.

Have a nice day.

[vlaiv]

1 minute ago, lock042 said:

Ok. Sorry but I stop here. This assertion is so wrong that I couldn't go further.

Have a nice day.

I'm sorry that you feel that way, but assertion is not wrong as all the light entering aperture of telescope is subject to vignetting / attenuation by optical train.

Flats correct for that and if sky was not subject to this - sky flats would not work.

Have a nice day too.