Enough with darks flats, they are useless

[clouzot]

56 minutes ago, vlaiv said:

As far as calibration goes - I would just check if darks are now consistent. Something silly like removing constant offset can compromise darks if not done properly.

It's not crystal clear to me but I think  I got your point.

So, if I understand well, you're saying that ZWO is artificially tampering with the signal in order to keep a constant median ADU value? And that what we see with ZWOs (said constant median value) isn't generally applicable to other brands, sensors etc?

If that is the case, then I also have similar measurements made on Altairs and QHYs, with sensors ranging from IMX290 to IMX533. All show the same behavior, where the offset level is only a factor of the offset/brightness setting, and doesn't depend on the exposure. If someone is fiddling with the signal, then it's Sony , not ZWO.

1 hour ago, vlaiv said:

We want darks to be repeatable and well behaved

That is unfortunately directly dependent on the way the camera timing logic behaves. I've seen really odd things with the IMX294 (be it the color or mono version), with flat exposures of 0.999s being really 0.999s (as shown by the mean ADU value of the flat itself), and exposures set to 1.00s being all over the place between 0.8 and 1.2s. That's another quirk with that one sensor, something you don't see with more recent offerings, and something I didn't see with my IMX183 either.

 

1 hour ago, vlaiv said:

you don't have to worry about silly median values you are getting, but it also means that you should do proper calibration - darks, flats and flat darks for best results.

That's where I beg to disagree. I've seen too many different cameras having that same behavior (yielding a constant offset current whatever the exposure) to deem it silly. I find it more odd to see a more or less recent sensor whose median value changes substantially with exposure.

But I agree strictly with the second part of your sentence: yes, calibrating flats should theoretically be done with the exact same offset signal as what is in the flat proper. That's the math, agreed. 

However, a constant offset frame is, as was amply shown above, a very good model of said offset signal... (unless one has your ASI185MC, apparently), ampglow or not. It may not be mathematically exact, but the difference is so negligible (and hidden in other types of noise) that I don't see the point in painfully capturing hundreds of matched dark flats anymore.

Unless one has to capture long-exposure flats (that may still be the case with 3nm filters), that one "synthetic" calibration frame will work for all flat exposures. That's a substantial reduction in imaging hassles, so why not give a try?

 

 

[vlaiv]

6 minutes ago, clouzot said:

That's a substantial reduction in imaging hassles, so why not give a try?

To me it is not reduction in imaging hassles, so I won't be giving it a try, but anyone else is of course free to do so.

[tomato]

In the UK, with our cloud cover, taking calibration frames is never a hassle, it’s the lights that are difficult.

[clouzot]

56 minutes ago, tomato said:

In the UK, with our cloud cover, taking calibration frames is never a hassle, it’s the lights that are difficult.

It's a real miracle you guys can do any imaging from that fierce, ever-cloudy sea fortress (that's the one thing I don't miss about the UK, to be honest. The rest was -almost- pure fantasy from start to end).

[ollypenrice]

With CCD I always use a master bias as a flat dark to calibrate my flats. It produces a result which I cannot, by any means, distinguish from a result in which I've used dedicated flat darks. Taking dedicated flat darks would be needed every time I took flats and I'd need to shoot them for each filter - if I shot flats for each filter. Which I don't. I don't because I find that doing so is a waste of time, since my luminance flat gives the same result as my dedicated colour flats on the overwhelming majority of occasions. If, once every couple of years, they don't work on a particular filter, I shoot a dedicated flat.

I don't do AP to scratch some kind of perfectionist itch, I do it because I like it! When I find a short cut which works, I take it. I'd rather the put the time into something which will make a difference to my final image, and there I take as long as it takes.

Nobody has ever said to me, 'That's not a bad image but I see you didn't take a dedicated dark flat for the blue channel...'

lly

[lock042]

Thanks a lot @clouzot for your tests that perfectly match ours.

That shows that we are trying to say. Make your own tests with your setup, and I bet that in most cases you will find no differences. (Oh wait, maybe less noise  )

[vlaiv]

1 minute ago, lock042 said:

(Oh wait, maybe less noise  )

Can you explain how synthetic bias produces less noise?

[lock042]

27 minutes ago, vlaiv said:

Can you explain how synthetic bias produces less noise?

By not adding noise at all.

Subtracting image adds noise. Subtracting level don't.

Another test from Christian Buil (famous French astronomer which as created IRIS): http://www.astrosurf.com/buil/asi294mm.html

Comment. The offset signal is so uniform that I recommend for processing to equate it to an image of constant level: a synthetic bias, so the constant level is equal to the average (or better, median) intensity of a typical offset taken with the settings selected (gain, offset, temperature). This initiative is one way to reduce processing noise.

Edited December 20, 2021 by lock042

[alacant]

Just now, lock042 said:

maybe less noise

Hi

That's certainly the case with our dslrs. Not only noise, also other artefacts which make processing that much more difficult. Why put extra obstacles in the way?

In fact we've found that dark frames also mess up any further processing, so we've lost those too.

dslr summary: bias and flat calibration frames only and away you go.

@lock042 maybe you could describe your 'how to make a synthetic bias' implementation in simple terms (without spreadsheets!) for those of us here who take the pragmatic approach to imaging? Or maybe a list of bias settings for popular cameras so we can simply type in the required values into Siril?

Cheers and congratulations on reaching 1.0.0rc3.  

[vlaiv]

1 minute ago, lock042 said:

Subtracting image adds noise. Subtracting level don't.

If that is the case, can you run simple test for me with ASI294, just to confirm?

Take two bias subs.

On one of those, subtract offset and then analyze it for standard deviation (measure of noise). Then take again first sub and subtract second from it and again analyze it with standard deviation to compare which approach produces noisier result.

[lock042]

2 minutes ago, alacant said:

@lock042 maybe you could describe your 'how to make a synthetic bias' implementation in simple terms (without spreadsheets!) for those of us here who take the pragmatic approach to imaging? Or maybe a list of bias settings for popular cameras so we can simply type in the required values into Siril?

With DSLR it is very simple.

Take your master offset you did used. Measure the median value and it will be your level.

[alacant]

2 minutes ago, lock042 said:

Measure the median value

OK. So I loaded the master bias:

right click -> statistics

 Does this look OK?

 

[lock042]

2 minutes ago, alacant said:

OK. So I loaded the master bias:

right click -> statistics

 Does this look OK?

Sure. And as DSLR use power of 2 you can take 2048 as constant (you even can enter this value in the preferences : =2048).

This is the value I use with my DSLR.

[lock042]

14 minutes ago, vlaiv said:

If that is the case, can you run simple test for me with ASI294, just to confirm?

 

I'm sorry but this is signal processing basics.
With image operations, even with subtraction, the noise is added in the result, even with a master dark flat of 100 images. In the case of the synthetic bias you only use a constant. So noise stays the same.
So ok, difference is not huge with a master containing a lot of frame. But why would you add noise, even if it is not too much.

[vlaiv]

1 minute ago, lock042 said:

I'm sorry but this is signal processing basics.
With image operations, even with subtraction, the noise is added in the result, even with a master dark flat of 100 images. In the case of the synthetic bias you only use a constant. So noise stays the same.
So ok, difference is not huge with a master containing a lot of frame. But why would you add noise, even if it is not too much.

If you are so confident - why don't you do a test?

And indeed, it is signal processing basics - do your bias files have 0 stddev to begin with?

[lock042]

2 minutes ago, vlaiv said:

do your bias files have 0 stddev to begin with?

Of course not and this is why subtracting master bias or master dark flat adds noise, because they have noise. And this is why subtracting synthetic bias (i.e a constant) don't.

I think you don't get the point at all.

[vlaiv]

1 minute ago, lock042 said:

Of course not and this is why subtracting master bias or master dark flat adds noise, because they have noise. And this is why subtracting synthetic bias (i.e a constant) don't.

I think you don't get the point at all.

I think you don't and that is why simple test will show what is really going on instead of us going back and forth telling the other they are wrong.

[happy-kat]

2048 was the value from my Canon 1100d I tested bias files last night.

[inFINNity Deck]

2 hours ago, vlaiv said:

I think you don't and that is why simple test will show what is really going on instead of us going back and forth telling the other they are wrong.

I am still following this interesting thread and agree with @vlaiv that running this simple test provides more insight than telling each other they are right or wrong based on gut feeling only.

So if using a constant suffices, then the suggested test should indicate so.

Nicolàs

 

[andrew s]

Nothing new to add but when taking multiple short exposures (bias, flats, flat darks) I have found that the temperature control may not keep up.

So I add a delay between them to ensure stability of the electronics.

Regards Andrew 

[alacant]

Here is our test. These are the same portions of a fits of 5208x3476 from an eos700d. One with a master bias, the other with a constant median value.

 

 

Edited December 20, 2021 by alacant

[happy-kat]

Did you do one using a master dark flat too?

[alacant]

No. We try anything once or twice, but otherwise, we're strictly hands on. We do what works, not that which theory suggests should work. We gave up using any type of dark frame. It's just more to have to correct further down the line. This is just us with eos700ds.

We'd recommend trying @lock042's suggestion. It works fine here.

Cheers

Edited December 20, 2021 by alacant

[Martin Meredith]

Interesting thread. I'm convinced. Presumably you have to be careful about dead pixels (which a non-synthetic bias/flat dark would take care of, but which potentially leave you dividing by negative values once you subtract your constant)?

Martin

[drjolo]

On 19/12/2021 at 18:51, clouzot said:

It's not crystal clear to me but I think  I got your point.

So, if I understand well, you're saying that ZWO is artificially tampering with the signal in order to keep a constant median ADU value? And that what we see with ZWOs (said constant median value) isn't generally applicable to other brands, sensors etc?

This is not only ZWO, but also QHY case for sure.  Some value is subtracted  (probably not constant over frame, so it can minimize AG), that makes average ADU lower for longer exposures. As I mentioned before

Quote

For my QHY163M we have 733ADU for 100ms dark, and 710ADU for 180s dark. At some point I could go below zero. 

but of course the SD is higher for longer exposures. 

And we should also remember that calibration is not only specific for eye catching images. We use calibration for example for photometry, and for this application it is not enough just not to see a difference in the image. We need to be as accurate as possible, plus we need to be able to determine the error. 

Edited December 20, 2021 by drjolo