Master dark low ADU

[david_taurus83]

Finally finished collecting data on a project I've been on for the last month. I've started on the pre processing side of things and today I produced some master flat darks. What stuck out for me was the set of flat darks I collected at the weekend. The single 5.25s flat dark looks ok when stretched but the master looks like a bias master. The values look very low as well. See below. The mean value of 132 looks very low compared to what I'm used to. Master on top and single flat dark below. Its definitely 5.25s long as you can see from the FITS header.

 

 

Here is a stretched flat I took just before the flat darks. The target ADU was 25000 and its come out about right.

 

 

 

Now, here is a 2s master flat dark and a single 2s flat dark. The mean value of around 764 is about what I've gotten previously.

 

 

 

 

I don't always run statistics on darks and flats but the visual appearance of the 5.25s master surprised me. The only thing that's different from before is I wasnt running my guide camera and it's a new laptop. Any ideas? Worth shooting them again?

[vlaiv]

New laptop means new drivers installed?

Although you have MM-Pro version, it would be worth checking ASCOM driver settings - namely gain, offset and USB speed.

I know that some of the settings are missing in pro version, but just check and make sure if those are reintroduced in latest driver version. If so, you'll need to match them to settings used for shooting flats (and regular darks to lights if you changed your gear since you shot some lights need calibrating).

[david_taurus83]

It's the standard 139 unit gain and 50 offset. USB speed may be different. How does this affect the ADU values? The last set of data I took on Friday was all on the new laptop. Lights, flats and flat darks. Everything previous was on the old laptop.

[vlaiv]

I noticed that changing USB speed had impact on dark frames. That can also be concluded from ZWO recommendation - apparently with non PRO models (no memory buffer) - higher USB speed (faster readout) reduces amp glow.

What is worrying is that average 132 ADU value.

Did you by any chance divide your data with 16 and forgot about it? (ASI1600 is 12 bit camera and I usually divide all my subs with 16 to get exact values as camera multiples values with 16 to make use of MSB bits of 16 bit per pixel).

If not - new offset settings are wrong.

Old offset is about 50 and that can be seen from stats. Average ADU in old master flat dark is ~764 - which divided with 16 gives ~ 47.75 -that is to be expected from offset set to about 50.

If we apply same logic to new master flat dark - we get offset of only ~2. Pronounced bias signature can also be sign of offset too low - it will show up due to clipping, might not be visible in single sub because clipped values will be randomly distributed but in stack they will tend to average out and bring out the pattern.

 

 

 

[david_taurus83]

Thanks vlaiv. You might be onto something there. It puzzled me why my Lum flats were taking 5 seconds to get up to 25k. I usually have to turn the panel brightness down just so they get over 1 second. I will have to check the driver settings against both laptops. Slightly concerned now as I've 2 nights (3 hours in UK!) worth of light frames with new laptop!

[vlaiv]

1 minute ago, david_taurus83 said:

Thanks vlaiv. You might be onto something there. It puzzled me why my Lum flats were taking 5 seconds to get up to 25k. I usually have to turn the panel brightness down just so they get over 1 second. I will have to check the driver settings against both laptops. Slightly concerned now as I've 2 nights (3 hours in UK!) worth of light frames with new laptop!

I might be wrong, but it's best to check it anyway (not much work to be done).

[david_taurus83]

Ah....mystery solved...

 

 

I am not happy with myself. ?

[vlaiv]

9 hours ago, david_taurus83 said:

I am not happy with myself. ?

Pity. Gain 0 is not such an issue, but low offset is.

[david_taurus83]

I did check when I installed but didn't pay enough attention. When you hover over the bar it gives a prompt saying Unity Gain 139. I must have glanced at this and assumed it was at unity. Oh yeah, not just 2 nights wasted, but in fact 3!!!

 

I might just process what I've already got and take it as a lesson learned to check, check and check again!!!

[drjolo]

I think you can still calibrate your frames properly. You need darks made with the same settings as your lights (time, temperature, gain, offset, usb traffic). Unless this single dark has many pixels with value 0, you can properly dark calibrate your light frames.

As for flats, you can do flat frames with completely different settings. Just use the same settings for flats and dark flats, create master flat and use it to calibrate lights. 

Edited April 30, 2019 by drjolo

[vlaiv]

16 minutes ago, drjolo said:

I think you can still calibrate your frames properly. You need darks made with the same settings as your lights (time, temperature, gain, offset, usb traffic). Unless this single dark has many pixels with value 0, you can properly dark calibrate your light frames.

As for flats, you can do flat frames with completely different settings. Just use the same settings for flats and dark flats, create master flat and use it to calibrate lights. 

It can certainly be tested, but there are couple of issues that are likely when signal is clipped.

First, there will be a build up of pattern that won't be removed, and second - there is a good chance flat calibration will fail also. It depends on signal in sub being that of light only (after dark subtraction). Clipped values will give it an offset and that will create either under correction or over correction with flats (depends which way offset goes).

All of the above issues could be fairly small and one might not even notice them, but they can also be significant to cause trouble when processing.

[drjolo]

Yup, it all depends how dark frame made at these 0/10 settings looks like - ie how many pixels are clipped. If there are 100 or 1000 pixels clipped, then no big deal. 

[david_taurus83]

How do I tell if theres significant clipping?

[vlaiv]

21 minutes ago, drjolo said:

Yup, it all depends how dark frame made at these 0/10 settings looks like - ie how many pixels are clipped. If there are 100 or 1000 pixels clipped, then no big deal. 

 

4 minutes ago, david_taurus83 said:

How do I tell if theres significant clipping?

Well, above post by drjolo made me think - can we estimate number of clipped pixels per single dark sub?

This is really a "guesstimate", but I like to think that is "educated" one

Offset is put to 10e, and gain is at 0. Read noise at 0 gain is about 3.7e. This means that all pixels below 10/3.7 = 2.7 sigma will be clipped (gaussian distribution, noise is sigma and anything below 0 gets clipped to 0).

This gives probability of clipped pixel at ~ 0.0034389, or 0.35%

Since there are 4656×3520 pixel or 16389120 pixels total, this gives on average 57361.92 clipped pixels per dark sub.

Again, this is guesstimate - have no idea if camera actually behaves like this (if offset 10 really gives mean of 10 or slightly below - something I noticed, like in above case where offset 50 gave ~47e average on whole dark, and 3.7e is read from chart so might be slightly off, and each camera can behave slightly differently), and there is also bias signal that will impact this distribution (maybe average of dark is less than offset due to bias signal?)

Anyway, my guesstimate is about 60000ish pixels clipped per each dark sub - I think this is significant, but can be lessened with use of sigma clip algorithms for both master dark and light stacks.

[david_taurus83]

Think I need to apply the "take a picture, see how it looks" approach. I'll calibrate each set with it's own darks, 0 gain and unity, stack them all and stack the unity by itself. See how each set turns out.

[drjolo]

43 minutes ago, vlaiv said:

Well, above post by drjolo made me think - can we estimate number of clipped pixels per single dark sub?

Thats quite interesting problem I think. In CCD cameras there is also offset (bias), sometimes also available to set (like in QHY9), but as far as I know it is very low level parameter. It is the charge added to each pixel before exposure starts to avoid negative values read out from AD conversion. But I do not know how it works in CMOS cameras. It may be as in CCD, or it may be added in the driver level, or after conversion. 

I am also not sure if offset parameter is expressed in electrons. From my observations it is not the case for my QHY163 camera. 

[vlaiv]

3 minutes ago, drjolo said:

Thats quite interesting problem I think. In CCD cameras there is also offset (bias), sometimes also available to set (like in QHY9), but as far as I know it is very low level parameter. It is the charge added to each pixel before exposure starts to avoid negative values read out from AD conversion. But I do not know how it works in CMOS cameras. It may be as in CCD, or it may be added in the driver level, or after conversion. 

I am also not sure if offset parameter is expressed in electrons. From my observations it is not the case for my QHY163 camera. 

I think it is pretty much the same as CMOS offset with a difference that CMOS has amp on each pixel. Maybe each amp unit is reading offset from same register or similar. In general offset is not exact, at least that is what I've noticed - it is usually a bit less than set value with ASI1600.

If you look at first images from this post - old master dark and new master dark. In first case offset was 50 and gain was unity, so you would expect mean master dark to be about 50e (a bit more due to dark current). Mean ADU value is 764 (16 MSB) or 47.75 when divided with 16 to shift it down to LSB and get real values.

New master dark has average ADU of 132. That divided with 16 gives value of 8.25. That is "consistent" with offset of 10 (again about 2e less than that). Note that in this case gain is 0, or e/ADU is ~5. This means that offset is added after gain is applied and before ADC phase, so actual offset in electrons in this case is about 41e when converted back.

This goes a bit against my assumption that offset is the first thing that is added to signal, prior to gain stage - it looks like it is after gain stage. Not much difference except true offset will depend on gain used.

[drjolo]

In ZWO drivers it looks much more consistent then. In QHY guys are playing with it from one driver version to another. I used to have unity gain at 130 and I had offset set to 50 then to have average bias frame value at 500ADU, and minimum value at 50ADU (more less). With current version I have unity gain again at 13, and even with offset set to 0 I have mean bias frame ADU over 4000   But changing gain in this driver version does not affect much mean bias frame value, so it would confirm the assumption, that offset here is added after gain stage. 

Edited April 30, 2019 by drjolo

[david_taurus83]

I deleted the data. 4 hours worth. You live and learn.