How critical is dark temperature. Test with darks

[han59]

Today I tested the criticality of the dark temperature compared with the light temperature. I noted that calibration with master dark of a lower temperature gives a poorer result  then calibration with a master dark of equal temperature. See noise and hot pixel values below.

Han

 

Edited August 8, 2023 by han59

[ollypenrice]

When you say it gives a poorer result, are you looking at the picture or at some numbers on a spreadsheet?

Olly

[han59]

The measured values in the spreadsheet.

[Elp]

I think the point was what does the final image look like? I use uncooled cameras more often than cooled, the difference in quality with the final image result isn't that different, granted temperatures here don't really get too warm. A lot of people use DSLRs which get quite warm perfectly fine.

Edited August 9, 2023 by Elp

[wimvb]

So, if I interprete the results correctly, @han59:

• Cooled is better than not cooled, regarding both noise and hot pixels
• It's beneficial to use (matching) darks even if you don't cool your camera
• Darks should match lights within a few degrees.

Two questions come to mind:

1. How many degrees difference can there be between lights and darks? (At 26 C, 3 degrees seems ok; does that hold at -10 C as well?)
2. How critical is temperature for amp glow removal (the other reason we use darks)?

[tomato]

I use ASI 178 cameras which have major ampglow, the key to calibrating it out was to match the temperature for lights and darks. This was achieved by retro fitting Peltier coolers to the cameras.

[han59]

On 09/08/2023 at 11:51, Elp said:

I think the point was what does the final image look like? I use uncooled cameras more often than cooled, the difference in quality with the final image result isn't that different, granted temperatures here don't really get too warm. A lot of people use DSLRs which get quite warm perfectly fine.

I have exported four images as 16 bit PNG unstretched but first shifted the mean background to level 1000. Then combined them into one image using Gimp. Exported again in 16 bit PNG and stretched in ASTAP. See below the results.

I have attached the unstretched 16 bit PNG image in a ZIP file. You can measure yourself.

On 09/08/2023 at 14:41, wimvb said:

• How many degrees difference can there be between lights and darks? (At 26 C, 3 degrees seems ok; does that hold at -10 C as well?)
• How critical is temperature for amp glow removal (the other reason we use darks)?

How, I don't know at the moment. But I assume it becomes more critical at higher temperatures. The slope  delta noise/delta temperature becomes steeper and steeper.

I haven't tested the amp glow. The images where all taken from sensor center 250x250 pixels.

 

combined level 1000 v3.zip

Edited August 11, 2023 by han59

[han59]

I did an other test with scaling the darks. This only works for dark above 0 degrees Celsius when the dark current is significant.  The factor 4.34 comes from 17.8 e-/4.1 e- (noise 26°C/noise 11°C)

 

 

 

 

 

 

Edited August 11, 2023 by han59

[sharkmelley]

On 11/08/2023 at 20:57, han59 said:

I did an other test with scaling the darks. This only works for dark above 0 degrees Celsius when the dark current is significant.  The factor 4.34 comes from 17.8 e-/4.1 e- (noise 26°C/noise 11°C)

The factor of 4.34 is the wrong factor to use because it is calculated from the ratio of noise.  Generally speaking accumulated dark current in an exposure will double for each 5C-7C increase in temperature, so the thermal noise increases by the square root of this.   The factor used for dark scaling is usually calculated using max entropy or calculated from the relative brightnesses of the "warmer" pixels.  Alternatively use "trial and error" to determine the factor that minimises the noise in the final image.

[han59]

The results of the trial and error method for darks at -1C and -10C are displayed in the bottom part of the screenshot but still no improvement.

A part of the noise is the read noise which will be likely stable.  Which factor would you propose?

 

Edited August 24, 2023 by han59

[vlaiv]

We don't use darks to reduce noise. That is common misconception.

Darks are there to remove dark current. They even increase associated noise somewhat when applied.

If we don't use matching darks we will not properly remove dark current. This as a result has wrong calibration (any value that we read of from image will be skewed) and also - flat calibration will fail because of residual dark signal that was not removed.

Best way to ensure that we remove dark current is to match the temperature as dark current is temperature dependent.

Yes, we can try dark scaling - but it is not guaranteed to work with every camera (or indeed with every set of data). We need "well behaved' sensor for it to work (not much messing in firmware and trying to optimize things or remove amp glow and such).

Here is procedure:

1. we first need to remove any bias / offset that is common for both ligths and darks. For this to work we must have stable bias and exposure must be truly linear with exposure length - measured mean ADU values for dark frame must be straight line starting with bias and then moving on to longer subs

2. Once we remove bias we can take dark and use arbitrary starting scale factor (usually 1 or we apply the rule of ~6C doubling of dark current mentioned by @sharkmelley above). We apply that scaling factor to darks and do calibration of our data - meaning scaled dark removal and then flat calibration.

3. We take resulting sub and calculate its entropy (information, see here: https://en.wikipedia.org/wiki/Entropy_(information_theory) ). Since we will be using 32 bit floating point precision, it is useful to round pixel values to fixed point number of bits in order to be able to properly calculate entropy of the image (say 16-20 bits per choice)

4. We then adjust scaling factor using some search algorithm - like gradient descent type and go to step 3 until we find scale factor with least entropy level

Rationale in this approach is that improperly calibrated image will have higher entropy level / more information then properly calibrated image because there will be additional things present in image - dust shadows, gradients from poor flats, amp glow, ....

Of course, poor data can mess up this process - if we for example have wrong flats - algorithm will be thrown off as it will try to both compensate for dark current (correctly) and for poor flats (incorrectly) and will end up with mess.

[Xilman]

On 09/08/2023 at 10:51, Elp said:

I think the point was what does the final image look like? I use uncooled cameras more often than cooled, the difference in quality with the final image result isn't that different, granted temperatures here don't really get too warm. A lot of people use DSLRs which get quite warm perfectly fine.

An important question is: are you looking at the image or measuring it?

In the latter case noise is important but as long as the signal to noise ratio is high enough (and this depends on the desired accuracy of the measurement) it doesn't really matter that much as its contribution can also be estimated from the data.

[Elp]

Personally, if an image or video doesn't contain noise, it looks unnatural and synthetic. I usually leave a bit in or don't touch the noise.

[han59]

On 24/08/2023 at 11:57, vlaiv said:

We don't use darks to reduce noise. That is common misconception.

Yes, use as many darks as possible to remove the noise and keep pixel inequality.

 

On 24/08/2023 at 11:57, vlaiv said:

If we don't use matching darks we will not properly remove dark current. This as a result has wrong calibration (any value that we read of from image will be skewed) and also - flat calibration will fail because of residual dark signal that was not removed.

Best way to ensure that we remove dark current is to match the temperature as dark current is temperature dependent.

Yes, we can try dark scaling - but it is not guaranteed to work with every camera (or indeed with every set of data). We need "well behaved' sensor for it to work (not much messing in firmware and trying to optimize things or remove amp glow and such).

I have done an extensive automated test to find the best scaling factor . The noise of two darks was measured e.g: ∑(12x60sec_+5°C) - X * ∑(100x60sec_0°C) at the best X factor was found empirical.   See attached report.

Table 1 contains the best factors.

Table 2 contains the noise if no scaling is applied.

Table 3 contains the noise if best scaling factor is applied. Significant improvement above +10°C. Below +5°C the effect is limited

The following empirical formula works well for my ASI1600:

factor:=1/(exp(Δt * 0.1)
corrected dark:=dark * factor -  mean_dark * (1-factor)    I didn't have time to test it with real lights.

See pdf file for more details.

 

 

 

dark_test2.pdf

[sharkmelley]

I'm a bit confused by the expression:

 corrected dark:=dark * factor -  mean_dark * (1-factor) 

Normally the scaling factor is applied to the (bias-subtracted) master dark before it's used to calibrate the light.