Can someone explain why we need an offset value?

[StuartT]

I read the following here:

The offset value of a camera is a little bit more straightforward. This automatically fills a small amount of the pixel-bucket before any light is gathered. This is to ensure that there are no 0-value pixels on each frame. We want to avoid 0-value pixels at all cost because we then lose any small amount of information that might be there. It is impossible to create information out of nothing, so ensuring that no pixel has a value of 0 when it is read out helps us.

What I don't understand is, if there is a tiny bit of signal, then there wouldn't be any zero value pixels anyway surely?

i.e. if the tiny bit of signal is 1 and we have no offset, we'll end up with 0+1=1 So why would setting an offset of, say, 25 help us? That just means we have 26. Either way, it's not a zero

Edited December 11, 2022 by StuartT

[PeterCPC]

How about this:

OFFSET

Offset refers to the DC component of a video or image signal, and effectively sets the black level of the image. The black level is the pixel level (in electrons, or volts) which corresponds to a pixel value of zero. This is often used with a histogram to ensure the full use of the camera bit-depth, effectively raising signal-to-noise. Pushing non-black pixels to zero lightens the image, although it gives no improvement in the data. By increasing the black level, offset is used as a simple machine vision image processing technique for brightening and effectively creating a threshold (setting all pixels below a certain value to zero to highlight features) for blob detection.

[symmetal]

It's to ensure that the analogue signal from the camera lies within the input range of the ADC (analogue to digital converter). The analogue amplifiers processing the signal before presenting it to the ADC, and the ADC itself need to work on a signal significantly away from the power rails of the amplifiers to ensure they can give a linear response. The camera offset effectively adds an analogue voltage to the camera output to ensure it's within the acceptable input range of the ADC and avoids black clipping or zero ADC output values.

Alan

[StuartT]

err...

thanks 😬

[symmetal]

3 hours ago, StuartT said:

err...

thanks 😬

You can see if your camera offset is sufficient by looking at a the image statistics of a bias frame. Every image will include the camera bias signal added to it and during calibration this bias component is subtracted. If this bias frame contains zero value pixels calibration won't work correctly.

Noise will also be superimposed on the bias frame so while adding say 50 offset should add 50 ADU to each pixel, if you examine individual pixels, while most will be a value near 50, some will be higher and others lower, due to noise. Noise can add as well as subtract from the ideal pixel value and a few pixels may be ADU 20 or so.

Looking at the bias image statistics this will give the average ADU value as well as the minimum and maximum ADU value. Offset is added until this minimum ADU value is always above zero, with a bit more added on for safety.

So the offset ensures that the camera analogue output with positive and negative noise values added will always be higher than the minimum analogue voltage the ADU will accept to do a successful conversion.

Image calibration will subtract this offset and any other fixed signal values added or subtracted during the read process, but the noise will remain. Stacking many images reduces this noise component.

The number of actual ADU added by increasing the camera offset by 1 depends on the camera design and the camera bit depth, but I assumed it's 1 here to make it a bit simpler.

Hope that helps more and isn't more confusing. 😊

Alan

[StuartT]

17 minutes ago, symmetal said:

You can see if your camera offset is sufficient by looking at a the image statistics of a bias frame. Every image will include the camera bias signal added to it and during calibration this bias component is subtracted. If this bias frame contains zero value pixels calibration won't work correctly.

Noise will also be superimposed on the bias frame so while adding say 50 offset should add 50 ADU to each pixel, if you examine individual pixels, while most will be a value near 50, some will be higher and others lower, due to noise. Noise can add as well as subtract from the ideal pixel value and a few pixels may be ADU 20 or so.

Looking at the bias image statistics this will give the average ADU value as well as the minimum and maximum ADU value. Offset is added until this minimum ADU value is always above zero, with a bit more added on for safety.

So the offset ensures that the camera analogue output with positive and negative noise values added will always be higher than the minimum analogue voltage the ADU will accept to do a successful conversion.

Image calibration will subtract this offset and any other fixed signal values added or subtracted during the read process, but the noise will remain. Stacking many images reduces this noise component.

The number of actual ADU added by increasing the camera offset by 1 depends on the camera design and the camera bit depth, but I assumed it's 1 here to make it a bit simpler.

Hope that helps more and isn't more confusing. 😊

Alan

I've just looked at the pixel values in my master bias (using PI statistics) and they are between 0 and 1 (so normalised). I assume you want the 16-bit values? (as it's a 16-bit camera). Is this what you mean by ADU? If so here they are.

Master_bias_1x1_15C_gain_105_offset_50

count (%) 100.00000
count (px) 26091648
mean 500.326
median 500.219
avgDev 0.795
MAD 0.667
minimum 478.881
maximum 554.740

 

So they are a lot more than 50! I'm not really sure what I am meant to be looking at here. for comparison, here is a master dark

Master_dark_35sec_gain_105_offset_50_bin_15_C_bin1x1

count (%) 100.00000
count (px) 26091648
mean 500.328
median 500.280
avgDev 1.092
MAD 0.920
minimum 468.100
maximum 23832.271

 

(it's interesting that the dark frame is not lighter on average than the bias frame. I assume that means this camera has negligible dark current?)

 

Edited December 11, 2022 by StuartT

[Stuart1971]

6 hours ago, StuartT said:

I read the following here:

The offset value of a camera is a little bit more straightforward. This automatically fills a small amount of the pixel-bucket before any light is gathered. This is to ensure that there are no 0-value pixels on each frame. We want to avoid 0-value pixels at all cost because we then lose any small amount of information that might be there. It is impossible to create information out of nothing, so ensuring that no pixel has a value of 0 when it is read out helps us.

What I don't understand is, if there is a tiny bit of signal, then there wouldn't be any zero value pixels anyway surely?

i.e. if the tiny bit of signal is 1 and we have no offset, we'll end up with 0+1=1 So why would setting an offset of, say, 25 help us? That just means we have 26. Either way, it's not a zero

What you are doing is adding a tiny bit of signal to all areas of the image, even the darkest areas with very little light hitting them, so with a value of 1, this will only help the bright areas, you need a value of approx 25 offset, to add signal to the darkest areas of an image…if the whole image is very bright right into the corners, then you could use a lower offset indeed…but that it normally not the case at all…

[StuartT]

22 minutes ago, Stuart1971 said:

What you are doing is adding a tiny bit of signal to all areas of the image, even the darkest areas with very little light hitting them, so with a value of 1, this will only help the bright areas, you need a value of approx 25 offset, to add signal to the darkest areas of an image…if the whole image is very bright right into the corners, then you could use a lower offset indeed…but that it normally not the case at all…

I'm still not really getting what is so bad about a 0 pixel though. If there is no signal, isn't it ok for it to be 0? And if there is a tiny bit of signal then it won't be zero anyway (because the signal will give it a value). 

Or is it because noise can be negative as well as positive, so you have to add a baseline signal so that you avoid having a negative pixel value?

Edited December 11, 2022 by StuartT

[symmetal]

I assume this is your ASI2600 and these values are normal. Increasing offset by 1 adds 10 ADU, so your standard offset of 50 adds 500 ADU which may seem high, as your minimum ADU value on your dark is 468. In reality this is not significant as your output dynamic range has dropped from 65536 ADU to 65036 ADU which is a drop of 0.7% which is insignificant.

If you increase the camera gain then your minimum ADU value will drop as the noise is amplified as well as the signal. The default setting is there to cope with all usual gain values. If you're never going to go higher than your current gain then you could likely use an offset of say 20 and get away with it. When the ASI1600 first came out it was normal to change the offset along with the gain to maximize dynamic range but it was soon realised what a pain this was as you needed separate bias possibly darks for each offset change, and it was changed to a fixed higher offset for all gain settings.

43 minutes ago, StuartT said:

I'm still not really getting what is so bad about a 0 pixel though. If there is no signal, isn't it ok for it to be 0? And if there is a tiny bit of signal then it won't be zero anyway (because the signal will give it a value). 

A zero pixel value can't be stretched in processing and will always be zero giving black dots in your image. If there is a tiny bit of signal it can actually end up below 0 due to the negative values of noise added. These negative values will just be clipped to 0 giving more black dots.

Here's a master bias frame from an ASI224 as an example. The histogram peak is at 314 ADU, set by the offset, but note the bias image data goes from 223 to 602 ADU due to the noise distribution. With no offset used, all the noise values below the histogram peak will be set to ADU 0 and no amount of image stacking will reduce the noise present in the image at those pixels with ADU 0. They will remain as annoying black areas in your image.

This is at gain 134, offset 20 as indicated in the filename. Being a 12 bit camera all ADU values are multiplied by 16 so offset 20 becomes 20 * 16 = 320 ADU. Note the peak is lower at 314 so the actual pixel zero signal output is slightly below the minimum value the ADU can accept. Another reason for adding an offset. The gaps in the histogram are due to presenting 12 bit data on a 16 bit graph.

Alan

Edited December 11, 2022 by symmetal

[newbie alert]

Not fully understand it but with a ccd it's factory preset.. with CMOS it's has a variable gain so the offset is to set the blackpoint( I think) 

 

[Stuart1971]

The whole idea is not to have any pure black pixels and no pure white pixels, so neither are clipped….

[scotty38]

@dghent Dale has a good writeup here:

https://daleghent.com/2020/08/understanding-camera-offset

[symmetal]

6 hours ago, Stuart1971 said:

The whole idea is not to have any pure black pixels and no pure white pixels, so neither are clipped….

There will almost always be white clipped pixels in astro images due to the stars but they don't cause a problem. Black clipped pixels are an issue only because we apply massive amount of stretching in the dark areas of the image to bring out the details, and black pixels which can't be stretched, then stick out like sore thumbs. 🙂 The white image areas are never stretched, and are often compressed so a clipped white pixel doesn't look bad.

Increasing offset will actually increase the number of white clipped pixels as it shifts the whole histogram uniformly to the right, and it's why it's called brightness in some capture programs.

Alan

[StuartT]

Thanks everyone. I think I kind of understand this. I think I need to do some more reading though.

[StuartT]

(ok. Full disclosure. I don't understand this at all)

Surely what we are all trying to capture here is light. So pixels with interesting information in them will be non-zero (i.e. light). Black pixels are just space. So why should we care about them?

Edited December 13, 2022 by StuartT

[PeterCPC]

You are basically trying to avoid black clipping any data by moving the black point.

[sharkmelley]

The reason for the offset is to ensure that the values coming from ADC are unbiased, given that the ADC has read noise.  Suppose the offset is 100 and the pixels have received no light.  The ADC will report some pixels as 100, some as below 100 and some as above 100, all because of the read noise.  But the average is exactly 100, which is the correct (i.e. unbiased) value for a pixel that has received no light.

Now suppose the offset is zero and again the pixels have received no light.  The ADC will report some pixels as 0 and some above 0 but it cannot report values below zero because the ADC does not produce negative values.  Therefore, the average ADC value will be greater than 0 which means the ADC is giving a biased estimate of the true value.  This would cause all sorts of problems for low light imaging.

Mark

[StuartT]

5 hours ago, sharkmelley said:

The reason for the offset is to ensure that the values coming from ADC are unbiased, given that the ADC has read noise.  Suppose the offset is 100 and the pixels have received no light.  The ADC will report some pixels as 100, some as below 100 and some as above 100, all because of the read noise.  But the average is exactly 100, which is the correct (i.e. unbiased) value for a pixel that has received no light.

Now suppose the offset is zero and again the pixels have received no light.  The ADC will report some pixels as 0 and some above 0 but it cannot report values below zero because the ADC does not produce negative values.  Therefore, the average ADC value will be greater than 0 which means the ADC is giving a biased estimate of the true value.  This would cause all sorts of problems for low light imaging.

Mark

Perfect! Thanks so much. I get it now. 🙂