Problem with BasicCCDParameters

[mgutierrez]

Hi guys,

I'm trying to do a sensor analysis of my Nikon D3300 dslr. BasicCCDParameters returns this output:

Gain is unexpectedly high, since ISO800 is near unity gain. The only way I get a consistent outcome is by setting a readout depth of 16bit. As far as I know "readout depth" corresponds with the bit depth the camera writes the data. That is, whether is rescaled or not. It seems is not rescaled in this case. A max value from statistics shows 4095 (in pixinsight stats module, in 16bit) and that makes me think that the camera does not rescale any value.

I think I'm missing something but not sure what/where/why.

Thanks in advance

[vlaiv]

2 hours ago, mgutierrez said:

Gain is unexpectedly high, since ISO800 is near unity gain.

High e/ADU value actually means low gain not high. Many electrons per single ADU value means high signal = low output value or a lot of light = darker image.

[mgutierrez]

you are right Vlaiv; I actually meant a high raw number. Thanks for pointing it out. In any case, I think it should be near 1

[vlaiv]

8 hours ago, mgutierrez said:

Thanks for pointing it out. In any case, I think it should be near 1

Maybe who ever measured e/ADU for ISO800 - measured scaled ADU values in range of 0-65535

In that case ADU is multiplied with 16 and values would be indeed ~1e/ADU

So something like 17e/ADU when ADU is 16 times larger - would be in fact 17/16 e/ADU

It is not "error" per se - it is wrong interpretation of the fact that ISO800 gives gain close to 1 - maybe it does, but only if values are scaled - if they are not scaled - you get 15-17 range like you have in that table?

[mgutierrez]

so you think @vlaiv that the analysis in the screenshot is correct, and that the adc converter really gives 1ADU per ~17 electrons at ISO800?

[vlaiv]

10 hours ago, mgutierrez said:

so you think @vlaiv that the analysis in the screenshot is correct, and that the adc converter really gives 1ADU per ~17 electrons at ISO800?

If you don't scale you data and leave it as 12bit, then yes. At 16bit scaled (multiplied with 16) - it would be very close to 1e/ADU.

[mgutierrez]

I'm confused @vlaiv. Unity gain is 1e=1adu, independently of the rescaling which happens later. According to the test I did, ISO800 gives 17e/adu. According to other websites, ISO800 is almost 1e/adu. I don't take into account if the adu is rescaled or not. It should not matter, no? I mean, gain is measured in e/adu (a true adu)

 

PS: I'm reading in another post that  "readout depth" parameter is a quite confusing name. It seems it has the same meaning as the dropdown that I see (for example) on stats module. That is, I have to put into "readout depth" in basciccdparameters the bit depth I need to put in stats (or other module) to see the true adu. In case of a DSLR, always (always?) 16bit, since they don't rescale. This would explain everything

[vlaiv]

ADU is short (confusingly) from analog-digital-unit (not sure who named it that way) and represents a number without unit that is assigned to a pixel upon readout.

We mostly think of pixels having value in unit of ADUs - but that is wrong as ADU does not have a unit nor it is itself a unit - meaning ADU from one camera and ADU from other camera can be and will be different (unlike for example meters or grams measured with different rulers or different scales - result will be the same).

Gain of a camera is given in e/ADU and is measure of how these particular ADU values are derived. So electron count has a unit and two different cameras should agree on electron count (if they have the same QE or QE has been adjusted for).

Same number of electrons and same gain should produce the same ADU number always.

But - let's discuss following case:

We have two cameras that are the same (or different - it does not matter in this case). They have 12bit ADC and can produce maximum value of 4095. You take one camera and record bright white light that saturates and record pixel value after gain conversion and you get 4095 in float point representation (so you don't know how this fits into 12 or 16 bit numbers).

Now you take second camera - record the same thing, but by convention you decide to exploit whole 16bits of 16bit format you'll be using to record image, so you multiply value with 16 and from 4095 you get number 65520. You then convert that to floating point representation.

You hand over those two images in floating point to me and say - gain was set equal on these two images.

Since I have different ADU values and you told me that gain is equal - only conclusion to draw from that is that we had different number of electrons to start with - but that is not the case, we had the same number of electrons. How come?

Well - e/ADU is valid only for the same scaling as scaling changes ADU value of pixel.

If you had 1000e to start with and you ended up with 4095 then your e/ADU is equal to 1000e/4095ADU = ~0.2442e/ADU

If you had 1000e to start with and you ended up with 65520 ADUs then your e/ADU is equal to 1000e / 65520 = ~0.0152625e/ADU

Although you used the same ISO in this case - your gain in terms of e/ADU changed because you introduced another step - scaling.

Makes sense?

[mgutierrez]

I completely agree and that makes sense, sure.

But:

Quote

Now you take second camera - record the same thing, but by convention you decide to exploit whole 16bits of 16bit format you'll be using to record image, so you multiply value with 16 and from 4095 you get number 65520

This is what I mean. If I run the basicccdparemeters correctly, I guess it should return 1e/adu. Cause, finally, both cameras with the same adc return the same. But the second rescales afterward. Sorry if I'm not explaining well.

I underlined "correctly" cause, according to other post I'm reading, I need to put 16bit in readout depth with cameras that do not rescale the result, just as in this case.

[vlaiv]

12 minutes ago, mgutierrez said:

But the second rescales afterward. Sorry if I'm not explaining well.

I underlined "correctly" cause, according to other post I'm reading, I need to put 16bit in readout depth with cameras that do not rescale the result, just as in this case.

First - I'm guessing that we use rescale term differently. To me proper value for 12bit ADC are in fact in 0-4095 range and when we write them in 16bit in 0-65535 range then we are scaling them to that.

I'm reading scaling as 12bit -> 16bit.

I have a sense you are reading it the other way around and that 16bit -> 12bit is scaling for you? Not that it matters for e/ADU part, but just to be on the same page.

Now, important part is - there is not rescaling afterwards. e/ADU measurement is last thing "in the chain" - because ADU values are as they will be on the end of the chain. You can't calculate e/ADU and then change ADU values and say still e/ADU applies.

This is why I told the story of you handing over data in floating point. e/ADU for that data must be valid afterwards so I can perform "backward" transformation and calculate electrons from ADU readings that I perform.

In any case - if you have 16e/ADU in 12 bit mode - it is perfectly normal to expect 1e/ADU in 16bit mode, provided that conversion between 12bit mode and 16bit mode is done "the standard way" - by multiplying pixel values with 16 (two to the power of difference between bit counts so 2^(16-12) = 2^4 = 16).

Here is example of what might happen:

you capture 16000e and you end up with 1000ADU - you do the math 16000e / 1000ADU = 16e/ADU

Now you do the same - but this time using 16 bit mode:

you capture 16000e and you end up with 1000ADU but this time - you multiply that number with 16 to scale it to 0-65535 range and you get 16000ADU. Now you calculate e/ADU conversion factor by 16000e / 16000ADU and you end up with 1e/ADU.

So for same ISO and same number of captured electrons - we end up with two different e/ADU values because we decided to mess with actual pixel values and scale them.

[mgutierrez]

8 minutes ago, vlaiv said:

you capture 16000e and you end up with 1000ADU but this time - you multiply that number with 16 to scale it to 0-65535 range and you get 16000ADU. Now you calculate e/ADU conversion factor by 16000e / 16000ADU and you end up with 1e/ADU.

and this is where I think I'm getting confused.

The camera's adc outputs, let's say, numbers. Then, that number is multiplied or not (I don't know if the camera itself or the driver, or whatever). If the number is multiplied, for me that's not ADU anymore. Other users call them DN (digital number?). For me, ADU is exactly the adc output without altering it in any way. And that's the number I've always thought that gain definition (e-/adu) refers to.

In the second case, I cannot say that the gain is 16000e/16000adu but 16000e/(16000/16) adu. That is, 16e/adu. I could say that "gain" is also 16000e/16000dn=1e/dn

Thanks again vlaiv

[vlaiv]

5 minutes ago, mgutierrez said:

and this is where I think I'm getting confused.

The camera's adc outputs, let's say, numbers. Then, that number is multiplied or not (I don't know if the camera itself or the driver, or whatever). If the number is multiplied, for me that's not ADU anymore. Other users call them DN (digital number?). For me, ADU is exactly the adc output without altering it in any way. And that's the number I've always thought that gain definition (e-/adu) refers to.

In the second case, I cannot say that the gain is 16000e/16000adu but 16000e/(16000/16) adu. That is, 16e/adu. I could say that "gain" is also 16000e/16000dn=1e/dn

Thanks again vlaiv

You are welcome.

I'd just like to point out one thing - e/ADU is used to convert from ADU back to electrons. If you don't know ADUs have been altered after e/ADU is specified - you can never get correct e-values back. In that sense, e/ADU should be stated for last ADU that you output and not just from first step from ADC.

[mgutierrez]

2 minutes ago, vlaiv said:

If you don't know ADUs have been altered after e/ADU is specified - you can never get correct e-values back

absolutely. You need to know that. It seems most of the dslr won't rescale (multiply by 2^16/2^adcbit). In fact an oversaturated flat from my nikon shows a max value of 4095 in pixinsight stat's module in 16bit format. That's why I need to put 16bit in readout depth within basicccdparameters so it returns an ~1e/adu of gain; as some websites report

[vlaiv]

3 minutes ago, mgutierrez said:

absolutely. You need to know that. It seems most of the dslr won't rescale (multiply by 2^16/2^adcbit). In fact an oversaturated flat from my nikon shows a max value of 4095 in pixinsight stat's module in 16bit format. That's why I need to put 16bit in readout depth within basicccdparameters so it returns an ~1e/adu of gain; as some websites report

I think that it really depends on implementation of how .cr2 is read.

I sometimes get very strange values with command line DCRAW if I specify that I want raw output - minimally modified.

In any case, you should not concern yourself with actual numbers as long as e/ADU and ADU values match. Now at least know where discrepancy comes from, but all that 16bit scaling thing is simply unnecessary and causing issues. My belief is that ADC numbers should be kept as they are regardless of bit format used to record them.

Unfortunately, because of the way software works - developers decided to step in and change the numbers to suit people's expectations. If you over expose your camera - you expect to get the white screen, right? If your camera is 12bit and you record that as 16bit - you'll get only 1/16th of full range that bit format supports.

Now put that image in software that works with 16 bit images (and expect full 16bit range to be utilized) - and you get very dark gray image. Not what people expect.

So developers stepped in and said - ok, let's stretch intensity to 0-1 as that is how it is expected (I wrote 0-1 as it is universal thing regardless of bit format - for 16 bit it should read 0-65535 and in general from 0-max). Now we no longer have numbers that are result of ADC - but do have numbers that average Joe expects.

[mgutierrez]

completely agreed with your statements.

Thanks vlaiv; it's a pleasure to have such conversations with you.