MikeODay

Cheers Mark

8 hours ago, sharkmelley said:

There is something I find odd about the results from the D7500.  Nikon cameras use a spatial filtering algorithm in longer exposures in order to remove hot/warm pixels.  It's not a severe algorithm like the "star eater" spatial algorithm used on Sony cameras but it should still remove isolated hot/warm pixels.  So I'm surprised to see such a long right-hand-side histogram tail. 

I would certainly be interested in taking a look at a representative raw dark frame from both the D5300 and D7500.

Mark

Hi Mark

please see below links to single dark frames:

5300, 240sec, ISO250 : https://www.dropbox.com/s/fu27b1tx5wbu00c/D5300_dark_240s_250iso__030.nef?dl=0

7500, 240sec, ISO400: https://www.dropbox.com/s/cymscuakzpcu9mp/D7500_240s_400iso__0030.nef?dl=0

Cheers

Mike

I have taken the master darks, added an offset, subtracted the master bias files,  scaled the gains so that the D7500 @ 400ISO is roughly equiavlent to the D5300 at 250ISO, and shifted them so that the medians line up with the following results.

Master Darks ( 100% crops of the centre, identical screen transfer function applied via histogram tool )

D5300:  

D7500:  

Histograms @ 1x vertical scale:

D5300:  

D7500:  

Histograms at 23x vertical scale:

D5300:  

D7500:  

I have positioned the the tab marks at the bottom of the histograms to encompass 99% of pixels in the relevant master dark.

That is, 99% of pixels are in the range:

D5300 :  -8 to + 16

D7500:  -4 to + 97

So, I guess I might say that ( in the range ~ -8 to 8 about the median) the D5300 has more cool and slightly warm pixels than the D7500 but the D500 has way more very warm and hot pixels.  For example, in the inital examination ( from the initial post above ) the D5300 had ‘only’ 6000 pixels above 44 whereas the D7500 had 140,000 above 44.

( note that whilst subtracting the bias from the offset D5300 master dark did make a difference, the difference was not as large as I anticipated in a previous post ).

7 hours ago, sharkmelley said:

Your bias subtracted darks are an anomaly because of the large number of clipped values.

Mark, 

Perhaps it would be more appropriate to add an offset to the master darks before I subtract the bias frames so as to prevent clipping - would that be a better representation of the non-random signal in the master darks that is due to fixed pattern variations in the thermal response of the pixels across the sensor ( what I have been calling “thermal pattern noise”)?   What do you think?

Cheers

Mike

7 hours ago, sharkmelley said:

From your histograms it certainly does look like the D7500 sensor has a greater number of outlying values.  On the other hand those histograms also support your earlier blog where you stated that the dark noise in the D7500 is lower.  Your bias subtracted darks are an anomaly because of the large number of clipped values.

I think the overall message is that the D7500 is an improvement over the D5300 as long as you use well matched darks.  But that tentative conclusion needs to be tested further.

Mark

Hey Mark

The problem I have is that with an un-cooled camera, quickly changing night time temperatures and no on-sensor temperature sensor it is practically impossible for me to accurately match darks.  So, ideally, I want a camera with very low thermal pattern noise ( as opposed  to true random noise that I can reduce somewhat with more subs ).  

My thinking was that the calibrated master darks show that both cameras have thermal pattern noise but I thought they indicated that the level of that signal in the D5300 is much less (?).  So I do not really understand your comment that “ Your bias subtracted darks are an anomaly because of the large number of clipped values”.  I thought that the histogram of the bias calibrated master darks shows the uncorrected pattern noise that will be present in the bias corrected lights.  

That is, in my current workflow, I have not found it necessary to use a master dark with the D5300.  So I thought that the bias corrected darks indicate how much worse off I will be if use a D7500 without using a master dark.

I guess another way of looking at is this...

The D7500 has an almost constant value across the sensor in the master bias frame ( with only a couple of ADU variation along the bottom edge ).  So, subtracting the master bias ( other than shifting the peak ) has very little impact on the master dark.  However, the D5300 master bias has significant variation of values ( ie.  a significant pattern in the bias frame ) and subtracting this from the master dark has significant impact.  That is, the histogram of the D5300 master dark is made up of a significant ( mostly temperature independent ) bias component + the temperature dependent thermal pattern noise.  Subtracting the master bias leaves a relatively small thermal pattern component.  Whereas, the histogram of the D7500 master dark is essentially all thermal pattern noise.  That is why I thought the calibrated master darks were a good indication of the relative levels of thermal pattern noise in the two sensors.

But I guess I am only focused in my analysis on the fixed position signals showing up in the bias and dark frames ( the former because I can simply remove them and the latter because I can’t ).  What do you think, is my approach sound or is there a better way of trying to determine which sensor will have the greater problem with patterns in the thermal noise signals?

Thanks

Mike

Ok some more histograms ...  unfortunately I think I might have deleted some of the files so I these are at different ISO values ( 250 for the D5300, 400 for the D7500 )

These have not been bias corrected.

D5300: 

D7500: 

D5300: 

D7500: 

Note that a significant portion of the left hand part of the histogram for the D5300 is due to bias and is removed by subtracting the master bias ( refer histogram in original post ).  This is not so much the case with the D7500.  

The D7500 histogram clearly shows the very large number of ‘warm pixels’ that, whilst randomly distributed across the sensor, are in fixed positions in each frame and result in significant ‘pattern noise’.

13 hours ago, sharkmelley said:

 I've realised that the black clipping in the histogram is because you've bias subtracted the master dark.  The shape of the D7500 histogram is very puzzling though.   Unfortunately both your histograms go off the top of the scale, so it's difficult to make a meaningful comparison.

Thank you for your reply Mark.

I was focused on trying to determine the extent of the differences in the pattern noise and this is why I was looking at the relatively brighter portions of the master darks.  I have been less concerned with the true random noise in each sub as I can deal with this by taking more subs but the pattern in that noise is far more difficult to address.  I will dig out and post a different view of the histogram with a lower zoom of the Y axis.

I read somewhere that the D7500 uses a two stage amplifier from around 400iso up.  In order to see if this was playing a part I repeated the test at 250iso.  The result was essentially the same - at around 20deg C, the D5300 has much less thermal pattern noise than the D7500. 

Thanks Robertl; by all accounts Caroline Herschel was a remarkable person.

Thanks Mark for the head's up about the book - I'll see if I can find it online and not too expensive.