I read somewhere that the D7500 uses a two stage amplifier from around 400iso up.  In order to see if this was playing a part I repeated the test at 250iso.  The result was essentially the same - at around 20deg C, the D5300 has much less thermal pattern noise than the D7500. 

A very interesting read!  There are very few people doing proper testing of thermal noise in DSLR cameras but it is the one thing that can make or break an image, especially on the warm nights that we rarely have in the UK.

A couple of comments on the histograms.  Both of them seem to be clipped at the low end i.e. the left hand tail of the histogram is missing.  Why is this - I thought Nikon had fixed the black clipping issue?   The multi-peak nature of D7500 histogram is very strange indeed.  It appears to indicate different populations of pixels behaving differently.  I don't recall ever seeing this before.

If you still have both cameras you could potentially perform a side by side test to measure how fast the noise rises from frame to frame.  This might provide a lot of useful information.  It's the approach I took on the testing in the graph below:

Successive 5 minute exposures were taken from a camera switched on after acclimatising to the 20C ambient temperature.  The noise in a difference frame (2 successive darks frames subtracted) was measured and converted to a dark current estimate for each point in the graph.

The reason I subtract the dark frames is so the graph represents the thermal noise levels expected from calibrated frames.

If you have a good set of darks taken under controlled conditions I'd love to see how they would compare on that graph.  Generally speaking it's the Canons that become very noisy.

[Later Edit]  I've realised that the black clipping in the histogram is because you've bias subtracted the master dark.  The shape of the D7500 histogram is very puzzling though.   Unfortunately both your histograms go off the top of the scale, so it's difficult to make a meaningful comparison.

Mark

13 hours ago, sharkmelley said:

 I've realised that the black clipping in the histogram is because you've bias subtracted the master dark.  The shape of the D7500 histogram is very puzzling though.   Unfortunately both your histograms go off the top of the scale, so it's difficult to make a meaningful comparison.

Thank you for your reply Mark.

I was focused on trying to determine the extent of the differences in the pattern noise and this is why I was looking at the relatively brighter portions of the master darks.  I have been less concerned with the true random noise in each sub as I can deal with this by taking more subs but the pattern in that noise is far more difficult to address.  I will dig out and post a different view of the histogram with a lower zoom of the Y axis.

Ok some more histograms ...  unfortunately I think I might have deleted some of the files so I these are at different ISO values ( 250 for the D5300, 400 for the D7500 )

These have not been bias corrected.

D5300: 

D7500: 

D5300: 

D7500: 

Note that a significant portion of the left hand part of the histogram for the D5300 is due to bias and is removed by subtracting the master bias ( refer histogram in original post ).  This is not so much the case with the D7500.  

The D7500 histogram clearly shows the very large number of ‘warm pixels’ that, whilst randomly distributed across the sensor, are in fixed positions in each frame and result in significant ‘pattern noise’.

From your histograms it certainly does look like the D7500 sensor has a greater number of outlying values.  On the other hand those histograms also support your earlier blog where you stated that the dark noise in the D7500 is lower.  Your bias subtracted darks are an anomaly because of the large number of clipped values.

I think the overall message is that the D7500 is an improvement over the D5300 as long as you use well matched darks.  But that tentative conclusion needs to be tested further.

Mark

7 hours ago, sharkmelley said:

From your histograms it certainly does look like the D7500 sensor has a greater number of outlying values.  On the other hand those histograms also support your earlier blog where you stated that the dark noise in the D7500 is lower.  Your bias subtracted darks are an anomaly because of the large number of clipped values.

I think the overall message is that the D7500 is an improvement over the D5300 as long as you use well matched darks.  But that tentative conclusion needs to be tested further.

Mark

Hey Mark

The problem I have is that with an un-cooled camera, quickly changing night time temperatures and no on-sensor temperature sensor it is practically impossible for me to accurately match darks.  So, ideally, I want a camera with very low thermal pattern noise ( as opposed  to true random noise that I can reduce somewhat with more subs ).  

My thinking was that the calibrated master darks show that both cameras have thermal pattern noise but I thought they indicated that the level of that signal in the D5300 is much less (?).  So I do not really understand your comment that “ Your bias subtracted darks are an anomaly because of the large number of clipped values”.  I thought that the histogram of the bias calibrated master darks shows the uncorrected pattern noise that will be present in the bias corrected lights.  

That is, in my current workflow, I have not found it necessary to use a master dark with the D5300.  So I thought that the bias corrected darks indicate how much worse off I will be if use a D7500 without using a master dark.

I guess another way of looking at is this...

The D7500 has an almost constant value across the sensor in the master bias frame ( with only a couple of ADU variation along the bottom edge ).  So, subtracting the master bias ( other than shifting the peak ) has very little impact on the master dark.  However, the D5300 master bias has significant variation of values ( ie.  a significant pattern in the bias frame ) and subtracting this from the master dark has significant impact.  That is, the histogram of the D5300 master dark is made up of a significant ( mostly temperature independent ) bias component + the temperature dependent thermal pattern noise.  Subtracting the master bias leaves a relatively small thermal pattern component.  Whereas, the histogram of the D7500 master dark is essentially all thermal pattern noise.  That is why I thought the calibrated master darks were a good indication of the relative levels of thermal pattern noise in the two sensors.

But I guess I am only focused in my analysis on the fixed position signals showing up in the bias and dark frames ( the former because I can simply remove them and the latter because I can’t ).  What do you think, is my approach sound or is there a better way of trying to determine which sensor will have the greater problem with patterns in the thermal noise signals?

Thanks

Mike

7 hours ago, sharkmelley said:

Your bias subtracted darks are an anomaly because of the large number of clipped values.

Mark, 

Perhaps it would be more appropriate to add an offset to the master darks before I subtract the bias frames so as to prevent clipping - would that be a better representation of the non-random signal in the master darks that is due to fixed pattern variations in the thermal response of the pixels across the sensor ( what I have been calling “thermal pattern noise”)?   What do you think?

Cheers

Mike

I have taken the master darks, added an offset, subtracted the master bias files,  scaled the gains so that the D7500 @ 400ISO is roughly equiavlent to the D5300 at 250ISO, and shifted them so that the medians line up with the following results.

Master Darks ( 100% crops of the centre, identical screen transfer function applied via histogram tool )

D5300:  

D7500:  

Histograms @ 1x vertical scale:

D5300:  

D7500:  

Histograms at 23x vertical scale:

D5300:  

D7500:  

I have positioned the the tab marks at the bottom of the histograms to encompass 99% of pixels in the relevant master dark.

That is, 99% of pixels are in the range:

D5300 :  -8 to + 16

D7500:  -4 to + 97

So, I guess I might say that ( in the range ~ -8 to 8 about the median) the D5300 has more cool and slightly warm pixels than the D7500 but the D500 has way more very warm and hot pixels.  For example, in the inital examination ( from the initial post above ) the D5300 had ‘only’ 6000 pixels above 44 whereas the D7500 had 140,000 above 44.

( note that whilst subtracting the bias from the offset D5300 master dark did make a difference, the difference was not as large as I anticipated in a previous post ).

There is something I find odd about the results from the D7500.  Nikon cameras use a spatial filtering algorithm in longer exposures in order to remove hot/warm pixels.  It's not a severe algorithm like the "star eater" spatial algorithm used on Sony cameras but it should still remove isolated hot/warm pixels.  So I'm surprised to see such a long right-hand-side histogram tail. 

I would certainly be interested in taking a look at a representative raw dark frame from both the D5300 and D7500.

Mark

I have a D5300 I'd you want a dark ?

Tell me what ISO and exposure you want and 'rough' temperature. 

8 hours ago, sharkmelley said:

There is something I find odd about the results from the D7500.  Nikon cameras use a spatial filtering algorithm in longer exposures in order to remove hot/warm pixels.  It's not a severe algorithm like the "star eater" spatial algorithm used on Sony cameras but it should still remove isolated hot/warm pixels.  So I'm surprised to see such a long right-hand-side histogram tail. 

I would certainly be interested in taking a look at a representative raw dark frame from both the D5300 and D7500.

Mark

Hi Mark

please see below links to single dark frames:

5300, 240sec, ISO250 : https://www.dropbox.com/s/fu27b1tx5wbu00c/D5300_dark_240s_250iso__030.nef?dl=0

7500, 240sec, ISO400: https://www.dropbox.com/s/cymscuakzpcu9mp/D7500_240s_400iso__0030.nef?dl=0

Cheers

Mike

15 hours ago, MikeODay said:

Hi Mark

please see below links to single dark frames:

5300, 240sec, ISO250 : https://www.dropbox.com/s/fu27b1tx5wbu00c/D5300_dark_240s_250iso__030.nef?dl=0

7500, 240sec, ISO400: https://www.dropbox.com/s/cymscuakzpcu9mp/D7500_240s_400iso__0030.nef?dl=0

Cheers

Mike

 

Thanks for those.  There's nothing obvious that strikes me about them immediately, by means of explanation.  But the D7500 certainly seems to have more hot/warm pixels.  I will take a closer look later.

Mark

Cheers Mark

@MikeODay

Very informative read , thankyou Mark.

What was your conclusion in the end ? I have the opportunity to buy a 7500 soon.

Thanks for reviving this thread.  I've been playing with a Nikon D5300 recently and noticed that the spatial filtering (a.k.a. hot pixel suppression or HPS)was causing all kinds of problems with star colours - mainly turning them green, just like the latest variant of the Sony star eater algorithm.

It's all being discussed over on Cloudy Nights:  https://www.cloudynights.com/topic/635441-aa-filter-spatial-filter-and-star-colours/

It's early days and it's not yet fully understood.  However, it appears that the Nikon D810A (the one built for astrophotography) doesn't have the same issue because the spatial filtering is much less aggressive.

So it's just possible that the D7500 is also using less aggressive spatial filtering than the D5300 and that's why it appears noisier.  I'll take a new look at Mike's D7500 darks and see what I can determine.  It'll definitely provide another interesting data point for my analysis.

Mark

I'm looking at the D7500 as an all use camera or maybe a D5600 + a Hypercam 183, really not sure way to go yet

Having looked at this in more detail, it's certainly true that the D7500 is using less aggressive spatial filtering than the D5300.  However this does not explain the observed difference in thermal fixed pattern noise (FPN).   The D7500 really does appear to have a higher FPN than the D5300.

The practical consequence of this is that you are more likely to need dark frame calibration with the D7500 than with the D5300.

Mark