Jump to content

Stargazers Lounge Uses Cookies

Like most websites, SGL uses cookies in order to deliver a secure, personalised service, to provide social media functions and to analyse our traffic. Continued use of SGL indicates your acceptance of our cookie policy.

sgl_imaging_challenge_banner_dslr_mirrorlesss.thumb.jpg.5b348d6a5e7f27bdcb79e9356b7fc03b.jpg

Sign in to follow this  
  • entries
    4
  • comments
    17
  • views
    6,008

D7500 has high thermal pattern noise

MikeODay

3,097 views

In an earlier post I examined the noise in single dark frames over a range of exposure times.  My conclusion was that the Nikon D7500 was a lower noise camera than the D5300.  This was also backed up by an examination of master bias frames that again strongly favoured the D7500.

However ...

The first image I attempted to produce with the D7500 exhibited very strong streaks in the noise.

5a4d7733d427e_Galaxy_NGC_1365_Oct_2017_100pcCrop-IPADPro-compressed.thumb.jpg.18041249e7d07d813d6d7aff262acc13.jpg

5a4d781b23bf6_Galaxy_NGC_1365_Oct_2017_100pcCrop-IPADPronoise.thumb.jpg.4bbe1718af4186a65f611cd7c5516ad4.jpg

I had been in the habit of not using dark subtraction with the D5300 as it has very low thermal pattern noise.  Accordingly, I again only used bias and flat frame calibration in the workflow that produced the above images.  

Whilst the streaks are due to patterns in the noise being spread across the image due to errors in the application of dithering during tracking, it did indicate that the D7500 did have significant thermal pattern noise.  I found this surprising because the noise in individual frames ( when looked at in isolation ) seems to be completely random.  I thought that perhaps my memory had failed me and maybe the D5300 has the same level of pattern noise but my memory was being tricked.  That is, all my recent images with the D5300 were taken at lowish air temperatures ( ~5 deg or so ) whereas the image above was captured on warm nights ( low 20s ) and so maybe the D5300 would be just as bad at higher temperatures.   

To test this I produced bias corrected master darks for both the D7500 and D5300 from images all taken at around 20 deg or just over. 

The images below have all been stretched using the same screenTransferFunction applied to the Pixinsight histogram tool.  The results are striking ...

D5300 master dark ( 47 subs, bias corrected ) - red channel:

5a4d7c06d91e1_D530047integration_CFA0_R_small.jpg.e0a09fbfa623e7641252ca8611e30d02.jpg

D7500 master dark ( 281 subs, bias corrected ) - red channel:

5a4d7c30ede0b_D7500281integration_CFA0_Rsmall.jpg.65fd1ddc6f1598cb1f6c9497e4c981f0.jpg

D5300 master dark 100% centre crop - red channel:

5a4d7c298011c_D530047integration_CFA0_R_100pccentrecrop.jpg.608f678dc78cef5474f4fb8da0045a54.jpg

D7500 master dark 100% centre crop - red channel:

5a4d7c215701e_D7500281integration_CFA0_R100pccentrecrop.jpg.610e2669d6fbcd092d25e67f7b430527.jpg

And the histograms of the full size images ( red channels ) ...

D5300 master dark ( red channel ) histogram:

5a4d7c193d62c_D530047integration_CFA0_R_hist.thumb.jpg.e62b42cec1f256d7b5a06dbc7f65d48b.jpg

D7500 master dark ( red channel ) histogram:

5a4d7c1423da3_D7500281integration_CFA0_Rhist.thumb.jpg.d0e579c13ce8468de6daac2171ffbe28.jpg

The Pixinsight statistics tool calculates the following:

D5300:  mean 2.3,  standard deviation 9.3
D7500:  mean 7.5,  standard deviation 20.8

-------------

Analysis:

The images and histograms clearly show that the D7500 has higher pattern noise than the D5300.  In particular, from the histograms, 0.1% ( 6,286 ) of D5300 pixels are more  than 44 ADU whereas, for the D7500 this figure is 27 times as great at 2.7% ( 141,305 pixels ).  Furthermore, the master dark for the D5300 was only produced using 47 images -v- 281 for the D7500 so I would expect that this difference would be even higher with more D5300 frames. 

On the other hand, whilst not shown in the histograms above, my D5300 does have more 'very hot' pixels than the D7500 ( 579 pixels greater than 400 ADU -v- 10 pixels greater than 400 ADU ).  However, these hot pixels are very easily removed via dithering during tracking and sigma clipping when integrating.  The very large number of warm pixels however are very difficult to remove as dithering just places different warm pixels on top of each other.

I went back and examined the 'random' noise seen in the individual D7500 dark frames ... and yes they do look random when seen individually, however, when flicking between a number of frames it is clear that the 'random' pattern is repeated in each frame! 

Conclusion:

My D7500 has very significant thermal pattern noise, albeit randomly distributed in a fixed pattern.

 

............

 

Next steps (?)

- I could use in dark subtraction during calibration to reduce the impact of pattern noise - however, as my camera is not cooled and the night's temperatures are constantly changing, any master dark will not closely align to the actual thermal pattern noise and as such dark subtraction may help but will not solve the problem

- Using in-camera dark subtraction ( Nikon's long exposure noise reduction feature ) would almost completely remove the pattern noise from each frame.  However, due to the extra random noise being introduced by subtracting another noisy dark frame from each light frame, as well as the reduction in total light frames by 2, the resultant images will suffer from higher levels of random noise.  So whilst this would be an improvement with respect to the pattern noise, it is not a complete solution.

- Third option, sell the D7500 and go back to using the D5300 ... :undecided:



17 Comments


Recommended Comments

I read somewhere that the D7500 uses a two stage amplifier from around 400iso up.  In order to see if this was playing a part I repeated the test at 250iso.  The result was essentially the same - at around 20deg C, the D5300 has much less thermal pattern noise than the D7500. 

 

Share this comment


Link to comment

A very interesting read!  There are very few people doing proper testing of thermal noise in DSLR cameras but it is the one thing that can make or break an image, especially on the warm nights that we rarely have in the UK.

A couple of comments on the histograms.  Both of them seem to be clipped at the low end i.e. the left hand tail of the histogram is missing.  Why is this - I thought Nikon had fixed the black clipping issue?   The multi-peak nature of D7500 histogram is very strange indeed.  It appears to indicate different populations of pixels behaving differently.  I don't recall ever seeing this before.

If you still have both cameras you could potentially perform a side by side test to measure how fast the noise rises from frame to frame.  This might provide a lot of useful information.  It's the approach I took on the testing in the graph below:

DarkCurrentGraphs4.jpg.72a3e99dc0986362106f64dd585645db.jpg

Successive 5 minute exposures were taken from a camera switched on after acclimatising to the 20C ambient temperature.  The noise in a difference frame (2 successive darks frames subtracted) was measured and converted to a dark current estimate for each point in the graph.

The reason I subtract the dark frames is so the graph represents the thermal noise levels expected from calibrated frames.

If you have a good set of darks taken under controlled conditions I'd love to see how they would compare on that graph.  Generally speaking it's the Canons that become very noisy.

[Later Edit]  I've realised that the black clipping in the histogram is because you've bias subtracted the master dark.  The shape of the D7500 histogram is very puzzling though.   Unfortunately both your histograms go off the top of the scale, so it's difficult to make a meaningful comparison.

Mark

 

 

 

Edited by sharkmelley

Share this comment


Link to comment
13 hours ago, sharkmelley said:

 I've realised that the black clipping in the histogram is because you've bias subtracted the master dark.  The shape of the D7500 histogram is very puzzling though.   Unfortunately both your histograms go off the top of the scale, so it's difficult to make a meaningful comparison.

Thank you for your reply Mark.

I was focused on trying to determine the extent of the differences in the pattern noise and this is why I was looking at the relatively brighter portions of the master darks.  I have been less concerned with the true random noise in each sub as I can deal with this by taking more subs but the pattern in that noise is far more difficult to address.  I will dig out and post a different view of the histogram with a lower zoom of the Y axis.

 

Share this comment


Link to comment

Ok some more histograms ...  unfortunately I think I might have deleted some of the files so I these are at different ISO values ( 250 for the D5300, 400 for the D7500 )

These have not been bias corrected.

D5300: 412B6F1C-AFD2-4BB3-8E05-56CC2E807E87.jpeg.4e82fe5f2598c6a4c6e05d1e3e11fc77.jpeg

D7500: 0B57C0A2-B4E3-4801-8993-BCBE557CE49D.jpeg.b4b0971f56086a52655e2acfd722e774.jpeg

D5300: E424EB54-5E9E-40D2-8F81-C9FC195AC11D.jpeg.622deb1a5934cc50e77407d88008df7b.jpeg

D7500: 5CBD1D12-308E-4B0D-B5DA-8AB979BE4DBC.jpeg.073a57525612c908a825902295f75dd2.jpeg

 

Note that a significant portion of the left hand part of the histogram for the D5300 is due to bias and is removed by subtracting the master bias ( refer histogram in original post ).  This is not so much the case with the D7500.  

The D7500 histogram clearly shows the very large number of ‘warm pixels’ that, whilst randomly distributed across the sensor, are in fixed positions in each frame and result in significant ‘pattern noise’.

Share this comment


Link to comment

From your histograms it certainly does look like the D7500 sensor has a greater number of outlying values.  On the other hand those histograms also support your earlier blog where you stated that the dark noise in the D7500 is lower.  Your bias subtracted darks are an anomaly because of the large number of clipped values.

I think the overall message is that the D7500 is an improvement over the D5300 as long as you use well matched darks.  But that tentative conclusion needs to be tested further.

Mark

Share this comment


Link to comment
7 hours ago, sharkmelley said:

From your histograms it certainly does look like the D7500 sensor has a greater number of outlying values.  On the other hand those histograms also support your earlier blog where you stated that the dark noise in the D7500 is lower.  Your bias subtracted darks are an anomaly because of the large number of clipped values.

I think the overall message is that the D7500 is an improvement over the D5300 as long as you use well matched darks.  But that tentative conclusion needs to be tested further.

Mark

Hey Mark

The problem I have is that with an un-cooled camera, quickly changing night time temperatures and no on-sensor temperature sensor it is practically impossible for me to accurately match darks.  So, ideally, I want a camera with very low thermal pattern noise ( as opposed  to true random noise that I can reduce somewhat with more subs ).  

My thinking was that the calibrated master darks show that both cameras have thermal pattern noise but I thought they indicated that the level of that signal in the D5300 is much less (?).  So I do not really understand your comment that “ Your bias subtracted darks are an anomaly because of the large number of clipped values”.  I thought that the histogram of the bias calibrated master darks shows the uncorrected pattern noise that will be present in the bias corrected lights.  

That is, in my current workflow, I have not found it necessary to use a master dark with the D5300.  So I thought that the bias corrected darks indicate how much worse off I will be if use a D7500 without using a master dark.

I guess another way of looking at is this...

The D7500 has an almost constant value across the sensor in the master bias frame ( with only a couple of ADU variation along the bottom edge ).  So, subtracting the master bias ( other than shifting the peak ) has very little impact on the master dark.  However, the D5300 master bias has significant variation of values ( ie.  a significant pattern in the bias frame ) and subtracting this from the master dark has significant impact.  That is, the histogram of the D5300 master dark is made up of a significant ( mostly temperature independent ) bias component + the temperature dependent thermal pattern noise.  Subtracting the master bias leaves a relatively small thermal pattern component.  Whereas, the histogram of the D7500 master dark is essentially all thermal pattern noise.  That is why I thought the calibrated master darks were a good indication of the relative levels of thermal pattern noise in the two sensors.

But I guess I am only focused in my analysis on the fixed position signals showing up in the bias and dark frames ( the former because I can simply remove them and the latter because I can’t ).  What do you think, is my approach sound or is there a better way of trying to determine which sensor will have the greater problem with patterns in the thermal noise signals?

Thanks

Mike

 

 

Edited by MikeODay

Share this comment


Link to comment

 

7 hours ago, sharkmelley said:

Your bias subtracted darks are an anomaly because of the large number of clipped values.

Mark, 

Perhaps it would be more appropriate to add an offset to the master darks before I subtract the bias frames so as to prevent clipping - would that be a better representation of the non-random signal in the master darks that is due to fixed pattern variations in the thermal response of the pixels across the sensor ( what I have been calling “thermal pattern noise”)?   What do you think?

Cheers

Mike

Share this comment


Link to comment

I have taken the master darks, added an offset, subtracted the master bias files,  scaled the gains so that the D7500 @ 400ISO is roughly equiavlent to the D5300 at 250ISO, and shifted them so that the medians line up with the following results.

Master Darks ( 100% crops of the centre, identical screen transfer function applied via histogram tool )

D5300:  7F66B823-7B1A-4C48-BBF4-F2109E1104D5.jpeg.56bc7a14e1eeeecadcf5e096405cb6a3.jpeg

D7500:  917167D8-69A6-4735-A7DD-C279BA6DC492.jpeg.7dcbd137b8c0ae59a9217d8ae2a04df6.jpeg

 

Histograms @ 1x vertical scale:

D5300:  75E882C6-9618-4F7F-9FA4-AC73D9816D38.jpeg.1be70bc09c67637cb22ee0ef424569ae.jpeg

D7500:  0A655BB4-449E-47B5-9E7E-E6905FB64FB6.jpeg.1627ca38e871476c45e33e035fd6b1ba.jpeg

 

Histograms at 23x vertical scale:

D5300:  2F6913DA-5111-43D5-A8A4-6A8F782B576F.jpeg.19fb4c06012e0a5d845b5af8c77861c5.jpeg

D7500:  8D816FC0-8180-4D86-9663-44254DC68B2F.jpeg.1a27ad1003a5eaf3e807b695a3f776d4.jpeg

I have positioned the the tab marks at the bottom of the histograms to encompass 99% of pixels in the relevant master dark.

That is, 99% of pixels are in the range:

D5300 :  -8 to + 16

D7500:  -4 to + 97

So, I guess I might say that ( in the range ~ -8 to 8 about the median) the D5300 has more cool and slightly warm pixels than the D7500 but the D500 has way more very warm and hot pixels.  For example, in the inital examination ( from the initial post above ) the D5300 had ‘only’ 6000 pixels above 44 whereas the D7500 had 140,000 above 44.

( note that whilst subtracting the bias from the offset D5300 master dark did make a difference, the difference was not as large as I anticipated in a previous post ).

Edited by MikeODay

Share this comment


Link to comment

There is something I find odd about the results from the D7500.  Nikon cameras use a spatial filtering algorithm in longer exposures in order to remove hot/warm pixels.  It's not a severe algorithm like the "star eater" spatial algorithm used on Sony cameras but it should still remove isolated hot/warm pixels.  So I'm surprised to see such a long right-hand-side histogram tail. 

I would certainly be interested in taking a look at a representative raw dark frame from both the D5300 and D7500.

Mark

Share this comment


Link to comment

I have a D5300 I'd you want a dark ?

Tell me what ISO and exposure you want and 'rough' temperature. 

  • Like 1

Share this comment


Link to comment
8 hours ago, sharkmelley said:

There is something I find odd about the results from the D7500.  Nikon cameras use a spatial filtering algorithm in longer exposures in order to remove hot/warm pixels.  It's not a severe algorithm like the "star eater" spatial algorithm used on Sony cameras but it should still remove isolated hot/warm pixels.  So I'm surprised to see such a long right-hand-side histogram tail. 

I would certainly be interested in taking a look at a representative raw dark frame from both the D5300 and D7500.

Mark

Hi Mark

please see below links to single dark frames:

5300, 240sec, ISO250 : https://www.dropbox.com/s/fu27b1tx5wbu00c/D5300_dark_240s_250iso__030.nef?dl=0

7500, 240sec, ISO400: https://www.dropbox.com/s/cymscuakzpcu9mp/D7500_240s_400iso__0030.nef?dl=0

Cheers

Mike

 

Share this comment


Link to comment
15 hours ago, MikeODay said:

Hi Mark

please see below links to single dark frames:

5300, 240sec, ISO250 : https://www.dropbox.com/s/fu27b1tx5wbu00c/D5300_dark_240s_250iso__030.nef?dl=0

7500, 240sec, ISO400: https://www.dropbox.com/s/cymscuakzpcu9mp/D7500_240s_400iso__0030.nef?dl=0

Cheers

Mike

 

Thanks for those.  There's nothing obvious that strikes me about them immediately, by means of explanation.  But the D7500 certainly seems to have more hot/warm pixels.  I will take a closer look later.

Mark

Share this comment


Link to comment

Thanks for reviving this thread.  I've been playing with a Nikon D5300 recently and noticed that the spatial filtering (a.k.a. hot pixel suppression or HPS)was causing all kinds of problems with star colours - mainly turning them green, just like the latest variant of the Sony star eater algorithm.

It's all being discussed over on Cloudy Nights:  https://www.cloudynights.com/topic/635441-aa-filter-spatial-filter-and-star-colours/

It's early days and it's not yet fully understood.  However, it appears that the Nikon D810A (the one built for astrophotography) doesn't have the same issue because the spatial filtering is much less aggressive.

So it's just possible that the D7500 is also using less aggressive spatial filtering than the D5300 and that's why it appears noisier.  I'll take a new look at Mike's D7500 darks and see what I can determine.  It'll definitely provide another interesting data point for my analysis.

Mark

 

Edited by sharkmelley
  • Like 1

Share this comment


Link to comment

I'm looking at the D7500 as an all use camera or maybe a D5600 + a Hypercam 183, really not sure way to go yet

Share this comment


Link to comment

Having looked at this in more detail, it's certainly true that the D7500 is using less aggressive spatial filtering than the D5300.  However this does not explain the observed difference in thermal fixed pattern noise (FPN).   The D7500 really does appear to have a higher FPN than the D5300.

The practical consequence of this is that you are more likely to need dark frame calibration with the D7500 than with the D5300.

Mark

Share this comment


Link to comment
Guest
Add a comment...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
×
  • Create New...

Important Information

By using this site, you agree to our Terms of Use.