Jump to content

sgl_imaging_challenge_2021_4.gif.6a323659519d12fc7cafc409440c9dbf.gif

Are Nikon DSLR darks a waste of time?


Recommended Posts

You may or may not know that over on Cloudy Nights there is a continuing long thread about how Nikon spatial filtering (a.k.a. hot pixel suppression, HPS) affects the colour of small tightly focused stars:  https://www.cloudynights.com/topic/635441-aa-filter-spatial-filter-and-star-colours/  HPS is typically applied to all exposures longer than 0.25sec and it cannot be switched off.

There is another thread discussing the effects of HPS on dark subtraction:  https://www.cloudynights.com/topic/647035-dark-subtraction-and-spatial-filtering/

I'll repeat here the results of a dark subtraction experiment I performed using my Nikon D5300. 

I took 30 light frames of 2min at ISO 400 in a fairly dark place with a white diffuser over the lens so the back-of-camera histogram was in the usual recommended position for astro-imaging i.e. 1/3 from the left.  All noise reduction was switched off, as usual.  This was followed by 30 darks of 2min, again with noise reduction off.  As a control I took another set with long exposure noise reduction (LENR) switched on - this is where dark subtraction is performed in-camera. 

From those exposures I created a master light and master dark then subtracted the master dark from the master light.  I also created a master light with the LENR lights.  The results are here:

NikonD5300_DarkSubtraction.thumb.jpg.496da1075499df65fdbdb4870b40b98b.jpg

I've shown only the raw red channel but I've applied identical linear brightness scaling to each panel, to allow direct comparison.

Some obvious features can be seen:

  • The master light has plenty of isolated bright pixels i.e. the thermal fixed pattern noise (FPN)
  • In the master dark almost all of the thermal FPN has been removed by the HPS filtering, except for pairs of pixels that "protect" each other
  • Subtracting the master dark from the master light removes those bright pixel pairs but leaves most of the FPN in the image
  • By contrast, the in-camera LENR does an excellent job of removing the thermal FPN

So it seems that dark subtraction for typical light frames on the D5300 is pretty ineffective.  The D5500 will behave in the same way.  However, the algorithm used for Nikon HPS varies greatly from model to model so it is not yet possible to predict with confidence which models behave similarly.  But if you use a Nikon for astrophotography you might want to perform your own test, like I did.

The best workaround is what you are probably already doing: use dithering during image acquisition and use sigma rejection during stacking.

Mark

Edited by sharkmelley
Link to post
Share on other sites
2 hours ago, happy-kat said:

That's an interesting test. I might try that next time I do any imaging to turn back on noise control on my canon just for the darks.

Do the experiment by all means but Mark is referring to certain Nikon cameras and not Canon cameras. 

It wouldn't affect me as my single pixel stars will occupy nine at best and dozens at worst ?

Dave.

Link to post
Share on other sites
20 minutes ago, davew said:

Do the experiment by all means but Mark is referring to certain Nikon cameras and not Canon cameras. 

It wouldn't affect me as my single pixel stars will occupy nine at best and dozens at worst ?

What difference does the size of the stars make? It has no bearing on whether or not dark subtraction removes the thermal fixed pattern noise.

Mark

Link to post
Share on other sites
24 minutes ago, sharkmelley said:

What difference does the size of the stars make? It has no bearing on whether or not dark subtraction removes the thermal fixed pattern noise.

Mark

I was referring to your opening paragraph as a light hearted  comment.  I should make my intent more obvious.

Dave.

  • Like 1
Link to post
Share on other sites

It's hard to say from the images, as depending on the software used to create the screenshots the screen transfer function could be different (just seen that the stretch was the same, but the subtraction frame looks brighter than the master light). Are you able to grab a box of a few thousand pixels (omitting hot pixels) and look at the standard deviation? That might tell us more.

Billy.

Link to post
Share on other sites
2 hours ago, billyharris72 said:

It's hard to say from the images, as depending on the software used to create the screenshots the screen transfer function could be different (just seen that the stretch was the same, but the subtraction frame looks brighter than the master light). Are you able to grab a box of a few thousand pixels (omitting hot pixels) and look at the standard deviation? That might tell us more.

Billy.

How do you expect the standard deviation to help?  If you are unable to see visually that the thermal fixed pattern noise has not been removed then what do you expect standard deviation to tell you?

By the way, no STF was applied.  This is linear data scaled by the same multiplier in all 4 panels to brighten it sufficiently for screen display.

Mark

Edited by sharkmelley
Link to post
Share on other sites

Hi Mark.

The main use I was thinking of for standard deviation here was to quantify the noise in each image and the increase / reduction after dark subtraction (across a sample with no gradient). I'm interested in why the darks don't seem to be doing the job, and suspect that some of that noise isn't fixed pattern - standard deviation should help with that.

I think that missed the point though. More fundamentally, there seems to be more hot pixels in the light frame than the dark. That would explain why they don't subtract out, but I'm struggling to understand why it's like that. Also, am I right in saying the in-camera noise reduction subtracts a flat? That does seems to work, so maybe the pattern changes over time?

Lots of scope for exploration here. But it does look like darks might not be the answer. 

Billy.

Edited by billyharris72
Update
Link to post
Share on other sites

Interesting. I have 2 Nikons a D5100 which has been modified removing all filters back to bare chip. And an 880E which is 'normal'.

As this is a test of thermal noise generation/removal how did you eliminate sensor temperature changes during capture of the various data sets? If the sets were captured one after the other or on different nights/sessions, then there could be differences which would not show when you set camera noise reduction to operate on individual frames where sensor temperature variations would not be a factor?

Link to post
Share on other sites
14 hours ago, billyharris72 said:

Hi Mark.

The main use I was thinking of for standard deviation here was to quantify the noise in each image and the increase / reduction after dark subtraction (across a sample with no gradient). I'm interested in why the darks don't seem to be doing the job, and suspect that some of that noise isn't fixed pattern - standard deviation should help with that.

I think that missed the point though. More fundamentally, there seems to be more hot pixels in the light frame than the dark. That would explain why they don't subtract out, but I'm struggling to understand why it's like that. Also, am I right in saying the in-camera noise reduction subtracts a flat? That does seems to work, so maybe the pattern changes over time?

The reason is subtle.  The point of the spatial filter is to remove hot pixels.  On the D5300 the spatial filter works by capping pixels whose value is more than twice that of their (same colour neighbours).  In a dark frame the pixel values average out to zero so almost every bright pixel has its value capped unless it has a bright neighbour.  In a light frame, the bright pixels are "protected" by the non-zero sky-glow level.  That's why the bright pixels are obvious in the light frame but have been removed from the dark frame. 

Not all Nikon spatial filter algorithms work the same way.  So its a fruitful area for further investigation.

By the way, in-camera noise reduction works by subtracting a dark, not a flat.

Mark

Link to post
Share on other sites
6 hours ago, fwm891 said:

Interesting. I have 2 Nikons a D5100 which has been modified removing all filters back to bare chip. And an 880E which is 'normal'.

As this is a test of thermal noise generation/removal how did you eliminate sensor temperature changes during capture of the various data sets? If the sets were captured one after the other or on different nights/sessions, then there could be differences which would not show when you set camera noise reduction to operate on individual frames where sensor temperature variations would not be a factor?

I took long exposures for over an hour to let the camera thermally stabilise beforehand and only used the exposures after that point.  It was all done in a single imaging run. 

I don't have details about the D5100 and 880E spatial filtering so I can't tell if they will behave the same way as the D5300.

Mark

Edited by sharkmelley
  • Thanks 1
Link to post
Share on other sites

Now the penny drops - I hadn't quite clicked to the point about how the spatial filter is messing up the dark frame. I was wondering why you hadn't just done 2 sets of darks and subtracted one from the other, but that makes perfect sense.

Is the Nikon noise reduction subtracting a "proper" dark frame (I assume that's why it does a better job)? If so, what about trying to get a master dark based on those - would subtracting your noise reduced stack from the original stack result in something usable? If nothing else it could maybe be clipped and used as a defect map.

[edit - not so simple is it? More likely the dark would have modifications based on the light frame HPS, which complicates matters, though the approach might be salvageable, for a defect map at least].

I'll have a play with this myself - I'm using a d3300, which I suspect uses the same approach, but will be interesting to find out.

Billy.

Edited by billyharris72
More complicated than that....
Link to post
Share on other sites
14 hours ago, billyharris72 said:

Now the penny drops - I hadn't quite clicked to the point about how the spatial filter is messing up the dark frame. I was wondering why you hadn't just done 2 sets of darks and subtracted one from the other, but that makes perfect sense.

Is the Nikon noise reduction subtracting a "proper" dark frame (I assume that's why it does a better job)? If so, what about trying to get a master dark based on those - would subtracting your noise reduced stack from the original stack result in something usable? If nothing else it could maybe be clipped and used as a defect map.

[edit - not so simple is it? More likely the dark would have modifications based on the light frame HPS, which complicates matters, though the approach might be salvageable, for a defect map at least].

I'll have a play with this myself - I'm using a d3300, which I suspect uses the same approach, but will be interesting to find out.

Billy.

That's right, you've got it now!

The Nikon long exposure noise reduction (LENR) will be subtracting an unfiltered dark from an unfiltered light, which is why it works fine.

Mark

 

Link to post
Share on other sites
On 20/01/2019 at 13:04, happy-kat said:

I know it is about a Nikon camera, but I am curious anyway.

Me too but to let you into a little secret I have always used the Canon LENR rather than dark's on the basis they should be a perfect match for the preceding light frame, I will say that I only do widefield these days and anything from a single frame to a max of 10 so no real issues with loss of imaging time.

Alan

  • Like 1
Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
  • Recently Browsing   0 members

    No registered users viewing this page.

×
×
  • Create New...

Important Information

We have placed cookies on your device to help make this website better. You can adjust your cookie settings, otherwise we'll assume you're okay to continue. By using this site, you agree to our Terms of Use.