Noise and Temperature

[Stub Mandrel]

Getting frustrated with the amount of noise in one of my pics from the other night I checked things and found my darks were 60s long, but my lights were 120s.

So as an experiment I put the camera in the fridge for an hour, then set it to take 12 80-second darks.

The first dark was 5962 kb, the next one a bit larger and so on to 12 which was 6375 kb.

I assume that this reflected the camera slowly warming up and the noise level increasing.

What this will mean to my image quality remains to be seen.

[geordie85]

Why would you take darks at different exposure lengths to your lights?

They have to be the same duration

[Stub Mandrel]

Typo 1min 20 secs = 80 secs, not 120 secs!

[geordie85]

Ah now that makes much more sense. Have you tried using them yet? I'm curious to hear the results

[Stub Mandrel]

Not a big difference, they may have been too cold.

In fact this is the (stretched) central square of the 63-second master dark taken indoors after the session:

This is the (stretched) 83 second in the fridge master dark:

And this is the difference between the two stretched images:

It takes another a stretch, the difference is barely visible:

The histogram for the last image suggests that virtually all of the difference between the two master darks is in the bottom two or three bits of the data.

[andrewluck]

The histogram for the last image suggests that virtually all of the difference between the two master darks is in the bottom two or three bits of the data

That's what I'd expect. However, you'd also see this difference on identical length subs at the same temperature as this noise is random in nature. The cure for this is more exposures and the noise will average down.

For a thermally unregulated camera I don't recommend darks at all. Use dithering and a cosmetic correction to remove hot pixels and rely on averaging a lot more lights to reduce the noise.

Andrew

[D4N]

A bad pixel map and dithering are more help for a DSLR as the noise varies too much. Dark libraries are a bit of a nightmare if you can't control the chip temperature.

I did think you had nice pinpoint stars but lacking colour until I realised I was looking at your dark frame [emoji12]

/Dan

Sent from my iPad using Tapatalk

[Peje]

This is really interesting to me...

Just starting out with AP and really trying to figure out what method is best. I was hoping to build up a library of darks on the wet evenings but it seems Nikon didn't care to put temp values in the exif data so that ideas out the window. I'd love to avoid having to burn 50% of my time on darks each clear night.

Can you point me in the direction of some guide info about the bad pixel mapping? Pretty sure I've seen info about dithering and I'm almost certain this is supported in BYN through PDH2

[sharkmelley]

It is certainly true that dark current and the associated thermal noise will increase from exposure to exposure - the noise can be measured.  Here's a graph I've often posted showing how dark current increases because of sensor warming in a sequence of 5 minute exposures at a constant ambient temperature:

So one master dark can never hope to match every exposure in an imaging sequence.  You can try building a dark library but can you really be confident of the sensor temperature of the darks used to build the library?  Also, are you confident that the temperature seen in the EXIF is actually the sensor temperature?

It's best to use dark scaling (aka dark optimisation).  One master dark can be used and the software scales it to match the different thermal noise in each individual light frame.  PixInsight does this very well.  DeepSkyStacker also does it but I've never tried it out.

Dithering (with sigma rejection stacking) is also pretty effective at getting rid of those hot and warm pixels caused by dark current.

Me, I use both dithering and dark optimisation, together.  

Mark

[Peje]

Sounds like something I'd like to try. Do you have any info on how to do this? Like a method or process?

[Stub Mandrel]

I've noticed that if you set the camera to take a sequence of darks, the file size increases from frame to frame. I imagine this is increasing noise reducing the smoothness of the image and therefore increasing the file size with lossless compression.:

This is a camera that has been in a fridge set on low for quite a while, taking successive sessions of 15 darks with 15 seconds between each dark and about ten minutes cool down between sessions. Size is in kB. These are 1'30" darks.

Interestingly, the graph is smoother than anything I would expect from a lab experiment! It show just the first signs of the slope dropping away, I imagine the maximum file size could be around 6400-650kB.

What would be useful would be seeing how long a cool down period is needed to keep the file size constant - waiting longer between subs might be a fair price to pay for keeping the noise down.

It also suggests that, when making darks, you need the starting temperature to be right and that the gap between exposures should match the interval between subs.

More radically, it suggests that the ideal strategy would be to take your darks in between your lights - or say you are taking a sequence of ~60 subs make every fifth sub a dark. This would mean putting a hat or similar over the end of the scope/lens and demand a lot more attention than just letting the camera run unattended.

[D4N]

It highlights to me the futility of darks without optimisation on a DSLR.

Even if you take them between exposure they are only really relevant to the subs either side of the dark. Since using a single dark frame can add as much noise as it removes this is also futile. You would have to have a stackable number of darks between the lights but once again this will increase the temperature more.

Optimisation will help but the software is limited in how much of a difference it can account for so the quality of the results may vary.

A bad pixel map is basically a master dark with the low level noise ignored.

The bad pixels will be repaired using adjacent pixels rather than subtracted.

The random noise is then dealt with during stacking rather than pre-processing.

/Dan

Sent from my iPad using Tapatalk

[Peje]

It highlights to me the futility of darks without optimisation on a DSLR.

Even if you take them between exposure they are only really relevant to the subs either side of the dark. Since using a single dark frame can add as much noise as it removes this is also futile. You would have to have a stackable number of darks between the lights but once again this will increase the temperature more.

Optimisation will help but the software is limited in how much of a difference it can account for so the quality of the results may vary.

A bad pixel map is basically a master dark with the low level noise ignored.

The bad pixels will be repaired using adjacent pixels rather than subtracted.

The random noise is then dealt with during stacking rather than pre-processing.

/Dan

Sent from my iPad using Tapatalk

I'm getting a bit lost! Is there a guide on how to create the pixel map? And also could you list a (high level should be fine to start) the process you use when taking the images? Or even point me at a topic where this is discussed?

I appreciate I'm asking loads of questions but it's because I'm genuinely fascinated in what you are talking about.

Pete

[Peje]

I guess one thing I have going for me is I have a thermal chamber in work so I can set accurate ambient (to within .1 oC) and build up a library...though I'm not 100% sure that is even what you are suggesting.

[Stub Mandrel]

It highlights to me the futility of darks without optimisation on a DSLR.

Even if you take them between exposure they are only really relevant to the subs either side of the dark. Since using a single dark frame can add as much noise as it removes this is also futile. You would have to have a stackable number of darks between the lights but once again this will increase the temperature more.

Optimisation will help but the software is limited in how much of a difference it can account for so the quality of the results may vary.

A bad pixel map is basically a master dark with the low level noise ignored.

The bad pixels will be repaired using adjacent pixels rather than subtracted.

The random noise is then dealt with during stacking rather than pre-processing.

/Dan

Sent from my iPad using Tapatalk

But isn't the whole point of stacking darks that you reduce the noise so that the repeatable bad pixels are what's left?

[pete_l]

What would be useful would be seeing how long a cool down period is needed to keep the file size constant - waiting longer between subs might be a fair price to pay for keeping the noise down.

I don't think that would happen. As you pointed out in the graph, the dropping off of the slope is the key. When the file size is more or less constant then the innards of the camera are in thermodynamic equilibrium with the external temperature.  That seems to take quite a long time. However, the flip-side is that when you're out under the sky, your darks would only be "textbook" valid when the camera has equalised with the ambient temperature of your observing location - which (again) could take quite some time. You might even find that the "warm-up" time of your camera is greater than the "cool-down" time of your optics

I think the only use for dark frames is for hot / warm pixel removal. The background noise is random, so the best you can hope for is to define a "noise floor", below which all signal is deemed noise, and remove all data below that point - but you don't need to take dark frames for each session, just to have a table of CCD sensor temperatures vs. "floor" pixel values and then, if all your subs are taken at the same CCD sensor temperature, set the black level at that value.

[D4N]

It really depends on what software you use.

In Nebulosity it is Batch > Make bad pixel map

Then you just use it instead of a dark frame when you calibrate (you also tell it that the dark is a bad pixel map with the drop down).

In PI you use the cosmetic correction process.

I don't know about other software.

/Dan

Sent from my iPad using Tapatalk

[Peje]

It really depends on what software you use.

In Nebulosity it is Batch > Make bad pixel map

Then you just use it instead of a dark frame when you calibrate (you also tell it that the dark is a bad pixel map with the drop down).

In PI you use the cosmetic correction process.

I don't know about other software.

/Dan

Sent from my iPad using Tapatalk

Looks like I'm going to have to buy PI...my wife will be pleased LOL

So when you have this map is it just a single image that you put in place of multiple darks?

Does that then mean you just take lights on the night? How about the bias / flats, are these just a library effort?

[D4N]

Bias last a long time, takes loads stack them into a master then ditch the files.

Flats don't last as long as any dust that moves will reduce their effectiveness. Also moving the camera will invalidate them.

Bad pixel maps stay good until more pixels die on your sensor. But they will not cause any harm to your image like bad flats or darks would, they will just not correct new bad pixels.

I generally delete the calibration files once I have created a master.

A bad pixel map in Nebulosity is used in place of the darks.

In PI the cosmetic correction is more like a list of bad pixels.

Edit: you won't regret buying PI, it is fantastic software.

/Dan

Sent from my iPad using Tapatalk

[Peje]

Thanks Dan, I've been debating buying it so I guess this is just reinforcing that side of the argument. The thing I like here is that I can do alot of this while the collimation on my scope isn't sorted so effectively I can get started anytime

So to create my bad pixel map, I just need a lot of darks. Is this temperature / exposure time related? I expect not for the latter, possibly not the former. If so, how many images and what sort of conditions / settings would you recommend?

EDIT: - Also, in PI do you just not insert any darks & just use the Cosmetic Correction instead?

[Peje]

Bias last a long time, takes loads stack them into a master then ditch the files.

Flats don't last as long as any dust that moves will reduce their effectiveness. Also moving the camera will invalidate them.

Bad pixel maps stay good until more pixels die on your sensor. But they will not cause any harm to your image like bad flats or darks would, they will just not correct new bad pixels.

I generally delete the calibration files once I have created a master.

A bad pixel map in Nebulosity is used in place of the darks.

In PI the cosmetic correction is more like a list of bad pixels.

Edit: you won't regret buying PI, it is fantastic software.

/Dan

Sent from my iPad using Tapatalk

Got a notification you replied but don't see a post?

[D4N]

When I use a bad pixel map I do not use darks, any remaining noise should be sorted out during stacking.

There are bad pixels that are either always on or always off, however there are also bad pixels that only turn on after a while.

I just use master darks from the exposure length I am using to create a bad pixel map.

I guess you could just use the longest exposure to do it.

/Dan

Sent from my iPad using Tapatalk

[D4N]

Got a notification you replied but don't see a post?

Don't know why, maybe a delayed or double notification from the last reply?

Or maybe it is just getting ahead of itself

/Dan

Sent from my iPad using Tapatalk

[Peje]

When I use a bad pixel map I do not use darks, any remaining noise should be sorted out during stacking.

There are bad pixels that are either always on or always off, however there are also bad pixels that only turn on after a while.

I just use master darks from the exposure length I am using to create a bad pixel map.

I guess you could just use the longest exposure to do it.

/Dan

Sent from my iPad using Tapatalk

So, how many of each do you think is reasonable? I'm assuming high quantity for the shorter exposures? For completeness I'd probably do 1-10min in increments of 1.

[D4N]

I normally take 20 to 30.

It is easy to take loads for 1 minute exposures, not so much fun for 30 minute exposures though.

I have master darks for 5, 10, 15, 20 & 30 minutes.

/Dan

Sent from my iPad using Tapatalk