Darks, continued

[MishMich]

Recently there was a discussion about darks, etc. Then somebody said they don't bother with them, let the stacking deal with noise & hot pixels. Then on another forum somebody suggested somebody doing some single frame work just try using the in-camera dark frame subtraction.

So, rather than continue on the thread about darks & flats, I'm going to ask a simple question.

Why bother with darks at all, if the DSLR can do this itself?

On the KISS principle alone, this makes some sense. I'm assuming dark frame subtraction in stacking comes from dedicated CCD's, where there is no on-camera subtraction. But DSLR's have this, so why not use it? Is it going to use up noticeably more time than taking a load of darks, bias frames and flat darks? It would entail just taking lights and flats, and the issue about matching darks through a session in a way that the rise in sensor temperature would be sidestepped, because the camera would match the frames themselves. I started out doing images without darks, leaving the camera to take care of it, and while it took a bit longer between taking the shot and saving the image, it was not as long as the image itself. So, I'm thinking that overall, it may well not take longer.

That hints at what might be the only flaw in this idea - that the camera's method at subtracting darks on a 1-to-1 basis might be inferior, as it is not done as exposure-for-exposure. But the only argument I have heard for doing separate dark frame subtraction is that it takes up more valuable imaging time.

Is it true it takes up more time?

Is there some other reason for processing with darks separately?

M.

[narrowbandpaul]

first of all the onboard dark subtraction subtracts one dark per frame....because of the random noise contained in the dark, the SNR will go down after applying the single dark. remember random noise adds in quadrature.

what are we doing when we apply a dark? well we ARE NOT REMOVING THE DARK CURRENT nor the DARK SHOT NOISE.....the dark current is a single value, and the shot noise is random therefore cannot be removed.

no, what we are removing is DARK FIXED PATTERN NOISE, which are evident as pixels that are always slightly warmer or colder than the average. The pixels are fixed spatially, therefore the noise source here is Fixed

what we are doing is Despiking the image, to use a term used by James janesick, father of the use of CCD's in astronomy in the 1970's.

ok, so what we do is take a number of darks, which should match the temp and duration of the light frames.

we average them....this reduces the read noise by a factor sqrtN and the shot noise by sqrtN, but the fixed pattern noise is not reduced......

therefore the DFPN becomes more and more obvious (think higher SNR) as we stack more darks.....we want to approach the case where the random noise is virtually 0....

then we take this 'spike only' image composed of N darks averaged together, and subtract this from the light frames....

the light frames are now despiked.

so for eitheer CCD or CMOS take a stack of darks at the end or start of the imaging session, the darks should then work better.....

hope that helps

paul

[MishMich]

OK. So, practically then, if I image for 3 minutes, I get amp glow in the lower right hand corner of the frame. If I use on-camera dark subtraction, I do not get rid of the amp glow, nor fixed pattern noise (like streaking). If I subtract dark frames arrived at through stacking, the amp glow gets to be eliminated completely, along with other fixed-pattern noise. Is that right? This makes more sense of what I have experienced in my own images - those done with on-camera darks were harder to process, because of the amp glow and horzontal lines that appeared when I tried to stretch the data. When I took separate darks, this was eliminated.

Thanks for explaining it - now I can relate it to my experience, and am assured that it is worth taking the time to take separate darks.

M.

[MishMich]

P.S., I hope you don't mind me asking these questions. I find the answers very helpful in trying to understand what I am supposed to be doing. I am not a scientist (even though I have a Master of Science degree), so have to find other ways of understanding concepts that may well be basic to other people.

M.

[narrowbandpaul]

i thought when I read some of your posts that you had a science background, thats why i wrote that post a bit more technically.

i would advocate taking a bunch of darks at the end, rather than 1 per image

the DFPN can often be the major source of noise in the dark frame....ie the read and shot noise components can often be much smaller than the DFPN......

anything you dont understand just shout

[MishMich]

I had a terrible time with maths at school, and my experience of science was pretty much being bullied into learning. It became a block for me, and although I exelled in Logic at 'A' level (I was the first person in the UK to get an 'A' in Logic & Scientific Method at A level), I could be reduced to tears by the maths in Economics. I even managed to do the Philiosophy of Maths at uni, on the basis of my understanding of predicate calculus, yet maths itself was still inaccessible to me. I was always interested in science, but school really destroyed it for me. My breakthrough with maths came when I did post grad business studies, because by then there were computers to help me, and I was very good at using spreadsheets. I eventually did my Masters in Information Technology, which was essentially software engineering and database management. There was some maths in that - like writing programs to do FFT for image enhancement and working out the date for Easter for the next 400 years (I based mine on the table in the Book of Common Prayer, which made it more accurate than when done based on the moon). The rest of my maths has been Quantitative Analysis. But no science, per se. I'm happy to learn, but because I have no background in physics or electrical engineering or optics, it does not come easily to me - and this is an area that has traditionally been a huge barrier to me.

M.

[narrowbandpaul]

the teachers at dunblane were the opposite.....very good, very encouraging. It was fun to learn physics and chemistry at school.

again at uni the lecturers were excellent, and I really enjoy physics and astronomy (my degree is Physics with astronomy).

now I am a phd student in electrical engineering in Sheffield. I am characterizing CMOS image sensors.

so I do a lot of stuff with sensors, and CCD's are really cool if you ask me. I like testing the cameras from different manufacturers....I like it, they dont, I assure you. It allows you to find their flaws.

i believe you can improve your imaging by understanding whatt is going on at the sensor level.....

[Merlin66]

Mish,

Your doing well so far!

Re Darks....

In AstroArt V4 they call the Dark Frame = Bias Frame+Thermal Frame as it is very temperature and time dependent -

They suggest taking a series; the exposure temperature, and exposure time of each being the same as the Lights and applying "Average" or "Median" smoothing before subtracting from the Light.

They also suggest an interesting experiment to verify the "randomness" of the noise and that is to take two Darks at exactly the same temperature and exposure length ( ie one after the other) and subtract them. The "salt and pepper" noise you see is what would be added to your Light if you only used a single Dark.

Hope this helps,

Ken

[jgs001]

I have a question about the amp glow you mention... The 1000d and 450d are very similar, the 1000d being slightly newer. I do not suffer amp glow with the 450d, the only report of it was with someone using liveview to trigger the exposures as the liveview screen is amplified to try and enable you to see the image data.

Are you keeping liveview on during shooting ?

[Psychobilly]

Provided your amglow has remiained constant across the whole imaging sessison?

The Nikon D200 could take up to 10 subs for the ampglow to stabilise it would start of as a faint "purple" glow in the corner which would spread out tocover nore of the frame and along the length of one edge...

All the theories great and Paul certainly knows his stuff but I would still try and get some feel how your own particular camera performs...

Something to do on those rare cloudy nights when you cant get out under crstyal clear skies...

Billy...

[MishMich]

John, yes, I have left liveview on, but am no longer doing that.

Billy, tonight I will be leaving the camera in the cool of the conservatory, and testing setting up remote control imaging, and practice making some darks, bias, flats, flat darks that I can test out with some past images, and for a library of 'just in case' frames in case I find I need them one night.

Ken, I have been looking for some software, and I don't see the point in buying PS.CS4 when I won't use 90% of it. I have been using DSS or Registax, depending on the object. I would value your opinion of AstroArt, as it seems as if it will do a lot I want to do, but is much cheaper than Maxim. Can you do all the post processing and enhancement without using PSE or Gimp? I like Gimp, but it reduces images to 8-bit colour, as does PSE7 if you do anything involving Noel's elements.

M.

M.

[jgs001]

Gotca

According to (I think) it was the DSS help, you either need bias or dark flats but not both..

[Merlin66]

Mish,

I've used AstroArt since V3 and always found it did what I needed.

It can control the mounting, control the camera and provide pre and post processing of images etc etc etc.

Why not try the free download trial version and see what it can do for you.

I use it for my spectra with the DMK21, MX7c, DSI II, QHY5 and the Canon DSLR's.

Astroart 4.0

Ken

[MishMich]

Thanks for reminding me Ken - I did already. Just need to figure out where to start with it.

I've just seen your sig - I grew up just down the road from Cobham.

M.

[MishMich]

Figured out how to load frames - but looks like preprocessing is disabled in the demo version...

M