Darks, flats and bias files? could they be used over and over?

[jason32]

Do we really have to shoot darks and flats everytime we go out? Why cant we shoot darks like 100 darks and 100 flats and 200 bias frames and calibrate them and use them for every image we shoot? that will save 90% of the work?

[perfrej]

Of course you can! BIAS frames stay the same regardless of how you fiddle your rig, as do darks. Flats may have to be re-taken each night if you remove the camera from the scope, but possibly not even then.

 

/per

[pyrasanth]

I use my bias & dark files for perhaps six months then I redo them. If you want to use the same flats then you must, if using filters, not change there position & keep the CCD camera windows scrupulously clean- this is very difficult if you remove the camera. I suggest a flat panel might be a great help as you can shoot new flats whenever you need to. I even obtained a new dust bunny when I was shooting new flats forcing me to scrap 50 subs-I think a bit of debris must have fell on the CCD window as the flats were being taken.

I'm shooting a new set of flats as I write this. I integrate 50 subs at once as this gives me a SN ration of 7 (100 will give you root 100 or 10- law of diminishing returns kicks in). The new Sony cameras have very low noise but you will still need to flat field. Your results will vary with other brands so its good to get into the habit of darks & bias.

The difference between good calibration frames & no calibration is dramatic & well worth the effort.

[jason32]

2 minutes ago, pyrasanth said:

I use my bias & dark files for perhaps six months then I redo them. If you want to use the same flats then you must, if using filters, not change there position & keep the CCD camera windows scrupulously clean- this is very difficult. I suggest a flat panel might be a great help as you can shoot new flats whenever you need to. I even obtained a new dust bunny when I was shooting new flats forcing me to scrap 50 subs-I think a bit of debris must have fell on the CCD window as the flats were being taken.

I'm shooting a new set of flats as I write this. I integrate 50 subs at once as this gives me a SN ration of 7 (100 will give you root 100 or 10- law of diminishing returns kicks in). The new Sony cameras have very low noise but you will still need to flat field. Your results will vary with other brands so its good to get into the habit of darks & bias.

The difference between good calibration frames & no calibration is dramatic & well worth the effort.

I have a canon 100D and i use idas d1 lp filter. Ok I have a light box that I could use to get flats but its a hassle doing it everytime.

[pyrasanth]

Just now, jason32 said:

I have a canon 100D and i use idas d1 lp filter. Ok I have a light box that I could use to get flats but its a hassle doing it everytime.

Sadly hassle is all part of the game- when you aspire to produce the best result you can get sometimes you just have to dig in & bear it! Sadly astrophotography is not easy- it has made my hair go grey & my bank account empty.

[jason32]

Just now, pyrasanth said:

Sadly hassle is all part of the game- when you aspire to produce the best result you can get sometimes you just have to dig in & bear it! Sadly astrophotography is not easy- it has made my hair go grey & my bank account empty.

hahaha yeah I am learning the craft...I think the most important thing you learn with experience is being patient...lol

[perfrej]

Sky flats are so easy to shoot  I have a Neumann panel that hasn't been used for the past three years or so...

 

/er

[Catanonia]

Darks :Build a library of darks for all the exposure times you use, create a master for each and re-use them for about 6 months

Bias : If you are using darks, no need for bias as it is already in the dark

Flats : Shoot these every image session to ensure all dust bunnies etc are accounted for. If your image train isn't moved or rotated, then you can re-use the flats.

[The Admiral]

Forgive my ignorance on this, but as a beginner to astrophotography I thought that the darks were meant to correct for thermal effects. As such, I thought they'd be dependent on environmental temperature at which you did your imaging, and therefore needed doing each time. Unless you've a cooled chip of course.

Ian

[Catanonia]

11 minutes ago, The Admiral said:

Forgive my ignorance on this, but as a beginner to astrophotography I thought that the darks were meant to correct for thermal effects. As such, I thought they'd be dependent on environmental temperature at which you did your imaging, and therefore needed doing each time. Unless you've a cooled chip of course.

Ian

Not heard that before. Darks are taken to remove the noise the chip makes due to electronics.  Having a cooled CCD reduces the noise, but darks are best used to remove the random noise effect.

 

[The Admiral]

1 hour ago, Catanonia said:

Not heard that before. Darks are taken to remove the noise the chip makes due to electronics.  Having a cooled CCD reduces the noise, but darks are best used to remove the random noise effect.

 

But surely the noise from the chip and electronics is, at least in part, dependent on temperature? That's why cooling the sensor works.

Ian

[jason32]

Yes I thought darks are used to reduce thermal noise right? but is that random or will show on the same pixels everytime?

4 hours ago, The Admiral said:

Forgive my ignorance on this, but as a beginner to astrophotography I thought that the darks were meant to correct for thermal effects. As such, I thought they'd be dependent on environmental temperature at which you did your imaging, and therefore needed doing each time. Unless you've a cooled chip of course.

Ian

Yes I thought darks are used to reduce thermal noise right? but is that random or will show on the same pixels everytime?

[The Admiral]

For what it's worth, here's my understanding, from what I've read.

When a sensor is exposed for substantial periods of time, some significant signals will be present which are produced within the sensor/electronics, but which are not caused by the target itself. One is the presence of 'hot' pixels, or 'cold' pixels, and the other is a random background noise . Although random, this noise doesn't average out to zero, and will increase with exposure time and ISO setting. I believe this random noise is temperature dependent, i.e. increases as the temperature increases. So in order to remove these spurious components from our images, we cap off the 'scope and repeat our exposures (the darks) so that we record just these spurious effects, then average them, and subtract the result from our subs. It follows that the darks should be made under identical conditions to which the subs were made, i.e. ISO, exposure time, and temperature (because some of the noise is temperature dependent). Unless a sensor is artificially cooled, it will operate at a temperature which is a balance resulting from the heat which itself generates, and the cooling in ambient air. It will thus be dependent on the environmental conditions pertaining during making the subs.

Certainly all the advice I've read is that darks should be made under identical conditions as the subs, and that it makes good sense to make the darks during the same session as the subs. It would also follow that darks should only be related to an individual imaging session. Theory is one thing of course, but how much of this makes any practical difference I don't have the experience to say, though there are enough of you highly experienced folk out there who can say.

That's my two-pennyworth anyway.

Ian

[ollypenrice]

14 hours ago, Catanonia said:

Darks :Build a library of darks for all the exposure times you use, create a master for each and re-use them for about 6 months

Bias : If you are using darks, no need for bias as it is already in the dark

Flats : Shoot these every image session to ensure all dust bunnies etc are accounted for. If your image train isn't moved or rotated, then you can re-use the flats.

You will need bias, though, to subtract from your flats. If you don't bias-subtract flats they will introduce noise and will probably over-correct the lights.

If you don't have to remove your camera from the scope then flats will probably last for a very long time. Nothing on the objective appears in the flats. Bunnies come from the opposite end of the light path. If your scope has a rotator then everything at the bottom end turns together and so, again, the same flats will work. I'm using flats taken in July of last year on one scope. They are still perfectly effective.

I'm not a fan of darks and gave up on them, even with the very noisy Kodak full frame 11 meg chip. I subtract a bias instead of a dark and run a defect map and a very aggressive hot pixel filter. This gives the cleanest stacks I've ever had. I find the hot pixel filter does no damage at all. (This is stacking in AstroArt 5.)

Ian, you are right that darks should be taken under identical conditions of temperature and settings. Unfortunately, without set point cooling this is impossible. The temperature will not be the same and will vary during the exposure anyway. This is why darks can be more harmful than beneficial, especially on DSLRs without set point cooling. Also, capping off the scope does not work for me and will certainly never work on a Newt with an open bottom. However it does it, light does get into my refractors despite metal lens covers and sealed electric wheels. I can only shoot darks off the scope under a metal chip cover. (Atik supply hefty screw in ones for just this reason.) If I were using a DSLR I would use a master bias as a dark and dither widely (12 pixels or so) between subs. This will kill bad pixel noise in a sigma stacking routine and dramatically reduce colour mottle in the background. Thanks to Tony Hallas for this method.

Olly

[The Admiral]

That's interesting Olly, thanks.

1 hour ago, ollypenrice said:

If I were using a DSLR I would use a master bias as a dark and dither widely (12 pixels or so) between subs. This will kill bad pixel noise in a sigma stacking routine and dramatically reduce colour mottle in the background.

Sorry to hijack the thread, but as a stalwart Alt-Az user, given that the 'tracking' is nothing like as good as with a correctly set up EQ mount, I presume that the image meanderings would be akin to dither for this purpose? Why would you use a master bias as a dark, as opposed to just not using a dark at all?

Ian

[ollypenrice]

26 minutes ago, The Admiral said:

That's interesting Olly, thanks.

Sorry to hijack the thread, but as a stalwart Alt-Az user, given that the 'tracking' is nothing like as good as with a correctly set up EQ mount, I presume that the image meanderings would be akin to dither for this purpose? Why would you use a master bias as a dark, as opposed to just not using a dark at all?

Ian

On the dither effect, yes, you are right. For the same reason plenty of DS imagers using EQ mounts don't finessse their PA beyond a certain point unless they want 30 min subs. A tiny bit of 'rotational dither' is not a problem and can be an advantage.

Why subtract a bias? Because it is a very reliable source of information on inherent low level, consistent noise since it is not of thermal origin. Once subs become longer then thermal noise (dark current) dominates. The bias also resets the camera's brightness scale which is artificially boosted by a small amount at capture to avoid negative values. It is easy to shoot 50 or 100 bias frames and these, when combined into a master, will always (a dangerous term, so 'should always') have a positive effect on SNR. Subtracting a bias as a dark and dithering, then stacking using a sigma routine to reject outliers, is probably the best way to clean up a stack. Dither means that hot pixels will always be outliers and so rejected. They will be assigned the average value of the rest of the stack.

There is also 'shot noise' which is an unavoidable characteristic of light described by quantum theory. The only defence is the multiple sub exposure averaged into a stack.

Olly

[The Admiral]

OK, thanks Olly. So, let me be clear on this, if I was to be stacking in DSS, you are saying that I would load the image files as lights, and the bias files as darks? You are not saying to load the image files as lights, bias files as bias, and the bias files again (if DSS lets you) as darks, are you? Because if so, I'm not clear how that would work. I realise that DSS is a rather different beast compared to PixInsight, for example, where you have a lot more control over the processing prior to stacking.

Ian

[ollypenrice]

13 minutes ago, The Admiral said:

OK, thanks Olly. So, let me be clear on this, if I was to be stacking in DSS, you are saying that I would load the image files as lights, and the bias files as darks? You are not saying to load the image files as lights, bias files as bias, and the bias files again (if DSS lets you) as darks, are you? Because if so, I'm not clear how that would work. I realise that DSS is a rather different beast compared to PixInsight, for example, where you have a lot more control over the processing prior to stacking.

Ian

If it would be helpul in avoiding confusion you could rename a master bias as 'bias as dark' or some such thing. You would then load this into DSS in the section allocated to darks. You could also re-name a master bias as 'Dark for flats' and put that into the appropriate slot in DSS as well. Of course the names don't change anything but might help preserve a little sanity! It's easy to become boggle-eyed when stacking, especially for me with three cameras, three sets of calibration files and at least a triple helping of failing neurons in my bonce...

Olly

[derrickf]

One thing you may need to pay attention to is whether DSS reads the FITS header IMAGETYP value to determine whether a submitted file is a valid dark, bias etc. (I have never used DSS so I don't know, but Maxim and PI do read this field and won't allow a mismatch). The best way to find out (unless the documentation makes it clear) is to to try what Olly suggests above - if DSS complains then you will need to find a way to edit the appropriate value in the FITS header (MaxIm and, I think, PI both make this easy).

HTH

 

Derrick

[happy-kat]

Very interesting read, if I can skip the darks I could take more lights and bias in the time I have using the Altaz mount. I'll follow this post.

[ollypenrice]

15 minutes ago, happy-kat said:

Very interesting read, if I can skip the darks I could take more lights and bias in the time I have using the Altaz mount. I'll follow this post.

Don't use imaging time for darks and bias! Terrible waste of time. More subs win hands down. Bias can be done any time and stored. As for darks, without set point cooling they may well be worse than useless. You could always try a temperature-based darks library but from what I gather darks are going rapidly out of fashion amongst expert DSLR imagers.

Olly

[Catanonia]

Some confusion with my post. I was referring to cooled CCD's and hence a library of darks at a set temperature. Apologies for any confusion and of course they do reduce thermal variations in CCD readout. I thought somehow you meant thermal currents in the tube.

Also when I take my darks library,  I take the CCD camera off the scope and put a proper lens cap on it to ensure no stray light as Olly says will always get into a newt.

Darks, flats, bias = massive debate with everyone doing their own thing

[ollypenrice]

Yup. Experiment rules the day. I have moved away from darks with setpoint CCD because I get better results (very significantly better and very consistently better) from BPM/bias/hot pixel filtration. But that doesn't mean everyone will. The only thing I never do is allow myself to be ruled by the theorists because this would be unscientific. In these discussions the theorists have a slight tendency to regard the experimentors as rural numpties armed with pitchforks and flagons of cider. But it ain't the histogram you hang on the wall and it ain't the SNR that you hang on the wall. Its the flagon of cider image!

Hic.

Olly