Dark versus bias-as-dark.

[ollypenrice]

I just pass this on because I found it interesting. For a long time I've used bias-as-dark and bad pixel map rather than darks. In the Atik 11000 I found it better and easier. In the Atik 460 I found it easier and it made no difference.

But... we have just shot a respectable 48x15 minutes luminance on VDB141, the very faint dust nebula, and the result was lamentable. Not a patch on a similar run I did years ago using an Atik 4000 in the same scope. Noise was worse and delicious little structural details just brighter than the background sky simply weren't there at all. My collaborators suggested going back to darks, so I shot some and the data is transformed, not only in reduced noise but in what is clearly resolved just above the background.

Why has 'bias as dark' stopped working in this camera? I don't know, but it is very hot here at the moment and the 460 is only making around -8C. Could it be that 'bias as dark' only works when the thermal noise is held down by the cooler? It normally runs between -15 and -20.

(Meanwhile one of our regular guests has emailed me from the UK to say that he can't stop his darks from introducing vast amounts of noise, so should he try bias-as-dark?) Which only goes to show that astrophotography is a funny business...

lly

[tooth_dr]

Interesting, proving there is no right answer for everyone!

Just to clarify - you use bias-as-darks also with bias-as-bias?

[Star101]

Interesting Olly.

Last night I went back to my Atik 4160EX OSC as I was finding the ZWO 183mm pro a little tedious to work with at the moment with the heat/darks etc.

I am currently creating some new DARKS for the Atik.

I have never tried Bias as Darks before. It's getting cooler here now....I might have to give it a try. 

[andrew s]

I suspect, as you do, the increased  in thermal noise at -8 is your issue. I have the SX 694 which has the same chip as the Atik 460. I measured the number of hot/warm pixels as a function of temperature  by changing the cutoff level for what counts as a hot pixel at each temperature.  This showed the chip had in addition to permanently hot pixel a significant number of warm pixels which increased as in number with temperature.

I will try to find the data and post it.

All darks, bias or real, will add noise however, the more dark frames you use the less it will be.

Regards Andrew 

[andrew s]

Here is the data:

At -10c

and at -20c

others found a similar effect see here http://www.spectro-aras.com/forum/viewtopic.php?f=8&t=1919

Sometime science does work @ollypenrice

Regards Andrew

[ollypenrice]

On 27/08/2019 at 10:33, tooth_dr said:

Interesting, proving there is no right answer for everyone!

Just to clarify - you use bias-as-darks also with bias-as-bias?

No, I don't use bias at all if using darks. It's contained within the dark already. (I do calibrate my flats using bias as a dark for flats but this only works with CCD, not CMOS.)

On 27/08/2019 at 11:22, andrew s said:

Here is the data:

At -10c

and at -20c

others found a similar effect see here http://www.spectro-aras.com/forum/viewtopic.php?f=8&t=1919

Sometime science does work @ollypenrice

Regards Andrew

That's great and does offer an explanation. Many thanks.

Don't put me down as anti-science, I'm far from that! I just suspect that, in imaging, there are many hidden variables which make a mess of the best laid theoretical plans.

Olly

[andrew s]

26 minutes ago, ollypenrice said:

Don't put me down as anti-science, I'm far from that! I just suspect that, in imaging, there are many hidden variables which make a mess of the best laid theoretical plans.

Olly

I accept you are not anti-science Olly. It is often an issue when a too simplistic theory is applied or a good theory applied without an understanding of the assumptions and constraints. The Nyquist criteria is a prime example even in pro circles!

What does surprise me is, that given how easy it is, that owners don't  characterise their own CCD and CMOS cameras.  It is easy to check linearity, gain, read noise, hot and warn, pixels  etc. I know you review cameras and if I were doing it I would make these measurement. I assume there is no call for it.

Maybe none if this matters in artistic as opposed to scientific imaging. Both, in my view,  have a legitimate place in our hobby.  However,  I find the more I understand the easier things become and the less often I get caught out.

Regards Andrew 

PS I almost said "pretty picture" imaging but that might have sounded derogatory and in no away would I wish to imply that to your and others skill and creative excellence.

[tooth_dr]

@ollypenrice Thanks again.  I'll try that:

1) a bias as a dark

2) the same bias as a dark flat

3) the flat as normal

4) what about a BPM??

[vlaiv]

May I chip in?

"Wrong" calibration can work in some instances, but not in all, and therefore I think that we should always advocate for proper calibration.

I have nothing against using "wrong' calibration if that works for someone, provided that they understand why it is working for them. The minute it stops working for them (with any particular data) - if they don't know why, there will be trouble. Once one understands why it is working in certain circumstances, it is ok to use it if you accept approximation that you are doing.

Using bias as dark in calibration can work if you have low dark current and exposure such that low dark current does not build up to certain level, and if dark current is uniform across the sensor.

If it builds up too much - it will mess up your flat calibration. If it is not uniform, you will introduce a sort of "noise" that is predictable in nature (unwanted signal) that you could have removed with proper calibration.

Yes, calibration increases random noise but there are ways to deal with this, and often benefits of removing predictable unwanted signal outweigh a bit more random noise.

Take a lot of calibration frames, and dither and you will minimize impact of added noise in calibration step.

[ollypenrice]

38 minutes ago, tooth_dr said:

@ollypenrice Thanks again.  I'll try that:

1) a bias as a dark

2) the same bias as a dark flat

3) the flat as normal

4) what about a BPM??

I simply followed AstroArt's instructions regarding the making of a BPM. I took my master dark and used Arithmetic-Clip and clipped 2000 ADU off the black point.

18 minutes ago, vlaiv said:

May I chip in?

"Wrong" calibration can work in some instances, but not in all, and therefore I think that we should always advocate for proper calibration.

 

This is a fair point but I tried the 'bias as dark plus bad pixel map' method on my 11 meg camera to start with just out of curiosity when I first read about it. I found it gave cleaner results than using proper darks. (I'm careful in doing darks. I do it off-scope with the metal chip cover on and the right temperature. I also shoot a reasonable number, at least 20 even if they are half hour subs. A while ago I did a new set of darks for this camera to make sure my bias method was still an advantage and I found it was.

The specific issue I found with darks was the appearance of scattered hot pixels in the stack. These did not correspond with hot pixels on the defect map, they were an artifact of stacking with darks and disappeared when I used a bias instead. I've no explanation for this and it didn't happen every time but with bias-as-dark it never happens.

On the  other hand I'll be using darks from now on with Atik 460.

Olly

[Davey-T]

7 minutes ago, vlaiv said:

Take a lot of calibration frames,

Hi Vlaiv, be interested on your take on number of calibration frames needed and the general assumption that more is better.

If you plot them on a simple graph starting with number of frames on left vertical and noise reducing on the bottom axis this becomes more or less a flat line after 16 frames so in your opinion is there any advantage in taking 50 odd as some folk do ? not done any scientific experiments but visually I can't really see it making any difference in my admittedly less than average images taken from my light polluted back garden and 600 second darks take quite a while to collect.

Dave

Rough sketch from memory from an imaging experts book, Tony Hallas possibly, noise reduction versus number of dark frames, curve not accurate 

Will try to find the original to see if my memory of it is right.

[vlaiv]

Here is simple reasoning behind number of darks.

You can either do read noise alone if dark current is low, or you can do combined calculation. I'll do combined calculation in two extreme cases:

a) perfect dither (no subs have aligned pixels)

b) perfect opposite - every pixel is stacked against the same pixels (here pixel means X,Y position on sensor rather than on sky).

Let's examine atik 460 as an example, read noise is 5e and dark current is 0.0004e/s/px at -10C. Let's do reasonable sub length - let's say 15 minutes.

Dark current in this case will be 0.36e per exposure - which is low enough not to impact flat calibration much. Associated noise will be 0.6e. Again not as significant, but let's include it anyways.

Total noise per sub will be - 5.034 (so really not much difference to read noise alone). But here is important thing. Stacking 20 of darks will lower this value by sqrt(20), so it will be 1.126e

With each calibration you are "injecting" 1.126e of noise back into a sub. This means that in

a) - each sub will not contain 5.034e of noise (read+dark) but rather 5.1584e of noise - slight increase but not too terrible.

b) - you will end 1.126e noise to final stack instead (because this values will be constant on each pixel they won't be random and can't add as random, but rather are "pulled out in front of parenthesis").

Imagine you did 4h worth of imaging, that means 16 subs of 16 minutes, so your read+dark noise in final stack will be reduced from 5.034 down to 1.2585, but when we add 1.126e to that, we will end up with 1.6887e of noise.

That is like we stacked a bit shy of 9 subs as far as read+dark noise is concerned.

Let's be a bit more aggressive with number of darks and see what the difference is - let's take 100 instead.

Now master dark will have 0.5034e of noise instead of 1.126e (less then half of original)

in case of :

a) single sub will have 5.059e instead of 5.034e - almost no increase this time

b) 1.2585e of noise + 0.5034e of noise = 1.3554e of noise, or it's like we stacked 13.8 frames instead of 16 as far as read+dark noise is concerned. Much better.

So yes, dither and use a lots of calibration subs. I use as much as 256 of each darks, flats and flat darks because I use shorter exposure time.

[vlaiv]

7 minutes ago, Davey-T said:

If you plot them on a simple graph starting with number of frames on left vertical and noise reducing on the bottom axis this becomes more or less a flat line after 16 frames so in your opinion is there any advantage in taking 50 odd as some folk do ? not done any scientific experiments but visually I can't really see it making any difference in my admittedly less than average images taken from my light polluted back garden and 600 second darks take quite a while to collect.

Dave

Rough sketch from memory from an imaging experts book, Tony Hallas possibly, noise reduction versus number of dark frames, curve not accurate 

Will try to find the original to see if my memory of it is right.

Well, I just did some calculations above. I agree that noise goes down as square root of subs stacked, but in general - it is not how low you get your read noise, it's how that read noise impact the rest - if it is not significant to start with (there are other noise sources high enough) then you won't make much difference, but if it is comparable to some other noise source - then you'll make a difference.

Much larger difference is obtained by dithering then going with large number of subs, but still, I advocate going with large number of subs because you won't be loosing anything. No imaging time to be lost, you can do it during day time and over multiple days.

[kirkster501]

Just clarifying Olly.....  You mean that when you calibrate your lights you use the master bias as the dark to calibrate the lights with?  And you also use exactly the same master bias file as the, well, bias?  So you use the same master bias frame on both steps??????

You also do some odd trick with flats as well I seem to recall?

I have Atik 460 and the darks are very clean.  Conversely, the 8300 chip in my Moravian has filthy darks.  I feel as long as I keep my dark library up to date then I see no issues in continuing to use real darks and bias frames

Edited August 29, 2019 by kirkster501

[Magnum]

Olly, was reading on another site that Bias frames can sometimes be affected by ambient temperature, in regards to the rest of the electronics on the board rather than the CCD itself.

So maybe worth doing a new set of bias for when its very warm or cold.

I was just trying your method of using Bias as darks with my recently purchased Atik383 and it indeed gives an identical result to using darks. even the ADU counts came in identical. I then tried just using Bias as Bias and the result was identical again. so not sure if I will us e bias as darks or just do bias as bias I can't see a difference either way.

So for me Bias and dithering is definitely the way to go for my lazy style of imaging. 

Lee

[tooth_dr]

On ‎28‎/‎08‎/‎2019 at 10:49, ollypenrice said:

No, I don't use bias at all if using darks. It's contained within the dark already. (I do calibrate my flats using bias as a dark for flats but this only works with CCD, not CMOS.)

 

On ‎29‎/‎08‎/‎2019 at 10:07, kirkster501 said:

Just clarifying Olly.....  You mean that when you calibrate your lights you use the master bias as the dark to calibrate the lights with?  And you also use exactly the same master bias file as the, well, bias?  So you use the same master bias frame on both steps??????

Steve - I've attached Ollys reply when I asked the same Q earlier in the thread.