Standard deviation

[Ken82]

Maybe someone can help with this, I’m trying to evaluate noise in a calibrated image in comparison to uncalibrated. 
 

My understanding is that after Calibration with a master dark the standard Deviation should decrease ?

although I appear to have the opposite?

thanks 

[Merlin66]

Your images seem to show the Standard Deviation reducing from 485.9 to 334.7.......

[wimvb]

3 hours ago, Merlin66 said:

Your images seem to show the Standard Deviation reducing from 485.9 to 334.7.......

The first image is "after".

If the dark is noisy, there may be a substantial increase in std dev

The square of the std dev after equals the sum of the squares of the std dev of the uncalibrated image and the dark (if the noise is truly random)

S_cal^2 = S_uncal^2 + S_dark^2

That's why it's important to take many darks.

[pete_l]

What dark frames do is remove part of the offset signal that the camera generates and also whatever hot pixels there are. This is the part that is dependent on the temperature of the camera sensor and the exposure time. It is important that sufficient numbers of dark frames are taken so that the  resulting dark frame is smooth: no noise, or at least very little noise.

The top frame has a lower "min" value, so it seems that the offset subtraction has worked to some extent. However, bear in mind that the dark frame you subtract cannot be noisy - you can't reduce the random noise in an image by subtracting more random noise We remove the image noise by stacking multiple calibrated subframes.

[vlaiv]

Standard deviation will "capture" any sort of signal - be that random signal such as noise or non random signal.

Only uniform "DC offset" type of image will have stddev of 0.

If you want to evaluate noise between two images - calibrated one and non calibrated one, then do the following - select patch of the sky where there is nothing - not even a single star. That might be tricky if you use a single sub since you don't really know if there is very faint star in there. Best approach is to create a stack aligned to the sub you are inspecting and make a selection on actual stack - because stack will have better SNR and you should be able to spot empty part of the sky much more easily. Save selection to be applied to calibrated and not calibrated sub.

Apply selection and do statistics on selection.

Even having a background gradient will skew your stddev calculations.

In a single sub - calibrating with master dark will increase noise (very slightly - depending on number of dark subs in master - more subs less noise increase), but can decrease stddev even if you have completely empty patch of the sky and no gradient. This is because dark calibration removes offset signal. Offset signal is a signal - and it will impact stddev as well.

[Whirlwind]

You've still got at least some hot pixels in the image so it can't be fully calibrated?  These pixels will skew your standard deviation as they don't represent random (gaussian) noise.  It's unclear whether you've used the same size box either so they might not be directly comparable (especially if you are picking up more hot pixels)?  You may also be picking up actual data whereas before it was more dominated by noise (so you might be sampling signal not noise, but it is difficult to know because the image is too small) - was this just a bias or dark frame?

[Ken82]

27 minutes ago, Whirlwind said:

You've still got at least some hot pixels in the image so it can't be fully calibrated?  These pixels will skew your standard deviation as they don't represent random (gaussian) noise.  It's unclear whether you've used the same size box either so they might not be directly comparable (especially if you are picking up more hot pixels)?  You may also be picking up actual data whereas before it was more dominated by noise (so you might be sampling signal not noise, but it is difficult to know because the image is too small) - was this just a bias or dark frame?

Yea that’s a good point which is why I’m a bit confused. The calibration doesn’t appear to have removed many of the hot pixels at all. Yet they are in the same position on the lights and master dark. I know dithering will remove these but surely dark calibration should have done this ?

For information the master dark is a stack of x100 at same temp etc etc. 

[wimvb]

3 hours ago, Ken82 said:

For information the master dark is a stack of x100 at same temp etc etc.

That shouldn't increase std dev much. 

3 hours ago, Ken82 said:

I know dithering will remove these but surely dark calibration should have done this ?

Calibration should. What camera do you have? Darks don't always work with dslrs for example. But since you've matched temperature, I assume you have a cooled astrocam. Btw, if it's a cmos, make sure you don't scale the master dark.

[vlaiv]

3 hours ago, Ken82 said:

Yea that’s a good point which is why I’m a bit confused. The calibration doesn’t appear to have removed many of the hot pixels at all. Yet they are in the same position on the lights and master dark. I know dithering will remove these but surely dark calibration should have done this ?

For information the master dark is a stack of x100 at same temp etc etc. 

We need to distinguish two types of hot pixels here - or possibly even more cases (probably 4 in total).

First is - saturated hot pixel and second - not saturated hot pixel.

Either of the two can happen in darks and in light sub. To further complicate things - since we are creating master dark from multiple dark subs - any combination of hot pixels can exist in those subs - some saturated and some non saturated.

Hot pixel is just that - pixel that behaves as if not cooled. This means that it's value can be very big compared to other pixels because of accumulation of dark current. It is also "noisy". We know that dark current is poisson process and has thermal "shot" noise associated with it.

What values can calibrated hot pixel produce? That depends on type of hot pixel.

Easiest case is two saturated hot pixels - one in light and one in master dark (which means it was saturated hot pixel in all dark subs or most dark subs if some sort of sigma rejection was used). In this case - we are left with "hole" - or pixel that has 0 value. This is because saturated pixel can have just one value - max value that sensor can record. If you subtract same value you are left with 0.

"Smart" dark calibration should be able to recognize this case and replace such pixel with average of surrounding pixels. Even after regular dark calibration - simple subtraction, you can still "recover" from this case if you use cosmetic correction that removes "dead" pixels (it will be seen as dead pixel because it will have value of 0).

Other cases are rather unpredictable. Smart calibration algorithm will recognize pixel that is hot enough to have saturation value in some of the subs. It will replace it with mean value of surrounding pixels without trying to calibrate it. It can saturate in light sub (or other light subs if algorithm is very smart to examine light subs for such defects as well), or some of the dark subs so we will have reference point.

Worst case is hot pixel that does not saturate. In principle, such pixel will calibrate properly - but due to very high value, it will also have very high error - and it can seem as hot after calibration.

Take for example case where hot pixel has value of 4000 (out of 64000) - this is in fact 40000 +/- 200 (error or noise is square root of value). You now calibrate that pixel, but it will have +/-200e still remaining. If it is on the background - it will look brighter or darker than surrounding pixels.

In the end - CMOS sensors suffer from FPN / Telegraph type noise. This can also produce what seems to be a hot pixel - and it indeed is "hot" but not in regular term. It also has more electrons captured but not because of heat and thermal motion, but rather because of imperfections in silicon and current leakage.

Such pixels often come in pairs and "leakage" is between two such pixels - sometimes leaking into one and sometimes into the other. Here is what this looks like on my ASI1600 camera:

This is stddev stack of 64 dark subs. Higher pixel brightness means that particular pixel has higher standard deviation - on in another words it's values are more noisy then rest. You can clearly see that some pixels are more noisy than others and that often they come in pairs and that those pairs are in diagonal direction.

If I animate that part of darks like this:

You will see how such pixels behave and why it is called telegraph type noise. You can also see why such pixels can be mistaken for hot pixels although they are not.

Solution to this is - take many subs of each kind (both light and calibration subs) and dither.

[Ken82]

Thanks all

I've had a look this evening and Im sure the hot pixels (at least most of them) haven't been removed by calibration. 

Maybe someone can take a look and either tell me I've done something wrong or explain whats happening?

Master dark

https://drive.google.com/file/d/1qf4o9jPVwQ8HYKd62tnodcdZSivny0Jq/view?usp=sharing

Uncalibrated sub

https://drive.google.com/file/d/1OVsJV1MT6jlhA_TnTiMooi_4zwwm4cnr/view?usp=sharing

Calibrated sub

https://drive.google.com/file/d/1-FaT_B0NqHQhk-xcWEEiiSpg8MpRvo44/view?usp=sharing

Thanks Ken

Edit - I think its correct but would like to better understand whats happening 

 

Edited June 16, 2020 by Ken82

[Ken82]

Cancel my last, its removing hot pixels. Whats left over is just warm pixels 👍

[Helen]

I've tidied up the thread to keep it on topic 🙂  (fascinating discussion!)

Helen

[Ken82]

Interesting although standard deviation has increased the median has decreased by the same amount as the median in the master dark. 

[Whirlwind]

 

21 hours ago, Ken82 said:

Cancel my last, its removing hot pixels. Whats left over is just warm pixels 👍

It looks like you are using Pixinsight.  It is worth playing with the Pixel Rejection options as this can be quite effective at removing these sort of pixels - but this is a bit OT.

 

4 hours ago, Ken82 said:

Interesting although standard deviation has increased the median has decreased by the same amount as the median in the master dark. 

This shouldn't be a surprise because of how errors propagate

So for example on a single pixel you take 5 30s darks.  The values you get are:-

300; 301; 310; 281; 295;

So you average value is 297.4 and the standard deviation is 10.644; hence commonly referenced as 297.4 +/- 10.644.  This means there is a 67% chance that the 'true' result is between 286.756 to 308.044; there's a 95.4% chance that the result lies between 276.112 and 318.688; and a 99.7% chance that the true value lies between 265.468 and 329.332.  This commonly referenced as the 3 sigma value and is usually the minimum that would be considered a 'result' (which is also why figures splashed on news/papers/adverts are all generally nonsense because they don't provide the error but that's another topic).  However your camera noise is 297.4, your random noise is 10.644.

Back on topic.  Lets suppose you now have your uncalibrated image and the same pixel on that has values of 

781 +/- 15.344 (i.e. 765.656 to 769.344)

Now a dark is subtracted so would be the uncalibrated image minus the master dark

So how is this undertaken with the errors.  For simple subtraction you have to find the two largest extremes

So hence subtract the most extreme values giving the following result 457.612 to 509.588.  The average value being 483.6 (also being 781-297.4) so hence the error value is 483.6 - 457.612 = 25.988.  

Your new calibrated result is 483.6 +/- 25.988.  Notice the error is also (10.644 + 15.344).

As such although you've subtracted the dark you've added and compounded the errors.  

Therefore from a simple error calculation it should not be a surprise to see your error increase (now you have to add in your flat which is a divide and that makes the errors compound in different ways - see here for example:-

https://thefactfactor.com/facts/pure_science/physics/propagation-of-errors/9502/

 

Edited June 17, 2020 by Whirlwind
Slight tweak

[Ken82]

40 minutes ago, Whirlwind said:

 

It looks like you are using Pixinsight.  It is worth playing with the Pixel Rejection options as this can be quite effective at removing these sort of pixels - but this is a bit OT.

 

This shouldn't be a surprise because of how errors propagate

So for example on a single pixel you take 5 30s darks.  The values you get are:-

300; 301; 310; 281; 295;

So you average value is 297.4 and the standard deviation is 10.644; hence commonly referenced as 297.4 +/- 10.644.  This means there is a 67% chance that the 'true' result is between 286.756 to 308.044; there's a 95.4% chance that the result lies between 276.112 and 318.688; and a 99.7% chance that the true value lies between 265.468 and 329.332.  This commonly referenced as the 3 sigma value and is usually the minimum that would be considered a 'result' (which is also why figures splashed on news/papers/adverts are all generally nonsense because they don't provide the error but that's another topic).  However your camera noise is 297.4, your random noise is 10.644.

Back on topic.  Lets suppose you now have your uncalibrated image and the same pixel on that has values of 

781 +/- 15.344 (i.e. 765.656 to 769.344)

Now a dark is subtracted so would be the uncalibrated image minus the master dark

So how is this undertaken with the errors.  For simple subtraction you have to find the two largest extremes

So hence subtract the most extreme values giving the following result 457.612 to 509.588.  The average value being 483.6 (also being 781-297.4) so hence the error value is 483.6 - 457.612 = 25.988.  

Your new calibrated result is 483.6 +/- 25.988.  Notice the error is also (10.644 + 15.344).

As such although you've subtracted the dark you've added and compounded the errors.  

Therefore from a simple error calculation it should not be a surprise to see your error increase (now you have to add in your flat which is a divide and that makes the errors compound in different ways - see here for example:-

https://thefactfactor.com/facts/pure_science/physics/propagation-of-errors/9502/

 

Thanks whirlwind, 

In terms of the pixel rejection I always use linear fit clipping for a stack of more than 20+ frames which I believe gives the best result. ? I tend to keep the settings as standard. 

I don’t see any problems here I just wanted to better understand what these algorithms are doing etc so thanks for your detailed response 😀

 

[IanL]

If you're looking to compare noise, e.g. using patches of the background sky, use MAD (Median Absolute Deviation). It is a much more robust measure than StdDev. For example removing outliers (e.g. hot pixels) will have a significant effect on StdDev whereas it won't on MAD and thus allows you to compare the underlying noise distribution of the before and after images.

Edited June 18, 2020 by IanL

[Whirlwind]

20 hours ago, Ken82 said:

Thanks whirlwind, 

In terms of the pixel rejection I always use linear fit clipping for a stack of more than 20+ frames which I believe gives the best result. ? I tend to keep the settings as standard. 

I don’t see any problems here I just wanted to better understand what these algorithms are doing etc so thanks for your detailed response 😀

 

In PI when there is an option in ImageIntegration for Pixel Rejection (2).  In here there is a sigma low and sigma high value. These settings help you tailor how many outlier pixels are discarded.  There is a balance (too much and real data gets discarded) but have found it useful to get rid of the remaining hot/warm pixels if used carefully.