Dithering about dithering

[philhilo]

Last night was a great night, camera out all night clicking away on M101, awesome. But when I look at the subs this morning I only end up with 4 hours or so of images out of a potential 5 hr 30min (lost 1.30 due to leaving APT on pause, doh). So where did all that time go? Dithering is a big time loss. Living in a bortle 8+ area I can't run subs of much more than 3 minutes (even with an IDAS LP D2 filter) with my DSLR set up, and last night I ran 2 and 3 minute subs, dithering between each one. 

So my question is, do I need to dither between every sub? And with the suggestion that if you dither, darks are not needed, does that mean that if I do use darks (can be shot on cloudy nights) can I do away with dithering? 

My set up is a Canon 700D, 200PDS for galaxy season, HEQ5 Pro guiding at around 2 arc seconds when its playing nicely. all run by APT and PHD2. Bortle 8+area so even using a harsh LP filter (IDAS LPD2) I can only do 3 minute subs at ISO800.

Appreciate any suggestions or comments. Cheers.

[almcl]

Be interested to hear other opinions, as I have a similar set up, albeit probably Bortle 6 skies. 

I found dithering helped a bit with walking noise, but created problems of its own (even with 20+ subs, I still ended up with clusters of hot pixels in the stacked lights).  

Darks didn't help at all when the camera was uncooled, and even after I devised a external cooler and at least stabilised the temperature, didn't help much with some vicious  amp glow, or with hot pixels.  So now I rely on StarTools' increasingly impressive noise suppression abilities and, if really necessary, painting out individual hot pixels in Photopaint. 

I got rid of walking noise by ditching the guide scope and acquiring a Canon OAG.  This, after guiding, probably made the single biggest improvement in my imaging.

[vlaiv]

In my view:

- one should dither and preferably dither at every sub. If you find it to be too much a waste of time - dither every couple of subs. In reality, it's not that much a waste of time. My dithers last 10 seconds or so. I guide with 3-4s guide cycle and it takes 3-4 exposures to complete dither. That is at most 20% or so spent on dithering even with very short exposures of 1 minute. If I use 3-4 minute exposures, that goes down to less than 10% of time. This is compensated with additional half an hour of imaging for a night of 4h of imaging - so again not such a waste of time.

- If you have good guiding and work on lower resolution - dithering is very important. On high resolutions if you guiding is not so tight - subs will be somewhat dithered "naturally". Dithering is beneficial always, even if you have perfect camera with no issues like hot pixels and such. This is because dithering "spreads around" noise from calibration subs further.

Do remember that you need to use specific algorithms for dithered subs to work the best. You won't get anything special with simple average. You will get above mentioned clusters of hot pixels. Use sigma clip stacking (sigma-kappa, or sigma reject or however it is called in your stacking software) to benefit hot pixel removal from dithering. In fact use such algorithms always when dithering.

[alacant]

2 hours ago, philhilo said:

can only do 3 minute subs at ISO800.

Every other frame should be fine. Only when we go 5min plus do we dither ever frame. If you're at 1 minute, then every three naybe. The benefits it gives far outweigh the short time it takes to settle after dither. You seem to need around 16 dithered frames for the clip to click in so a minimum of 30 plus frames if you're dithering every other. 

HTH.

 

[philhilo]

3 hours ago, almcl said:

Be interested to hear other opinions, as I have a similar set up, albeit probably Bortle 6 skies. 

I found dithering helped a bit with walking noise, but created problems of its own (even with 20+ subs, I still ended up with clusters of hot pixels in the stacked lights).  

Darks didn't help at all when the camera was uncooled, and even after I devised a external cooler and at least stabilised the temperature, didn't help much with some vicious  amp glow, or with hot pixels.  So now I rely on StarTools' increasingly impressive noise suppression abilities and, if really necessary, painting out individual hot pixels in Photopaint. 

I got rid of walking noise by ditching the guide scope and acquiring a Canon OAG.  This, after guiding, probably made the single biggest improvement in my imaging.

Very similar set ups (visual but got sucked into the astrophoto hole?). Bortle 6, that'd be like being in a dark cellar compared with here (in a best Monty Python style). I had hoped to do away with some of the gradients by creating a 'dew shield' for the OTA, sealing the bottom of the OTA, tinfoil and gaffer tape on the view finder. Helped a bit but it stills looks like I have headlights beaming across my stretched images!

I have no walking noise as far as I can tell, hideous gradients and lots of noise yes.

Startools user, very interested. I have been trying to use it as the easiest processing software, but I cant seem to find a good tutorial on an up to date version preferably with some demo data. Is my data good or bad, I have no idea, put unknown data into unknown process and see what happens? Did you just play with it or is there a really good tutorial, so often they say do this and look at the change, but it looks the same. The wipe module baffles me presently, I hit the big buttons to see what looks best.

[philhilo]

3 hours ago, vlaiv said:

In my view:

- one should dither and preferably dither at every sub. If you find it to be too much a waste of time - dither every couple of subs. In reality, it's not that much a waste of time. My dithers last 10 seconds or so. I guide with 3-4s guide cycle and it takes 3-4 exposures to complete dither. That is at most 20% or so spent on dithering even with very short exposures of 1 minute. If I use 3-4 minute exposures, that goes down to less than 10% of time. This is compensated with additional half an hour of imaging for a night of 4h of imaging - so again not such a waste of time.

- If you have good guiding and work on lower resolution - dithering is very important. On high resolutions if you guiding is not so tight - subs will be somewhat dithered "naturally". Dithering is beneficial always, even if you have perfect camera with no issues like hot pixels and such. This is because dithering "spreads around" noise from calibration subs further.

Do remember that you need to use specific algorithms for dithered subs to work the best. You won't get anything special with simple average. You will get above mentioned clusters of hot pixels. Use sigma clip stacking (sigma-kappa, or sigma reject or however it is called in your stacking software) to benefit hot pixel removal from dithering. In fact use such algorithms always when dithering.

Yes I am using sigma kappa clipping - not much choice with starlink at the moment.  Certainly not got a perfect camera (hot pixels like stardust) and guiding is OK, but yes there is certainly a natural dithering effect.

[philhilo]

1 hour ago, alacant said:

Every other frame should be fine. Only when we go 5min plus do we dither ever frame. If you're at 1 minute, then every three naybe. The benefits it gives far outweigh the short time it takes to settle after dither. You seem to need around 16 dithered frames for the clip to click in so a minimum of 30 plus frames if you're dithering every other. 

HTH.

 

Very useful granular info - 16 dithered frames for the sigma kappa clipping. 200 should be sufficient be ok then.

[alacant]

4 minutes ago, philhilo said:

200 should be sufficient

LOL Maybe!

13 minutes ago, philhilo said:

wipe module baffles

Stay with it. It's the single most powerful gradient handler I've tried; no jumping through hoops with old mathematical algorithms required;)

 

[vlaiv]

1 minute ago, philhilo said:

Very useful granular info - 16 dithered frames for the sigma kappa clipping. 200 should be sufficient be ok then.

That really depends on parameters for sigma clipping.

Not sure if people understand how to properly select them. Here is quick explanation:

This is important graph to understand when thinking about kappa value in kappa sigma clip (or just plain old sigma clip algorithm - so many names for it ). It is famous "68/95/99.7 rule".

When we have gaussian distribution of samples - 68.27% of all samples will fall within one standard deviation away from true value. This means that if all subs are without issues - 68.27% of them will have pixel value closer to true value than single sigma.

Going further - 95.45% of pixels will have value within 2 sigma from true value and 99.73% will have value within three sigma from true value.

What does this mean for algorithm - it means that if you want to be sure that you include as much values from your subs - you should choose kappa to be 3.0. This will ensure that almost all proper pixel values will be included and that really anomalous pixel values will be excluded.

 

[philhilo]

2 minutes ago, alacant said:

LOL Maybe!

Stay with it. It's the single most powerful gradient handler I've tried; no jumping through hoops with old mathematical algorithms required;)

 

Aha, somebody who has won prizes and uses Startools. That's a good thing to know. Currently everything ends up looking like abstract art, in a fog, with pepper ground over the top.

[philhilo]

5 minutes ago, vlaiv said:

That really depends on parameters for sigma clipping.

Not sure if people understand how to properly select them. Here is quick explanation:

This is important graph to understand when thinking about kappa value in kappa sigma clip (or just plain old sigma clip algorithm - so many names for it ). It is famous "68/95/99.7 rule".

When we have gaussian distribution of samples - 68.27% of all samples will fall within one standard deviation away from true value. This means that if all subs are without issues - 68.27% of them will have pixel value closer to true value than single sigma.

Going further - 95.45% of pixels will have value within 2 sigma from true value and 99.73% will have value within three sigma from true value.

What does this mean for algorithm - it means that if you want to be sure that you include as much values from your subs - you should choose kappa to be 3.0. This will ensure that almost all proper pixel values will be included and that really anomalous pixel values will be excluded.

 

Not sure if DSS gives me the option on selecting the value of sigma (good job I have a qualification in Lean 6 Sigma, a quality and efficiency tool so I have some idea around this even after 2 nights of 2am...….and another 2 to go 8-) ). Will check it out.

[almcl]

17 minutes ago, philhilo said:

Startools user, very interested. I have been trying to use it as the easiest processing software, but I cant seem to find a good tutorial on an up to date version preferably with some demo data. Is my data good or bad, I have no idea, put unknown data into unknown process and see what happens? Did you just play with it or is there a really good tutorial, so often they say do this and look at the change, but it looks the same. The wipe module baffles me presently, I hit the big buttons to see what looks best.

As luck would have it, Guy posted a 'new' work flow this morning on the StarTools forum here

Happy-Kat pointed out a way to control (or at least ameliorate) noise here

Another user, RKonrad posted some videos which I found very helpful here and here. There are links to the data he used in the demos on there somewhere, as well.

You'll need to download the latest (Beta) version  1.6.392, it's stable, so should work fine.  

[x6gas]

Useful thread.

I've never bothered dithering as I can deal with hot pixels using darks and hot pixel correction but Vlad has, as ever, posted some information that makes me reconsider...

[philhilo]

2 minutes ago, almcl said:

As luck would have it, Guy posted a 'new' work flow this morning on the StarTools forum here

Happy-Kat pointed out a way to control (or at least ameliorate) noise here

Another user, RKonrad posted some videos which I found very helpful here and here. There are links to the data he used in the demos on there somewhere, as well.

You'll need to download the latest (Beta) version  1.6.392, it's stable, so should work fine.  

Awesome, thank you almcl. Once the clouds come back and I catch up on my sleep (although I hope I finally have automated meridian flips sorted so can get some zzzzz) I will get back on Startools. I did use Gimp for very quick stretches and dark backgrounds, but using the preferred settings in DSS for Startools made them impossible to process in Gimp. Inspired, cheers.

[vlaiv]

7 minutes ago, philhilo said:

Not sure if DSS gives me the option on selecting the value of sigma (good job I have a qualification in Lean 6 Sigma, a quality and efficiency tool so I have some idea around this even after 2 nights of 2am...….and another 2 to go 8-) ). Will check it out.

Kappa is sigma multiplier - above is two sigma range. Number of iterations is used to determine sigma and do clipping. In each iteration - all remaining values are averaged and standard deviation is calculated. Only pixel values that lie in avg +/- kappa*sigma range are "passed" to next iteration. Average value in last iteration is result of stacking for that particular pixel.

Above 2.00 and 5 settings are quite aggressive. Out of 100 subs, it will on average leave only ~68.3% samples and reduce SNR improvement from x10 to  ~ x3.85 if all subs are good and without issues!

(in each turn it will throw away ~32% samples and it will do that 5 times).

Much better settings would be - 3.0 and 3. Out of 100 subs that would reduce SNR improvement from x10 to x9.955

[vlaiv]

6 minutes ago, x6gas said:

Useful thread.

I've never bothered dithering as I can deal with hot pixels using darks and hot pixel correction but Vlad has, as ever, posted some information that makes me reconsider...

I did not want to get too technical, but here is what I find to be the most beneficial part of dithering.

People tend to use too few calibration frames. Let's for the moment just consider dark subs. Imagine we have a camera that has 3.5e of read noise (many modern cmos camera have less than that, but CCDs have more). You'll often see recommendation for 20-30 dark subs. Let's go with 25 - its easier to calculate. Stacking 25 dark subs will reduce read noise by x5 in master dark. With cooled sensors read noise is dominant type of noise in darks, so we can say that master dark is having 0.7e of noise in it (3.5e / 5).

Now imagine you have perfect guiding and there is no shift between the subs. Each sub is calibrated by first subtracting master dark, and then subs are added / averaged together. If you don't have shift between subs - same master dark will be removed from all subs for a given pixel. This means that we can "pull it in front of the brackets" - mathematically speaking.

To be very precise mathematically - it is like subtracting same value from each number in average - it is same as subtracting that number from average.

What happens is that you will stack your subs and in the end you will "add in" 0.7e of read noise back in. If you have a lot of subs (and we mostly do) - their total read noise will be small. For example, let's stack 256 light subs. They will also have 3.5e worth of read noise each, but after averaging them out, they will have 3.5 / 16 (16 being square root of 256) = 0.21875e

Now we remove our master dark and hence add 0.7e worth of read noise back in, and resulting read noise will be ~0.73338e - or almost the same as read noise of master dark!

As a contrast, let's examine case where we have perfect dither between every frame. In this case we know that no pixels from master dark will land in exact position twice and we can mathematically treat those values as linearly independent vectors - same addition applies as with regular random noise.

In this case we must calculate each sub - master dark read noise and then stack those 256 resulting calibrated subs. Every single sub will have sqrt(3.5^2+0.7^2) = ~3.56931366e of read noise. We are stacking 256 of them, so result will have x16 less noise, or ~0.22308210372e of noise.

If you don't dither and have perfect guiding read noise will be almost the same as master dark - in this case 0.7e, but if you dither perfectly, read noise of stack will be the almost the same as if you did not use master dark at all and it did not contribute anything - 0.22e

Or in another terms - just dithering in above imaginary example made our background contain x3 less noise (we did not include LP noise and other stuff - and granted it changes picture a little, but this holds - it is always better to dither because of this).

[alacant]

@vlaiv

Thanks for the details.

JTOL. One thing I'm not sure about is the 'natural' dither you mention (caused by poor guiding and/or tracking?). 

IIRC, For dither and the sigma thingy to be effective, the imaging camera should be moved several pixels between frames, not constantly during them. If the latter, the results are gonna look awful. If not, then the camera hadn't drifted far enough.

Cheers

[vlaiv]

1 hour ago, alacant said:

JTOL. One thing I'm not sure about is the 'natural' dither you mention (caused by poor guiding and/or tracking?). 

It can take couple shapes, but I'll point out one that I personally had. You have heard of differential flexure. I had the case of it - not severe enough to cause streaking in single sub, but enough to cause difference of 50+ px between first and last sub of the evening.

Each sub needed to be shifted something like half a pixel or so to be aligned with previous one - so it was not obvious in single sub, but if you compare every third or fourth sub - it looks like they are properly dithered. Result is the same as dithering every third or fourth sub.

This also depends on sampling resolution. For someone sampling at 0.5"/px (and later binning) it is easy to get 1px shift between subs due to drift and still have decently round stars. Let's say that FWHM is something like 2.5" - that will be 5px. Elongation by one pixel will make it 6px in that direction or about 20% larger diameter - that is 1.2 deformation - rather small to be seen, particularly if one orients sub RA-DEC to X-Y. It is easier to spot elongation if its not aligned to horizontal or vertical of the frame.

1 hour ago, alacant said:

IIRC, For dither and the sigma thingy to be effective, the imaging camera should be moved several pixels between frames, not constantly during them. If the latter, the results are gonna look awful. If not, then the camera hadn't drifted far enough.

Ideal dither would be exactly integer steps - even single pixel offsets, but never repeated - in random pattern. If you have integer offsets - you don't need to use interpolation when registering subs - you just shift whole image exactly integer number of pixels in X and Y. Unfortunately this is never so - offsets are always fractional. Interpolation algorithms use neighboring pixel values, sometimes even 5x5 pixels or more. This means that single hot pixel will be "smeared out" to multiple adjacent pixels after sub is registered. For sigma clip to work the best - you want to minimize number of outliers. More outliers you have - more impact they have on average value and they are less to be considered outliers. For this reason you want larger than few pixels dithers. Other reason is that dithering direction is random and software usually does not remember all previous dithers to be able to go where it was not before. To illustrate that - just imagine that you can go only left and right in your dithers. Now imagine that you dither by 1px - and in previous dither you moved right. In this dither you have 50% chance to go to position that you already were just one exposure before. Imagine now that you also dither only left and right, but you can go anywhere from 1 to 10px in single dither. Last dither was 5px right. Now you have only 5% chance to return to that exact position (10% chance to get 5px movement and another 1/2 to get direction right).

Btw, here is what single hot pixel looks like if you move it something like 0.121, 0.788 using quintic B spline interpolation:

It impacts almost 9x9 pixels around it.

[pipnina]

3 hours ago, vlaiv said:

Kappa is sigma multiplier - above is two sigma range. Number of iterations is used to determine sigma and do clipping. In each iteration - all remaining values are averaged and standard deviation is calculated. Only pixel values that lie in avg +/- kappa*sigma range are "passed" to next iteration. Average value in last iteration is result of stacking for that particular pixel.

Above 2.00 and 5 settings are quite aggressive. Out of 100 subs, it will on average leave only ~68.3% samples and reduce SNR improvement from x10 to  ~ x3.85 if all subs are good and without issues!

(in each turn it will throw away ~32% samples and it will do that 5 times).

Much better settings would be - 3.0 and 3. Out of 100 subs that would reduce SNR improvement from x10 to x9.955

So if I get this right, kappa=1 would throw away all but the most consistent 33% of pixel values? While kappa=3 would only throw away 0.27%? And if that's by 5 iterations, it throws away 0.27% five times?

Does this mean, for purely hot pixel removal, it's best to use kappa=3 and iterations 2-5? Does low kappa severely limit benefits of stacking for "going deep" into images or preventing faint details from emerging?

[vlaiv]

14 minutes ago, pipnina said:

So if I get this right, kappa=1 would throw away all but the most consistent 33% of pixel values?

We need to consider two different scenarios - pixels that have proper value and have gaussian (like) distribution. Strictly speaking pixel values are mix of gaussian and poisson distributions. Poisson distribution for large expectation value looks like gaussian distribution, and read noise is gaussian distribution (well, it should be, sometimes it is not quite if there is FPN and such). In any case - let's assume that pixel values are gaussian type distribution around some value.

These are "regular" pixels, but there are anomalous pixels as well. Hot pixels, or cosmic ray strikes or passing satellite or aircraft will sometimes make anomalous pixel value.

Pixel values are just numbers and we can't say that number 16 is valid pixel value and that number 52 is not valid pixel value - there is nothing about the number that says it is good or bad value. It is only when we look at them in terms of distribution that we can say something of that number.

Setting kappa to 1, in the case we have all proper pixel will eliminate ~31.73% of samples and leave 68.27% of samples. You will be "throwing away" a bit less than 1/3 of all samples.

This in itself does not limit how deep you can go - if you image 6 hours - you will stack ~4h in your stack. Image for 60h, you will end up with ~40h in your stack. No matter how many hours you image - kappa of 1 will remove about 1/3 of samples (a bit less). You can still go as deep as you like by accumulating more subs and then throwing 1/3 of them away - granted not sufficient.

But why would you choose kappa to be 1?

Let's for a moment go back to valid and invalid pixels - like I said you can't tell by a number if that particular number is valid or not.

In fact - every number is valid. Imagine you have pixel value of 10 and sigma of 2. This means that 99.73% of all samples lie between 4 and 16 (+/- 3 sigma). Is 52 anomalous number? In reality it is not. You need to "play dice" and decide if that number is valid or not.

In this particular case, that number is 14 sigma away. Probability that 52 belongs to mean 10 and sigma 2 distribution, is so low that you need to image for the whole time universe existed and you still would not end up with number 52 in any of your pixels on your own

Seven sigma for example is 1 in 390682215445.

So we can safely say that 52 is anomalous number - it is far more likely that it is hot pixel or passing airplane or something else than it being genuine member of distribution.

Kappa of 1 removes too much, kappa of 14 includes even almost impossible events. What is then a good kappa to take?

Well one that is low enough but does not impact your samples significantly if you don't have any outliers. Let's says you want your SNR to suffer at most 1%. This means that you want to throw away at most 2% of subs (1.01^2 = 1.0201 = 2.01%). Sigma of 2 corresponds to loss of about 4.5% - we need a bit higher than that. 3 sigma will eliminated only 0.27%, so that is good, but we can choose number between 2 and 3 do get us 2% of subs loss and 1% SNR loss.

This is of course if we use one iteration. What happens with more iterations?

In each iteration we can either remove some numbers or remove no numbers. If later happens - we are done regardless of the number of iterations because once we keep all the numbers - average remains the same and standard deviation remains the same - in next iteration we won't remove any numbers and so on ...

But if we do remove some numbers we both alter average and standard deviation. If kappa is too low - there is greater probability that we will remove some numbers and continue into next round to remove further numbers. Having kappa higher stops this process sooner even if all numbers are "proper".

In general you want your iterations to be 2 to 3 max. It should be based on how many different "hot pixel" values can be present in your numbers. 1 hot pixel group can happen. 2 is very unlikely (you need to have two close by hot pixels with different values and dithering needs to put them on same spot). 3 is almost impossible. I would say that 3 iterations just ensures things and won't change anything if kappa is proper over two iterations (as we have seen if nothing is removed iterations stop early).

This is why I recommend kappa 3 and iterations 3.

There are cases where above settings don't work well. If you happen to have very fast satellite that is also dim - it will produce values that are not very different than the background - like 10-20% brighter - it will be visible in final stack if you push your image and stretch aggressively. For this case I recommend you do two stacks - first is 3.0 / 3 stack and second is much more aggressive stack - like 1.0 / 4 or similar. Then you take only offending pixels from more aggressive stack and paste them over regular stack - this will remove trail even if its very faint - but it's a bit involving to do as you have to select bunch of pixels and copy paste.

 

[pipnina]

50 minutes ago, vlaiv said:

-snip-

 

An excellent explanation!

Kappa-sigma always gave me better results than the more plain methods in DSS, but I never knew what the settings were for.

Thanks for taking the time to write that!

[philhilo]

Thanks to Vlaiv for those explantaions. Will read them again after some cloudy nights (my wife is beginning to think I have given up sleeping). Off topic, but its my thread, a guy contacted me about meridian flips - please contact me again, sorry, I started a reply but now I cant find your message despite hunting through many different messaging apps (not helped by another long late night, one more to go then zzzzzzz).