RGB, drizzle and noise

[Joel Shepherd]

I'm having some challenges with noise/pixelation in my images. The attached is a sample of the background of an RGB image comprised from 6x300" lights in each channel (so 90 minutes integration total), calibrated with darks and flats, background extracted (DBE) and drizzle-integrated before combining the channels, all in PixInsight. I realize 90 minutes is not a huge amount of time -- it was what I could collect given our weather and location -- and the SNR will be lower, but the amount of noise surprises me and makes further processing a challenge. So I'm wondering if there's any way to reduce it.

One question is: Drizzle integration effectively increases the dimensions of the image by 2x: e.g., if the lights were 1000x1500 the resulting drizzle-integrated image would be 2000x3000. That seems sure to increase pixelation effects. Would it be a mistake, after integration, to resample the image down to the original dimensions of the lights? Or is drizzle integration not well-suited for a small number of lights?

Second question is: Would it be a mistake to apply noise reduction to the stacked lights for each channel before combining them? Would it help?

Last question is: Is there anything else I can do in pre-processing, or before combination, to get cleaner results in the combined images without giving up an excessive amount of detail?

More light would be fantastic, but one must play the hand dealt ...

 

[andrewluck]

Drizzle integration trades SNR for resolution. It requires a large number of dithered subframes so I wouldn't recommend it in this case.

Determining the source of the noise will require some trial and error. You did have the camera coolers turned on?

Re-stack the lights with no calibration or drizzling and measure the noise. Pixinsight Image statistics can be used for this.

Measure the noise in your master calibration frames.

Once you've done this you'll have an idea of the dominant noise source and can work to reduce it.

Andrew

[Joel Shepherd]

Thanks for the response. The CCD (an Atik 460ex mono) was cooled to 0c and the calibration frames were at the same temperature. I haven't played with the image statistics functionality, so I guess I'll start now.

Last night, I did try producing a second stacking, still using drizzle but with a scaling factor of 1 (so no change in dimensions in the stacked image). So, I'll toss that into the mix though from your response it sounds like if I'm already starting with an SNR problem (i.e. not a lot of lights, no dithering) that drizzle isn't a good choice.

[Joel Shepherd]

Okay, well, that was an interesting experiment.

I stacked six of the blue-filtered lights with different combinations of calibration, rejection algorithms and drizzle (with scale 1 and 2), and then ran the NoiseEvaluation script on them. For reference, the reported noise stddev for a sample light (no calibration) was 0.002815.

Observations:

1) For that small number of lights, percentile-based rejection of outliers gives slightly lower noise than average sigma rejection (0.0008917 vs 0.0009079).

2) My darks (30, stacked into a master) are hurting more than helping. With percentile-based rejection and no drizzle, the stack without darks was better than the stack with (0.0008917 vs 0.0009742). That surprised me.

3) Drizzle definitely helped, but drizzling with scale 1 gave slightly better results than with scale 2. E.g., with no darks, and percentile based rejection, integration without drizzle yielded a noise stddev of 0.0008917, with drizzle and scale=1, 0.0005112 (big improvement) and drizzle and scale=2, 0.0005227 (slightly worse).

So, at least for these lights, it seems like the darks are hurting a bit, as is drizzling with scale > 1. So I guess I'll reprocess everything without dark calibration and see where that ends up.

Any theories on why darks would make things worse ... other than bad darks (and I'm not sure how that would happen)?

[vlaiv]

Darks, bias and flat frames always lower your SNR.

Take for example dark frames, each dark frame has some read and some dark noise. Read noise is feature of sensor readout and there is nothing to do about it. Dark current noise is similar to shot noise - it is exposure time dependent. It is related to how much dark current gathers over time of exposure. This one can be controlled with cooling (lower temp gives lower total dark current - this leads to lower dark/thermal noise). When you stack your dark frames you improve it's SNR but some noise still remains. When doing arithmetic with two images that contain noise - noise is increased (formula being sqrt(sqr(noise_a)+sqr(noise_b))). So when calibrating with darks you inject some additional noise. Same goes with bias frames and flats.

There is no real reason for drizzle x1 to increase SNR, at least none that I can think of. This observation regarding x1 drizzle is more likely related to method used to measure/approximate noise level in final picture. My guess is that x1 drizzle influences image in such a way that used method for estimating noise level is sqewed somehow.

[ollypenrice]

Why are you only cooling to 0C? I'd have thought that, in Seattle, you should be able to go lower? If you can, you should. 0C is very high.

In your initial write up you don't mention dither guiding. Drizzle will only work with dithered subs.

Whatever the theory of darks, the practice does not always support their effectiveness. When I used darks, carefully taken and with set point cooling on the large format, noisy, Kodak sensiors I use, I usually had hot pixels to clean up. I now subtract a master bias as a dark and use a bad pixel map. My stacks come out much cleaner. This is the case even though, with the dual rig running different exposure times, it is impossible to dither guide. Plenty of experienced imagers find darks to be of mixed and unpredictable effectiveness. Many prefer 'bias as dark' subtraction, dither guiding and Sigma Reject stacking to remove hot or rogue pixels.

If you find a resolution advantage from drizzle in the high-signal part of the image, but at a cost in terms of noise, then the obvious thing to do would be to use Layers in Photoshop to cherry-pick the best of both.

Olly

[Joel Shepherd]

The 0c cooling is essentially a "left-over" setting from last summer. You are right, though, Ollie: I could cool it considerably more at this time of year. That said, supposedly the sensor in the 460ex is extremely low noise: "dark frames are no longer even a requirement" according to Atik's site. Marketing hype?

Dithering is on my short list of things to try next: I haven't played with it to date because I'm still struggling to get guiding working consistently. I'm looking forward to seeing the difference that dithering might make. We may have a couple clearish nights coming up so it might be a good time to give it a shot. Maybe that and/or more lights will make a difference.

<whine>RGB imaging seems about an order of magnitude harder than simple grayscale imaging.<whine> I can't help but feeling there is some insight I missing in terms of processing that will result in a manageable background. The high signal objects seem to have reasonable color when combined but the background is a psychedelic gritty mess, like hot cereal and food coloring mixed together and spread on a platter. The kids like it, but ...

[ollypenrice]

Ah, well if you didn't dither then there can be no possible benefit from drizzle stacking, which is based on the principle that slight movements of the target on the chip between subs introduce new sub-pixel information into a stack, provided the system is under sampled. Here's a characteristically clear explanation from the ever helpful Craig Stark. http://www.stark-labs.com/craig/articles/assets/Drizzle_API.pdf

I would expect you to have far less noise using a median or sigma stack.

I wouldn't call Atik's observation about darks 'hype.' It is predicated on cooling to a reasonable temperature and I'm not sure that zero C qualifies. It's marginal, at least. I know that the cooling on this camera could be more powerful for those blessed with warm nights. My first CCD camera arrived here in hte Alpine winter and worked fine. As we headed into the summer the noise became worse and worse. It transpired that the cooloer had never worked but that low ambient temperature made it OK. This, like yours, was a Sony chipped camera and with the cooler working it never needed darks.

The background issue seems odd. Are you shooting in LP? How bad, if so? I've just been helping a guest with some RGB from London and the background is certainly rather as you describe. Background Neutralization in Pixinsight is a good weapon in your favour. You can also colour select the background in Ps and lower the saturation.

Olly

[vlaiv]

3 minutes ago, ollypenrice said:

...

I would expect you to have far less noise using a median or sigma stack.

...

Why would you expect this?

[ollypenrice]

3 hours ago, vlaiv said:

Why would you expect this?

Becaue the drizzle would slightly increase low level noise? I'm not sure about that though.

Olly

[vlaiv]

4 minutes ago, ollypenrice said:

Becaue the drizzle would slightly increase low level noise? I'm not sure about that though.

Olly

I probably did not phrase myself correctly. I was under impression that sigma clip and median methods are used under circumstances that are specific. For example if you happen to have dead/cold pixels or hot pixels (when dithering), or there is some sort of cosmic ray impact event that you would like to circumvent. From what I understand for "normal" situations, where individual samples follow Gaussian distribution, median, mean and sigma clip all give same result. There also might be a benefit of using special stacking methods when stacking exposures of different length. In this case subs of lower exposure length would have their noise spread over bigger range of values (when scaled to appropriate exposure length), and sigma clip would ensure that all outliers are removed - it is greater probability that outilers come from shorter exposure - one with lower SNR. Anyway you mentioned that you would expect far less noise from these methods, and I was wondering if I'm missing something crucial in my understanding of how these methods work.

[ollypenrice]

If you decide not to use darks then the hot and dead pixels will always be outliers provided you dither. I think the OP should give this a try. (Subtract bias as dark, dither and stack in Sigma.) I'd expect Drizzle to sharpen the noise if it did anything at all other than resize on an undithered set of subs. 

Could Joel show us if his background were any quieter if stacked using Median or Sigma rather than drizzle?

[Joel Shepherd]

I'm shooting under lots of LP: a mile from downtown. Not as bad as for folks shooting from near the big European or US east coast cities, but still significant, probably Bortle 6.

Per the numbers I gave earlier, stacking with drizzle (scale=1) apparently reduced the stddev of the noise significantly which I'd expect to reflect a better SNR. But if it's actually increasing noise in low SNR areas, that could be a problem.

I'll take another look at Background Neutralization. I seem to recall trying it before without much success but it's entirely possible that my technique was lacking. :-)  Can certainly see how it could simplify processing if it lives up to its name.

[vlaiv]

I'm getting slightly confused with all of this

I was under impression that drizzle is way of distributing pixel values rather than stacking technique. So you can drizzle with mean stacking, sigma clip, or median or any other kind of stacking. From what I understand SNR decrease when using drizzle comes from fact that you are "spreading samples over larger surface" and as a result you decrease number of samples per pixel - same as stacking fewer subs. Drizzle on undithered set of subs would produce holes in final image (some pixels would end up with no value at all), and if dithering was inadequate some pixels will have only a few values for stack - this will result in very low SNR (essentially SNR of single or few subs for that pixel). On the other hand if drizzling x1 - you do nothing to pixel values - no spread, all values fall where they should fall so no difference in SNR is to be expected.

For hot pixels for example, even with proper calibration you might end up with outliers. Take for example really saturated pixel (90% of well) and add to it 50% well depth worth of signal - it will cilip at 100% and when you subtract dark you end up with value of pixel 10% - that is far away from 50% that you should get from signal. This is why sigma clipping is good, this sample will be discarded.

[Joel Shepherd]

It might be a few nights (skies finally cleared!) but I'll try to do a non-drizzle processing in the next few days and post a bit of representative background.

@vlaiv : your comment that "Drizzle on undithered set of subs would produce holes in final image" and Ollie's observation that drizzling depends on dithering makes me wonder if I've been processing things incorrectly for months now. A sprinkling of nearly black dots in my background is common. Maybe I just haven't been paying attention, but I swear this is the first I've heard that drizzle and dithering go together. Not questioning its verity; just scratching my head.

Anyway, I'm going to put this aside for a few hours and go try to catch some actual photons, but will reprocess the Leo Trio and post the same background when I get an hour or two.

Thanks for the interesting and informative discussion!

[ollypenrice]

Dither and drizzle absolutely do go together.  Check out the Craig Stark article in my earlier link. I've only used it in solar fast frame imaging where it's phenomenally effective because a) the seeing alone builds in natural dither (and you are running unduided anyway) and b ) you are working with hundreds of subs. In an undithered stack there is no information in one sub which is different from that in another in terms of locating the target on the chip.

Olly

[Thommy]

You could consider to do a mild Gaussian blurring of the background in each colour channel as well.

/Thommy

[wuthton]

This video of Craig Stark's is often posted but if you haven't seen it then it's well worth a watch. It's long but full of valuable information from beginning to end

 

 

[Joel Shepherd]

Alright ... took a little longer to follow-up than expected. Been putzing around with different processing techniques, but want to get back on track.

Attached is the background from the combined RGB image after doing combination of bias-subtracted lights (no dark substration) in average with percentile-based pixel rejection and no drizzle. That definitely made an improvement. For one thing, it enabled me to to do a reasonable job of background extraction on the individually stacked colors before combining. The result is still somewhat noisier than I'd like. But I'm leaning towards believing that the lights themselves have poorish SNR -- the air had a fair amount of moisture in it and I didn't 2x2 bin the colors -- and there's going to be a limit to what I can do with them.

This has been a good learning experience and I really appreciate the commentary. I have now 3 hours of 2x2 binned RGB and another hour of unbinned luminance taken in somewhat better conditions with poor man's dithering (pausing and slightly moving the scope between sets of exposures) that I can process and am hopeful that the improved SNR and lessons learned about drizzle and stacking will pay off.