Lightbulb moment - a new way to bin images

[vlaiv]

Just had AHA moment while popping down to store to get some sweets (sweet tooth strikes again).

It's about software binning, SNR increase and resulting loss of resolution.

With RC8" and ASI1600, imaging resolution is quite a bit oversampling for my average seeing and guiding (even borderline for best of conditions), so my usual workflow involves software binning my subs after calibration (and even some fancy tricks to minimize impact of FPN).

This increases SNR predictably (x2 for each sub prior to stacking), but has negative impact on resolution due to "pixel size" blur (consequence of area sampling as opposed to point sampling).

One way to go around this is to use scaling rather than binning - depending on interpolation algorithm used and frequency distribution in image, scaling will preserve information in image, but will have poorer SNR gain compared to plain binning.

Straight forward binning is in essence bilinear interpolation and scaling shifted by half pixel - same mathematical operation, although different "mental interpretation".

So different type of interpolation yields different results re SNR increase and loss of resolution. Here is quick "break down" of what happens, depending on method employed - results from test case scenario consisting of synthetic image with single star of known FWHM and "field" of pure noise. This image is scaled down using different techniques and resulting star FWHM and noise field standard deviation measured and compared with original to see any changes.

Base image:
FWHMx     7.0645    
FWHMy     7.0638
Stddev   2.0025

Bin x2:

FWHMx     3.5817    
FWHMy     3.5812
Stddev  1.0025

(Noise decreased as expected by factor of ~2, while FWHM went from 7.0645 to 7.1634 - again increase due to "pixel blur")

Cubic convolution:

FWHMx     3.5353     
FWHMy     3.5348
Stddev 1.23877

(Noise decreased but by factor of about x1.61 rather than 2, and FWHM suffered minimal increase from 7.0645 to 7.0706)

Quintic B-spline

FWHMx     3.5324     
FWHMy     3.5316
Stddev  1.67422

(again, SNR gain drops, and this time FWHM is almost the same as before at 7.0648)

Above results clearly confirm SNR resolution trade off and it dependence on scaling algorithm used. My guess is that Lanczos 3/4 scaling SNR increase will be minimal, but if original image is oversampled - sampling down will have virtually no impact on detail.

Question is, can we have our cake and eat it too? (see pun here related to sweet tooth from beginning of the post? )

Here is my idea, and it is very much inverse of already known algorithm - drizzle integration. In drizzle one uses under sampled data and "drizzles" it over stack thus increasing "space" between samples and turning some of "pixel blur" into noise thus lowering SNR for gain in resolution. My idea is opposite - take subs, split them in 4 smaller subs, by using appropriate pixels for each sub - first sub, even samples in both X, and Y, second sub - even X, odd Y, third sub - odd X, even Y and last sub odd X and odd Y - almost like splitting bayer matrix but instead getting different colors - getting 4 subs.

Since we are using exact samples at exact points - no need for interpolation, and in fact we will only loose frequency information in image due to lower sampling - and if we are over sampling to start with - no information will be lost at all. How about SNR? well each "sub sub" or part of original sub will not have any pixel altered so SNR in each new "sub" sub is exactly the same as in original hence SNR is the same, but now we have 4 different subs that we can stack (signal can be considered same, while each pixel will have random noise - so 4 subs have different noise and benefit from stacking) thus getting exactly the same SNR increase as with binning.

We get binning effect - x2 SNR at no or minimal information loss / blur introduced.

Now I just need to implement and test idea

What do you think?

[vlaiv]

Here is addition to above, I ran another test on sample image, this time using "inverse drizzle" binning.

I divided test image into 4 sub images (I used debayer plugin I wrote and a bit of fiddling because plugin averages green, so I used red and blue channels for different bayer patterns), and translated images to align them by needed 0.5 pixel shifts using Quintic B-spline resampling. After I took average of stack of 4 subs, and here are results:

Stddev  0.9033
FWHMx     3.5326     
FWHMy     3.5322

So this method of binning gives better snr ratio (due to sub pixel shift to align frames - this will not be the case if frames are used in stack directly rather than combining them back in single sub - they need to be shifted to be registered anyway so this value will differ but will be around x2) and FWHM value 7.0652 - almost as good as Quintic B-spline scaling down, better than Cubic Convolution.

Cake anyone?

 

[John78]

So isn't the problem software "binning" is fruitless, and that if properly implemented binning on CMOS sensors was implemented at a hardware level you wouldn't have to baffle yourself in math.

[Stub Mandrel]

The idea of pseudo-binning each pixel so you don't actually reduce resolution came to me when reading one of your other posts made today.

It strikes me there are approaches that have something in common with convolution/deconvolution, deriving a value for each pixel based on a weighted average of it and the surrounding eight pixels.

This could work very well for RGB data, keeping more detail than normal approaches?

[vlaiv]

6 minutes ago, John78 said:

So isn't the problem software "binning" is fruitless, and that if properly implemented binning on CMOS sensors was implemented at a hardware level you wouldn't have to baffle yourself in math.

Not sure what you are trying to say ...

Binning in software is the same as binning on CMOS sensors in "hardware" and almost the same as binning in hardware on CCD sensors.

Only difference is SNR - software binning and CMOS "hardware" binning are exactly the same - they increase SNR by x2, while hardware binning in CCD increases SNR a bit more than x2 (shot noise from both sky and target is reduced by factor of 2 as in software binning but read noise is reduced by factor of 4 unlike software binning where reduction factor is still x2 - so overall SNR gain is a bit bigger than x2 in hardware CCD binning - with largest impact when read noise is dominant component like faint narrowband targets).

That part is not in question here. There is however another part in the story, equally affecting both hardware, software bin and using x2 larger pixels (so it is not related directly to binning but rather effect of binning - surface of pixel). There is blur component that reduces resolution that is tied to pixel size. We know that pixel size is tied to sampling, and via Nyquist theorem to max frequency recorded - this is true for square pixels and for sample points, but square pixels having surface have additional blur component they introduce since they are not single points - which regular Nyquist sampling assumes.

When you take your image at certain pixel scale you will have certain star FWHM in your image - subject to seeing, guide performance, aperture size and used pixel size (tied to above sampling resolution and pixel surface). When you bin your image x2, whether it is in software or true hardware binning - you effectively increase your star FWHM - getting "fatter" stars, and lowering captured detail in image in general - small loss of information but it is there nonetheless. Way around this loss of information is using different kind of scaling for image signal - there are different methods / filters like bilinear, bicubic, cubic convolution, B-splines of certain degreee, O-moms, Sinc, Lanczos, etc ... Each have different properties, and for astronomical images two important properties are SNR gain and loss of resolution. Binning - which is form of bilinear sampling has increase of SNR x2, and on the other hand it has greatest loss of resolution / information of listed methods. Lanczos 3 should have the least loss of resolution but it also (probably - I did not test this) has the least SNR gain.

My idea is solving this - giving you expected SNR gain from binning, without associated resolution loss.

[vlaiv]

1 minute ago, Stub Mandrel said:

The idea of pseudo-binning each pixel so you don't actually reduce resolution came to me when reading one of your other posts made today.

It strikes me there are approaches that have something in common with convolution/deconvolution, deriving a value for each pixel based on a weighted average of it and the surrounding eight pixels.

This could work very well for RGB data, keeping more detail than normal approaches?

Depends on what sort of RGB data you are talking about. If OSC camera is in question - there will be resolution gain compared to regular debayering (interpolating missing values by some method), or super pixel mode - where green gets averaged between two pixels if you split sub into 4 images - 1 red, 1 blue, and 2 green, and then stack according to color.

If regular filter imaging - then we look at subs as mono and above is applicable if you would otherwise bin your data to recover some SNR because you oversampled in the first place.

Taking average of surrounding pixels is in effect a form of blurring - it will lower noise and increase SNR, but since you are introducing correlation between pixels - you are loosing resolution rather than keeping it. Binning does this in simplest form - it uses 4 adjacent pixels and adds or averages their value - it is easy to test that it increases star FWHM as I have done before.

There was discussion about DSO image resolution some time ago, and we were trying to identify all sources of blur that impact star FWHM and their relation to final result. Over the course of thread I've heard multiple times that there is "pixel related blur" but no one could provide me with explanation, and I maintained that this view is probably wrong - until I did my math and indeed concluded that it is related to surface sampling that is done with pixels. Later Andrew provided link to a paper where author examines different blur kernels resulting from different pixel shapes, and square pixel kernel matched my results.

Since that time I dropped idea to do surface based resampling (a form of supersampling) when aligning frames to increase SNR as it effects resolution adversely (binning is form of this with precise scale factor of x2) but I kept using binning as a form of resolution reduction. I will probably switch to above method form my RC8" subs anyway. Just need to implement a bit of Lanczos interpolation for all components to come together

As for RGB imaging and advances there, I have two interesting ideas in the "pipeline". One is related to LRGB imaging - we usually take L to be luminance, and take color from RGB and discard luminance (what a waste to discard data!) - so my idea is to exploit all data as much as possible - why not use Lum to enhance color, and RGB to enhance Lum?

(Hint: if RGB filters are properly subdividing L - ie R+G+B = L, then we can add additional L subs composed out of R+G+B subs to L stack and use algorithm that does weighted average based on noise in each sub - I recently implemented that with excellent results. We can also get additional color subs in similar way L-(G+B) for extra R subs, L-(R+B) for extra G subs, L- (R+G) for extra B subs - again these will have different SNR then subs taken with filter but noise weighted average should deal with that part )

Another idea is to form synthetic subs when doing RGB only or OSC - Lum extraction. Simple approach is to do R+G+B (or weighted sum) but that is not optimal for cases where some component is 0 - as is case for Halpha and OIII parts of target. If you add all three channels, and you have Halpha target - you will just add noise from G and B (since there is no Halpha signal in those two bands) to signal in R - resulting in lower SNR than just using R alone. On the other hand - places where there is no signal at all would benefit from averaging - it reduces noise by factor of ~1.7. It is obvious that there is need for "adaptive" addition of channels (clever one that will optimize SNR based on each pixel rather than assigning weights for whole image).

[John78]

So CCD binning connects photosites together to make effective bigger pixels, importantly, with the associated read noise and dark current of the single/sensor.  CMOS "binning" still samples every pixel as unique and accumulates the value in the a2d and reports it (after its been averaged down to 10bit, this is I hope a software problem to be fixed)

I could well be wrong, but I think a binned CCD gives you a 16bit output of n binned pixels BUT at the read noise/dark current etc... of a square root n pixel.  However you bin a CMOS image you're just averaging or stretching, if you took the 12 or 14 bit raw image and calculate the bin you don't get the advantage of big pixels, because every pixel in the bin has its errors.

So when I say "hardware" I think in the current technologies its absolutely fundamental to how photons are read off a CCD and a CMOS sensor currently.

I'm happy to be enlightened though

[vlaiv]

7 minutes ago, John78 said:

So CCD binning connects photosites together to make effective bigger pixels, importantly, with the associated read noise and dark current of the single/sensor.  CMOS "binning" still samples every pixel as unique and accumulates the value in the a2d and reports it (after its been averaged down to 10bit, this is I hope a software problem to be fixed)

I could well be wrong, but I think a binned CCD gives you a 16bit output of n binned pixels BUT at the read noise/dark current etc... of a square root n pixel.  However you bin a CMOS image you're just averaging or stretching, if you took the 12 or 14 bit raw image and calculate the bin you don't get the advantage of big pixels, because every pixel in the bin has its errors.

So when I say "hardware" I think in the current technologies its absolutely fundamental to how photons are read off a CCD and a CMOS sensor currently.

I'm happy to be enlightened though

You are right - binning in hardware as implemented in CCD is "summing" of electrons from adjacent pixel electron wells prior to ADC. In CMOS it is after ADC since every pixel has it's ADC unit and does not provide for "joining" electrons accumulated to single "place".

Since read noise gets injected in ADC phase - CCD bin injects single "dose" of read noise, but dark current noise, and shot noise are still per pixel - those happen prior to read out (one is related to amount of light and other to thermal vibrations during acquisition time). With CMOS you get 4 doses of read noise as each pixel gets one since it has its own ADC.

Regardless of read noise part - summing of pixel values, whether it is in hardware or software, increases SNR by factor of x2 (this is strictly true in case you sum 4 adjacent pixels and each pixel has same SNR - in reality it is not the case as signal varies between pixels and noise varies across pixels, but for general purpose they are close enough to be treated as same).

SNR increase is because of math of summing and not process of binning - so it works the same if you do it purely in software instead. It is simple to see why it happens if you look at the math involved:

Just look at signal part and noise part separately.

Signal sums as regular scalar values, so resulting signal will be simple sum of 4 values, and if each is the same it will be: 4 x Signal

Noise (truly random one) sums as linearly independent vectors (think rectangle sides and diagonal), so formula for adding noise is: SQRT( N1^2 + N2^2 + N3^2 + N4^2) or for same noise N = SQRT(4xN^2) = 2x SQRT(N^2) = 2xN

Result of summing is 4 times the Signal and 2 times the Noise, or when you divide resulting signal and noise (to get signal-to-noise ratio) you end up with 2xSNR

Simple as that - binning increases SNR by factor of two regardless of method (hardware or software, and with hardware it is even greater than two since read noise, being part of the noise is handled differently and is reduced by factor of 4 rather than 2 like above so total is reduced by more than 2 - and how much more depends on ratio of read noise to other noise sources).

[Stub Mandrel]

It strikes me that L + R + G + B is inefficient. = (2R + 2G + 2B)/3

Surely the most efficient route would be L + two colour channels and synthesising the third by subtracting the other two from L.

Even better might be L + two of CMY.

My guess is that for most target L + M + Y  =( 3R + 2B + 2G)/3  would gather the most data per unit time, therefore highest signal to noise ratio.

You might want L + M + C =(2R + 2G + 3B)/3 to maximise the blue data if it's faint but important.

I think the issue is that people don't trust the idea of deriving a colour channel by subtraction rather than addition.

 

[vlaiv]

2 hours ago, Stub Mandrel said:

It strikes me that L + R + G + B is inefficient. = (2R + 2G + 2B)/3

Surely the most efficient route would be L + two colour channels and synthesising the third by subtracting the other two from L.

Even better might be L + two of CMY.

My guess is that for most target L + M + Y  =( 3R + 2B + 2G)/3  would gather the most data per unit time, therefore highest signal to noise ratio.

You might want L + M + C =(2R + 2G + 3B)/3 to maximise the blue data if it's faint but important.

I think the issue is that people don't trust the idea of deriving a colour channel by subtraction rather than addition.

 

I think that any given approach will have a weakness depending on light distribution in data. Here is quick example on CMY vs RGB.

Let's take white source (equal distribution across channels). On first glance CMY would have advantage by virtue of getting more signal per channel than RGB. So let's suppose that source emits 100 units in each R, G and B band.

R, G and B band are simple to calculate, each will have SNR of 10 (100 source divided by square root of 100, being 10 - in this particular case we are looking at shot noise dominated regime, so read, thermal and LP noise is too small to have an impact and omitted from calculation).

Let's see what will happen when we go CMY route. Initial C, M and Y channel will have SNR of ~14.14 (200 divided by square root of 200), but we need R, G and B components for image composition thus we need math operations to convert channels.

Take R for example

R = (Y + M - C)/2

(for SNR calculations we can omit division by 2 as division and multiplication don't change SNR).

(200, 14.14) + (200, 14.14) - (200, 14.14)

Signal part is easy as it goes like scalar addition = 200 + 200 - 200  = 200, while noise calculation will be SQRT(3x14.14^2) = sqrt(3)x14.14 = ~24.5

So R (via CMY) SNR is 200 / 24.5 = ~ 8.16

By using R filter we had SNR of 10, and by using CMY, although it seems that this approach should have greater SNR because it allows more light in, same R channel will have lower SNR of 8.16.

This is of course for "strong" white source. Things will change if we have faint signal, thus read noise becomes important enough to make significant change, or we have LP (which can have its own distribution over bands), or source emits more in one part of spectrum the rest (red, yellow or blue parts of target).

I think it would be interesting to form a table of all interesting possibilities (mentioned in previous sentence) and calculate what would be best approach to image such target.