Qs for mathematicians

[vineyard]

Hello,

I was just idly musing on a couple of Qs which I couldn't figure out the answer to, hence this post!

1.  If I have say 8h of NB data on the same target, same filter over 2 nights.  Is there a mathematical difference between stacking 8h of data in one, or stacking each night separately and then adding them together (normalising each of the two nights stacked images to each other)?

2.  Linked to Q1, I notice that program like APP also allow you to specify if the lights are w different filters (rather than just different sessions).  So is it mathematically better to put all the lights into a single pile in a program like that, or to stack each filter separately and then combine via eg PixelMath?

Thank you!

Vin

(PS: not sure if this should be in Imaging Discussion or Imaging Image Processing subsection so apologies if its in the wrong place)

Edited February 28, 2022 by vineyard
added PS

[vlaiv]

47 minutes ago, vineyard said:

1.  If I have say 8h of NB data on the same target, same filter over 2 nights.  Is there a mathematical difference between stacking 8h of data in one, or stacking each night separately and then adding them together (normalising each of the two nights stacked images to each other)?

Depends on stacking algorithm used and conditions on both nights as well as number of subs produced each night.

Here is simple example of how it can be different. If you happen to have different number of subs taken on each night.

Imagine you have following set of data 10,10,10, 9, 9

Stacking all of the data (taking average) - will be 48 / 5 = 9.6

Now imagine you took 10, 10, 10 on first night and 9, 9 on second night. Stack of the first night (average) will give you 10, while stack of second night will give you 9.

In the end - you average two (10 + 9) / 2 = 9.5

9.6 is not equal to 9.5

Depending on the data - stacking it all together can have different result than stacking each night and then stacking result.

In ideal case - having same number of subs with same SNR each and using average stacking method will produce same result if you stack all together vs two separate stacks, but there is no guarantee that SNR of each sub will be the same on two consecutive nights - LP changes, imaging time changes, possibly number of subs changes, seeing changes and transparency changes.

55 minutes ago, vineyard said:

2.  Linked to Q1, I notice that program like APP also allow you to specify if the lights are w different filters (rather than just different sessions).  So is it mathematically better to put all the lights into a single pile in a program like that, or to stack each filter separately and then combine via eg PixelMath?

Again that will depend on stacking algorithms.

Most stacking algorithms are simply not geared towards handling different SNR subs efficiently.

Readily available algorithms can be divided into two groups - ones that try to handle SNR differences and those that don't. Ones that try to handle SNR differences - often use single number as weight to determine how much sub should contribute to final stack - however this is sub optimal solution.

There is no single SNR value for the image. Here is very simple example - there is galaxy and background. Galaxy contains some signal and hence has non zero SNR. Background does not contain any signal and has 0 SNR. This is of course over simplified - as there is whole range of signal values and associated noises in the image so there is broad range of SNR-s in single image.

There is no single weight that will satisfy such wide range of SNRs present in different subs.

I don't think there is automatic way to do what you propose - even pixel math afterwards is going to be rather flawed in best combining such data. In theory one could devise method to add two or more such sets of data - depending on what they want to achieve with "addition"

One thing that comes to mind is combining R+G+B to form luminance and combine that with L filter. What method of addition is to be used - depends on responses of R, G and B filters themselves and weight for each of components (at each pixel) will depend on standard deviation of samples.

[AKB]

20 minutes ago, vlaiv said:

In the end - you average two (10 + 9) / 2 = 9.5

However, if you do it properly and weight them according to the number of subs in each stack you get the ‘right’ answer

((3 * 10) + (2 * 9)) / (2 + 3) = 9.6

 

Edited February 28, 2022 by AKB

[vlaiv]

5 minutes ago, AKB said:

However, if you do it properly and weight them according to the number of subs in each stack you get the ‘right’ answer

But that is exactly the same as doing the stack of all subs at once so no wonder you get the same result as doing the stack of all the subs at once

You added all the values and then divided with number of added values - fact that you put brackets around them does not mean you did different procedure.

[AKB]

1 hour ago, vlaiv said:

no wonder you get the same result

Yes, of course, but the point is you can do this post-stack, so you don’t have to reprocess the subs you did previously.

[vineyard]

That's interesting @AKB - so I could just PixelMath the nightly stacks using the weightings of the number of subs from each night divided by the total?  That would save time as I could do that as I go.  Ofc I guess as @vlaiv says if the SNR is different night-to-night that might not be as accurate as stacking them all together when each light's SNR presumably gets factored in by the algorithm?

(I'm just trying to figure out ways of optimising my workflow - eg: now I blink & calibrate lights on a session-by-session basis, and will soon just start deleting my raw lights after that so that all I have are calibrated lights which can be used at any time in the future w/o having to do it all again)

[vlaiv]

34 minutes ago, AKB said:

Yes, of course, but the point is you can do this post-stack, so you don’t have to reprocess the subs you did previously.

No you can't

(a+b) / (c+d) is not equal to a/c + b/d

a/c + b/d = (a*d+b*c) / c*d

So first stack which is 30 / 3 and second stack which is 18 / 2 can't be added as 30+18 / (3 + 2) once you have already stacked them.

If you stack first stack and get 30/3 = 10 and you stack second stack - and get 18 / 2 = 9 - and give it to someone else - they will just have 10 and 9 and only thing they'll be able to do is - average those two. They will have no idea that you used 3 subs in first stack and 2 subs in second stack (if you don't provide them with that information).

[AKB]

21 minutes ago, vlaiv said:

if you don't provide them with that information

Exactly.  So you should use that information. 

[AKB]

54 minutes ago, vineyard said:

if the SNR is different night-to-night that might not be as accurate as stacking them all together

In this case, if you have estimated the stack variances (and many processing systems do just that) then there’s a simple formula which provides the minimum variance estimate of a weighted average of the stacks.

Edited February 28, 2022 by AKB

[Martin Meredith]

That info is often saved in the FITS header too. 'STACKCNT' comes to mind.

[vlaiv]

1 hour ago, AKB said:

In this case, if you have estimated the stack variances (and many processing systems do just that) then there’s a simple formula which provides the minimum variance estimate of a weighted average of the stacks.

As I explained above - simple weighted average does very poor job.

Here is an example - imagine you have only 3 different levels of signal in the image - background, outer parts of galaxy and the core.

We have two subs that we want to combine weighted. One is shot with 20% higher LP level.

Say that for first sub we have:

100e from LP, 10e signal from outer parts of galaxy and 150e from core. Read noise is 2e.

In second sub we have

120e from LP, 10e signal from outer parts of galaxy and 150e from core. Again, read noise is 2e.

In first sub - (wanted) signals will be 0 for background, 10e for outer parts and 150 for core. Noises will be ~10.2e, ~10.68e and ~15.94e for background, outer parts and core respectively.

In second sub, again (wanted) signal will be 0e, 10e, 150e but this time, noises will be ~11.36e, ~11.58e and ~16.55e - again for background, outer parts and core in that order.

What weight would you assign to each of those two subs and why?

 

[vlaiv]

1 hour ago, Martin Meredith said:

That info is often saved in the FITS header too. 'STACKCNT' comes to mind.

It might be - but let's imagine following scenario - some sort of sigma reject algorithm is used.

You can't really incorporate information on sigma reject, but it will work differently depending if you are stacking 1/2 or full set of subs. If one night has different conditions than the other - overall signal values might be different in that night. Transparency change of say 0.1 magnitude can make signal be 10% higher. Similarly - in slight change in LP levels can alter SNR.

Variances / standard deviations of two sets and combined set will be different and sigma reject will work differently and produce different results.

 

[AKB]

2 hours ago, vineyard said:

so I could just PixelMath the nightly stacks using the weightings of the number of subs from each night divided by the total?

Just try it.  I think you’ll have a good outcome. 

[vineyard]

I tried it @AKB.  Here are the results.  4 images (240s lights w 294MCPro).  The first is 4h HA on the first night.  The second is 2h56 HA on the second night.  The third is all 6h56 stacked in one go in APP (but shown as two sessions in APP).  The last image is the pixel math version of 1 & 2 (weighting being 60/104 of Night 1 + 44/104 of Night 2).  All images have only had ABE & HT done to them (2nd image has been registered to the first to allow the pixel math to work).

Not sure my eyes are good enough to tell much of a difference but maybe pixel peeping will reveal more?

 

[AKB]

2 hours ago, vlaiv said:

Here is an example - imagine you have only 3 different levels of signal in the image

Of course, you can always go on and construct more complex examples which make an algorithm fail, but it’s a moot point, as you said yourself…

 

6 hours ago, vlaiv said:

Most stacking algorithms are simply not geared towards handling different SNR subs

 

It’s really question of what’s good enough for practical purposes, and I think that the OP has hit the nail on the head…

 

6 minutes ago, vineyard said:

Not sure my eyes are good enough to tell much of a difference

Tony