Can you solve this puzzle about background noise?

[Lee_P]

Hi SGL Hive Mind,

I’ve got a real head-scratcher of a problem, and I’m hoping someone here can help me solve it. I’ve been experimenting with seeing the effects of increasing integration time on background noise levels. My understanding is that the greater the total integration time, the smoother the background noise should appear. But I’m finding that beyond one hour of integration, my noise levels see no improvement, and even maintain the same general structure.

I flagged this in another thread but think it deserves its own thread, so I thought I’d begin anew.

 

I figure either my understanding of integration and noise is incorrect, or maybe I’ve messed up something in pre-processing. I’ve conducted a lot of tests with different settings, copied below, but nothing seems to make much difference. I’ve uploaded my data to GDrive, in case anyone’s feeling generous with their time, and would care to see if they get the repeated noise pattern!  (Being GDrive, I think you need to be logged into a Google account to access).

My telescope is an Askar FRA400, and the camera is a 2600MC-Pro. All a series of 120-second images shot from Bortle 8 skies. For each test, I applied some basic functions in PixInsight just to get images to compare: ABE, ColorCalibration, EZ Stretch, Rescale to 1000px. I used SCNR to remove green from the first tests, but forgot that step for the second batch.

Any idea what's going on? Why isn't the noise smoothing out past the one hour mark?

 

 

Here are my PixInsight ImageIntegration settings:

[scotty38]

I cannot really add much apart from to say I have noticed ABE introduces noise similar to what you see here and, as yet, I've not been able to work out what settings I need to change or if it's my data. APP can make a better job of the stacking using its defaults. Having said that I know there are enough folk getting great results out of PI that I have basically accepted it's me/my data at fault.

I can see the screen shots but are you using WBPP and have you looked at Adam Block's videos for V2 of the script as there may be info in there that could help - I've not watched them all yet btw....

 

Edit just to be clear and what I mean re ABE adding noise is that if I use STF on a stacked image and then apply ABE the ABE version has noise whereas the STF image does not.

Edited May 13, 2021 by scotty38

[Seelive]

Last year I made some (rough) measurements of the effect of different stacking methods (I use DSS). I generally try to use at least 80 lights (180s each, so 4 hrs total).  I found that using Average stacking the resulting noise closely followed the 1 / ROOT(N) formula and similarly for Kappa-Sigma stacking, but only up to about 30 lights. After that the resulting noise leveled off and occasionally appeared to increase slightly.  These days I tend to Kappa-Sigma stack the lights in seperate batches of 20 to 30 and then average stack the results.  It seems to give me the same resulting noise as average stacking does but without the satellite trails and hot pixels.

[Laurin Dave]

Have you measured the different stacks in Subframe Selector? It looks to me as if the longer integration time stacks are getting harder stretches, maybe put the same stretch on each stack and then compare, also what's happening to M81 and M82 are they gettng better?

[vlaiv]

2 hours ago, Lee_P said:

Any idea what's going on? Why isn't the noise smoothing out past the one hour mark?

Can you post just couple of linear stacks without stretching them (as fits please as I don't have PI and can't open xifs)?

Visual feedback on how much noise there is - depends on stretch level. I think that you used PI automatic stretch - and that one works on basis of noise floor - this is why it stretched each sub so that noise in background is the same.

You can see that by size of stars in your animations.

there is much more "light halo" around left star then on the right - but they are the same star (you can see this in your animated gifs - just watch any single star and see it "bloat" as total exposure increases).

In order to measure noise accurately - take two stacks still linear and select piece of background not containing any signal - no stars, no objects, mostly empty and do stats on each image in that region.

Compare pixel standard deviation values between images.

If you want to visually asses difference between two images - make sure they are normalized (one with respect to another) - or that they are stacked against same reference frame. Then do "split screen" type image while you still have linear data - prior to any stretch. This is easily done by selecting half of one stack and pasting it in the same place onto another stack.

Only when you have such image - do the stretch to show background noise. This way you are guaranteed to have same stretch on both images - as you'll be doing stretch on single image consisting out of two halves of different stacks.

[Lee_P]

15 minutes ago, vlaiv said:

Can you post just couple of linear stacks without stretching them (as fits please as I don't have PI and can't open xifs)?

Visual feedback on how much noise there is - depends on stretch level. I think that you used PI automatic stretch - and that one works on basis of noise floor - this is why it stretched each sub so that noise in background is the same.

You can see that by size of stars in your animations.

there is much more "light halo" around left star then on the right - but they are the same star (you can see this in your animated gifs - just watch any single star and see it "bloat" as total exposure increases).

In order to measure noise accurately - take two stacks still linear and select piece of background not containing any signal - no stars, no objects, mostly empty and do stats on each image in that region.

Compare pixel standard deviation values between images.

If you want to visually asses difference between two images - make sure they are normalized (one with respect to another) - or that they are stacked against same reference frame. Then do "split screen" type image while you still have linear data - prior to any stretch. This is easily done by selecting half of one stack and pasting it in the same place onto another stack.

Only when you have such image - do the stretch to show background noise. This way you are guaranteed to have same stretch on both images - as you'll be doing stretch on single image consisting out of two halves of different stacks.

Thanks vlaiv, I'm going to read all that carefully -- for now, attached are four FITS stacks 😃

90_mins_darks_and_flats.fit

105_mins_darks_and_flats.fit

120_mins_darks_and_flats.fit

135_mins_darks_and_flats.fit

[Lee_P]

2 hours ago, scotty38 said:

I cannot really add much apart from to say I have noticed ABE introduces noise similar to what you see here and, as yet, I've not been able to work out what settings I need to change or if it's my data. APP can make a better job of the stacking using its defaults. Having said that I know there are enough folk getting great results out of PI that I have basically accepted it's me/my data at fault.

I can see the screen shots but are you using WBPP and have you looked at Adam Block's videos for V2 of the script as there may be info in there that could help - I've not watched them all yet btw....

 

Edit just to be clear and what I mean re ABE adding noise is that if I use STF on a stacked image and then apply ABE the ABE version has noise whereas the STF image does not.

Interesting, I thought that ABE / DBE appears to introduce noise because they're removing gradients (e.g. from light pollution) that are essentially masking the underlying noise. If that makes sense? I could be wrong! 

I am indeed making my way through Adam Block's WBPP2.0 videos -- it's those that prompted me to make the tests including FlatDarks and CosmeticCorrection!

[Lee_P]

2 hours ago, Seelive said:

Last year I made some (rough) measurements of the effect of different stacking methods (I use DSS). I generally try to use at least 80 lights (180s each, so 4 hrs total).  I found that using Average stacking the resulting noise closely followed the 1 / ROOT(N) formula and similarly for Kappa-Sigma stacking, but only up to about 30 lights. After that the resulting noise leveled off and occasionally appeared to increase slightly.  These days I tend to Kappa-Sigma stack the lights in seperate batches of 20 to 30 and then average stack the results.  It seems to give me the same resulting noise as average stacking does but without the satellite trails and hot pixels.

Ok, interesting. Could be worth me investigating. Thanks!

[Lee_P]

1 hour ago, Laurin Dave said:

Have you measured the different stacks in Subframe Selector? It looks to me as if the longer integration time stacks are getting harder stretches, maybe put the same stretch on each stack and then compare, also what's happening to M81 and M82 are they gettng better?

Oh, you're clever 😎

Looks like the noise level is going down?

Here's the noise level over 24 hours of data:

 

So thanks to this and @vlaiv's insight, I think I understand that the noise level *is* going down, it just doesn't appear to in my GIFs because the process I used was automatically stretching it. BUT! I still don't understand why the noise pattern is the same in every image. Is that because of the automatic stretching?

 

 

To answer your other question, here's M81 over 24 hours:

 

And a version of the same data adjusted in Photoshop to keep the galaxy's brightness consistent:

[vlaiv]

50 minutes ago, Lee_P said:

Thanks vlaiv, I'm going to read all that carefully -- for now, attached are four FITS stacks 

Well, there won't be much visual difference between 90min and 135min. That is only 50% increase in number of frames or about 22.47% increase in SNR. That is on edge of perception.

However, it can be seen with a bit of "trickery". Human brain is very good at recognizing patterns - and if you give it two patterns side by side - it will spot similarity and differences rather easily.

here is animated git that I made from 90minute and 135minute fits - green channel, linearly stretched and employed "split screen" trick.

Right part of the frame does not change - it is 135minute fits. Left side alters between 90 and 135 minute fits.

It is obvious that 135minute looks a bit smoother - smaller grain and more uniform than 90 minute data. Even with such a small increase in data - difference can be seen if you know where to look for it.

54 minutes ago, Lee_P said:

Interesting, I thought that ABE / DBE appears to introduce noise because they're removing gradients (e.g. from light pollution) that are essentially masking the underlying noise. If that makes sense? I could be wrong! 

No. Background extraction will remove very uniform component of background signal and leave background noise in place.

Here is analogy to help you understand better. Imagine that you have mountain trail full of rocks that you walk on. It's bumpy ride. That is noise. Background level is just difference between that path being on Mount Everest versus your local hill. Bumps in that trail won't change if you lower the mountain level.

Last and most important thing is - although size of bumps in the path does not change - their relative size to the mountain does change. Average size of rocks of 10cm on path compared to Mount Everest being 8848 or whatever meters - makes that almost one in 100,000, but if you "wipe" the mountain and leave local hill that is 100meters high - now rocks are 1 in 1000.

This makes no difference to the image - it just makes difference to processing parameters - where you put your black point. Depending on that - you'll get relative noise level - same image can appear quite noisy or very smooth - depending on how you process it. Here is 135 minute data in two stretch versions:

Same image - different background.

One of very important skills to learn in image processing is to push the data only to the point it allows to be pushed. It is SNR that is important. In my view - it is better to keep the noise at bay by sacrificing signal part - maybe you won't be able to show those faint tidal tails nicely or those outer parts of the galaxy will be too dark to your liking - but image will be smooth and with very little noise.

Not being able to show faint detail does not mean that you did not push your data enough - it means that you did not spend enough time imaging.

[Lee_P]

3 hours ago, vlaiv said:

Well, there won't be much visual difference between 90min and 135min. That is only 50% increase in number of frames or about 22.47% increase in SNR. That is on edge of perception.

However, it can be seen with a bit of "trickery". Human brain is very good at recognizing patterns - and if you give it two patterns side by side - it will spot similarity and differences rather easily.

here is animated git that I made from 90minute and 135minute fits - green channel, linearly stretched and employed "split screen" trick.

Right part of the frame does not change - it is 135minute fits. Left side alters between 90 and 135 minute fits.

It is obvious that 135minute looks a bit smoother - smaller grain and more uniform than 90 minute data. Even with such a small increase in data - difference can be seen if you know where to look for it.

No. Background extraction will remove very uniform component of background signal and leave background noise in place.

Here is analogy to help you understand better. Imagine that you have mountain trail full of rocks that you walk on. It's bumpy ride. That is noise. Background level is just difference between that path being on Mount Everest versus your local hill. Bumps in that trail won't change if you lower the mountain level.

Last and most important thing is - although size of bumps in the path does not change - their relative size to the mountain does change. Average size of rocks of 10cm on path compared to Mount Everest being 8848 or whatever meters - makes that almost one in 100,000, but if you "wipe" the mountain and leave local hill that is 100meters high - now rocks are 1 in 1000.

This makes no difference to the image - it just makes difference to processing parameters - where you put your black point. Depending on that - you'll get relative noise level - same image can appear quite noisy or very smooth - depending on how you process it. Here is 135 minute data in two stretch versions:

Same image - different background.

One of very important skills to learn in image processing is to push the data only to the point it allows to be pushed. It is SNR that is important. In my view - it is better to keep the noise at bay by sacrificing signal part - maybe you won't be able to show those faint tidal tails nicely or those outer parts of the galaxy will be too dark to your liking - but image will be smooth and with very little noise.

Not being able to show faint detail does not mean that you did not push your data enough - it means that you did not spend enough time imaging.

Thanks vlaiv, this is pure gold. I think I'm getting my head around it... 

I tried reproducing your example. I find it useful, but did I mess up the "linear stretch" aspect? If you could give me a pointer on how to do that in Photoshop, maybe I could make a version 2.

And here's a graph of noise against integration time.

Thanks again!

-Lee

[vlaiv]

1 minute ago, Lee_P said:

Thanks vlaiv, this is pure gold. I think I'm getting my head around it... 

I tried reproducing your example. I find it useful, but did I mess up the "linear stretch" aspect? If you could give me a pointer on how to do that in Photoshop, maybe I could make a version 2.

First thing you need to do is normalize frames.

They have to have the same background level and same star intensity. I think that PI should have this feature - but since I don't use PI - I have no idea how it's performed.

After that - when intensities are normalized - linear stretch is simple - you do same levels without messing with middle slider - just top and bottom - on all frames and you should get same linear stretch.

In fact - when frames are normalized - you don't have to do linear stretch - as long as you perform same levels / curves (can save that as preset) - you should get compatible result.

[Lee_P]

1 hour ago, vlaiv said:

First thing you need to do is normalize frames.

They have to have the same background level and same star intensity. I think that PI should have this feature - but since I don't use PI - I have no idea how it's performed.

After that - when intensities are normalized - linear stretch is simple - you do same levels without messing with middle slider - just top and bottom - on all frames and you should get same linear stretch.

In fact - when frames are normalized - you don't have to do linear stretch - as long as you perform same levels / curves (can save that as preset) - you should get compatible result.

I've been trying for the last hour but just can't crack the normalisation stage. I'm sure it's simple, so I need to find some tutorials. I'll pause this for now, but thanks for all your help!

 

Edited May 13, 2021 by Lee_P

[vlaiv]

1 minute ago, Lee_P said:

I've been trying for the last hour but just can't crack the normalisation stage. I'm sure it's simple, so I need to find some tutorials. I'll pause this for now, but thanks for all your help!

As simple fix - you can just "wipe" the background with simple process.

Using pixel math just subtract median value of each stack from it.

After doing that - all subs should have 0 median value and backgrounds should be normalized. There will still be some intensity mismatch due to atmospheric absorption on different altitudes of object during recording - but that is minor thing - you should get usable results from just setting median value across stacks to same value.

[Lee_P]

9 hours ago, vlaiv said:

As simple fix - you can just "wipe" the background with simple process.

Using pixel math just subtract median value of each stack from it.

After doing that - all subs should have 0 median value and backgrounds should be normalized. There will still be some intensity mismatch due to atmospheric absorption on different altitudes of object during recording - but that is minor thing - you should get usable results from just setting median value across stacks to same value.

Ok, thanks, this makes sense to me. I tried it and something is amiss, as after my PixelMath subtraction, the median value isn't 0. Could someone versed in PixelMath point out where I'm going wrong? 

 

Before PixelMath. Median value = 29.

 

My PixelMath expression to remove the median value:

 

After subtraction, the median value is 4. Shouldn't it be 0?

 

Further proof that I've got it wrong. The left half of the image is 2 hours of integration, the right half is 24 hours. These are after my evidently suspect process of subtracting median values!

 

[vlaiv]

1 hour ago, Lee_P said:

Could someone versed in PixelMath point out where I'm going wrong? 

It looks like you are working with 16 bit images.

Try working with 32bit floating point images when doing all of that. I think it will solve the problem but not 100% sure - in any case 32bit precision is much better.

[Lee_P]

1 hour ago, vlaiv said:

It looks like you are working with 16 bit images.

Try working with 32bit floating point images when doing all of that. I think it will solve the problem but not 100% sure - in any case 32bit precision is much better.

It looks like the images are 32bit, but the statistics panel only has the option to display info for up to 16bit 🥴

[vlaiv]

1 minute ago, Lee_P said:

It looks like the images are 32bit, but the statistics panel only has the option to display info for up to 16bit 

In that case - I have no idea (don't own/use PI).

I can do it for you however, if you wish. I'm using ImageJ and it's really easy for me to do it. Just attach fits that you want normalized (like you did with 90 and 135 minute exposures) and I'll be happy to do it for you.

Alternatively - if you want to install ImageJ / Fiji - I'll be happy to show you how to do it in that software.

[AstroMuni]

18 hours ago, Lee_P said:

To answer your other question, here's M81 over 24 hours:

There is definitely something not quite right. I am no expert in this and I got this literally as my first image and using around 500secs worth of data with no darks or flats. So comparing mine and 24hrs of yours I would have said you should have got a much clearer image.

 

[AstroNebulee]

Have you tried stacking the subs with the darks just to see if there's any improvement, as you say the background noise should smooth out with more light frames taken. I know with my sw 72ed AzGti Canon 600D I don't take any darks atall now, just flats and bias as I find the darks just add more noise, maybe its just my set up but quite a few others not take darks. Here's a stack of a few of mine only 50 mins-1hr total subs. Not award winners by any means atall. 

[Lee_P]

41 minutes ago, AstroMuni said:

There is definitely something not quite right. I am no expert in this and I got this literally as my first image and using around 500secs worth of data with no darks or flats. So comparing mine and 24hrs of yours I would have said you should have got a much clearer image.

 

That's a great picture, especially for such a short integration time. The images in my test just had some very basic edits done, to allow them to be compared fairly. This is a fully edited version with 24 hours of integration time. Looks a bit better than my simple tests:

What level of light pollution do you have? I'm going to guess you're imaging from some nice dark skies? I'm in a city centre, which puts me at a big disadvantage with broadband targets like M81.

[Lee_P]

46 minutes ago, AstroNebulee said:

Have you tried stacking the subs with the darks just to see if there's any improvement, as you say the background noise should smooth out with more light frames taken. I know with my sw 72ed AzGti Canon 600D I don't take any darks atall now, just flats and bias as I find the darks just add more noise, maybe its just my set up but quite a few others not take darks. Here's a stack of a few of mine only 50 mins-1hr total subs. Not award winners by any means atall. 

Nice pics! Back when I started with my camera I did some tests, and found that Darks + Flats gave me the best results. I might try again though, as I'm learning a lot about how to make proper comparisons.

[Lee_P]

1 hour ago, vlaiv said:

In that case - I have no idea (don't own/use PI).

I can do it for you however, if you wish. I'm using ImageJ and it's really easy for me to do it. Just attach fits that you want normalized (like you did with 90 and 135 minute exposures) and I'll be happy to do it for you.

Alternatively - if you want to install ImageJ / Fiji - I'll be happy to show you how to do it in that software.

Thanks, that's very kind. I'm here to learn, so I'll glady take you up on the offer of showing me how to do it in ImageJ. I've downloaded the software. If I fail then I'll take you up on your offer of just doing it for me 😂

[vlaiv]

7 minutes ago, Lee_P said:

Thanks, that's very kind. I'm here to learn, so I'll glady take you up on the offer of showing me how to do it in ImageJ. I've downloaded the software. If I fail then I'll take you up on your offer of just doing it for me 😂

Sure, it is rather easy to do. There are several ways, but I'll show you easiest one.

- Either drag&drop your fits onto ImageJ bar (I do it below toolbar - onto status bar - but it will show words drag & drop as soon as you drag fits anywhere on that window) or go File / Open ... and select your fits

- make sure you are working with 32bit fits - if not, convert it to 32 bit (just click on menu option to switch data type - it is Image / Type menu)

- If your fits is color one - you'll get "stack" (this is ImageJ terminology - not actual astronomy stacked image we call stack) of three separate images - each channel) - move bottom scroll bar onto middle one (being green) and then right click anywhere on the image, you'll get menu:

Marked are scroll bar and menu option - duplicate. Once you hit that - it will ask you if you want to duplicate "stack" (several slices) or just one slice:

Leave "Duplicate stack" unchecked and this will create new image - copy of just green channel.

Close original stack of three channels as we will be working just on green for the time being.

- Hit Analyze / Measure menu option

You should get stats for your image and Median value should be there (it's not show on screen shot as it is far right but Mean value is shown).

If you by any chance don't have median in results - you need to customize your measurements. It is done via Analyze / Set measurements menu:

There are quite a few options that you can use - just check Median and make sure you have enough decimal places for precision.

Now you can repeat measurement via Analyze / Measure and you should have Median value included.

- Note down median value (in my case - it is 0.05964.... ) and do Process / Math / Subtract

- After that - you can do another measurement to confirm results:

Here we go - old and new value

- In the end - File / Save as / Fits ...

Repeat with other files ...

 

[AstroMuni]

3 hours ago, Lee_P said:

What level of light pollution do you have? I'm going to guess you're imaging from some nice dark skies?

Your image is beautiful btw. I am probably class 6 and I have street lights around 40ft away from where I park my scope.

Edited May 14, 2021 by AstroMuni