Testing increasing integration times (OSC, Bortle 8)

[Lee_P]

19 minutes ago, ollypenrice said:

That's great. No question.  Personally I'd have another look at the core, though. I suspect that, with a dedicated stretch carefully blended in, you could resolve the central star as a round star with bright nebulosity around it. As your stack has increased, so has the saturated area around the progenitor star, but that can be prevented in processing. It's well controlled as it is but I bet you could perfect it.

Olly

I think you're right, and it's funny you should say that because I've been editing the IKO Iris Nebula data trying to do exactly that 😁

[ollypenrice]

2 hours ago, Lee_P said:

I think you're right, and it's funny you should say that because I've been editing the IKO Iris Nebula data trying to do exactly that 😁

With an LRGB image a good trick is to use only the RGB for the brightest parts. Often they are equivalent to short subs for this purpose and they'll carry more colour as well. I haven't processed this latest IKI set but that was how I did my own Iris and a good number of other images with bright parts needing control.

Olly

[Lee_P]

2 hours ago, ollypenrice said:

With an LRGB image a good trick is to use only the RGB for the brightest parts. Often they are equivalent to short subs for this purpose and they'll carry more colour as well. I haven't processed this latest IKI set but that was how I did my own Iris and a good number of other images with bright parts needing control.

Olly

Good tip, thanks for sharing 😁

[Lee_P]

On 01/05/2021 at 17:07, CloudMagnet said:

Thanks for putting this together, as you said looks like 10 hours onwards is the start of diminishing return. Definitely useful to see it in practice. Have you done anything similar for other types of targets like galaxies or emission nebula to see if the same integration time holds true?

Would be interesting to look at how different surface brightness could affect the "best" integration time as well.

As CloudMagnet suggested, I've repeated the experiment with an extreme crop of M81.

 

Looks like the same result to me: the first 10 hours are the most valuable. Data after that are still useful, but the impact certainly isn't as dramatic.

I'll repeat with L-eXtreme data on a nebula once galaxy season has passed!

[CloudMagnet]

48 minutes ago, Lee_P said:

As CloudMagnet suggested, I've repeated the experiment with an extreme crop of M81.

Looks like the same result to me: the first 10 hours are the most valuable. Data after that are still useful, but the impact certainly isn't as dramatic.

I'll repeat with L-eXtreme data on a nebula once galaxy season has passed!

Hmmm, my observation would be that there doesnt seem to be much change in background noise from 4-6 hours onwards. I thought that this would slowly improve but hits a limit much earlier than I would expect.

Thank you for putting this together. Would it be possible to have the galaxy the same brightness in each picture as well? Makes for a nice comparison for the finer detail in the core/dust lanes

[Lee_P]

1 hour ago, CloudMagnet said:

Hmmm, my observation would be that there doesnt seem to be much change in background noise from 4-6 hours onwards. I thought that this would slowly improve but hits a limit much earlier than I would expect.

I think you're right actually. I'm confused, surely it should get smoother with more data? Even the pattern of noise looks the same. Have I messed something up in pre-processing? 🤔

 

 

 

1 hour ago, CloudMagnet said:

Thank you for putting this together. Would it be possible to have the galaxy the same brightness in each picture as well? Makes for a nice comparison for the finer detail in the core/dust lanes

I'd be happy to do this, but you'll have to give me some tips on how! In the way I've been doing basic edits on these tests, the galaxy naturally gets brighter.

[CloudMagnet]

Were the images dithered? I would expect that would even out any noise with higher integration time (fewer frames occupying the same exact space).

In terms of making the images the same brightness, either by eye would be ok or using the eyedropper tool in Photoshop that will give you a brightness reading of a pixel sample- just put this in the same place on every image.

[Lee_P]

1 minute ago, CloudMagnet said:

Were the images dithered? I would expect that would even out any noise with higher integration time (fewer frames occupying the same exact space).

In terms of making the images the same brightness, either by eye would be ok or using the eyedropper tool in Photoshop that will give you a brightness reading of a pixel sample- just put this in the same place on every image.

Yep, they were dithered.

I'll have a crack at making the galaxies the same brightness.

[Lee_P]

26 minutes ago, CloudMagnet said:

In terms of making the images the same brightness, either by eye would be ok or using the eyedropper tool in Photoshop that will give you a brightness reading of a pixel sample- just put this in the same place on every image.

I went down the eyedropper route. In for a penny, in for a pound!

 

 

Whaddya make of that?

[CloudMagnet]

Very interesting! Huge difference from 2-4-6 hours then a steady drop in difference as we would expect. Think that the practical limit is still around that 10hr mark as before.

Main difference I can see from 10 hours onward is just around the very faint outer arms near the top left. Some aren't even visible in the early pictures but do show up later on. Might be helped by the steady darkening of the background-I assume this is the noise cancelling out with longer integration.

Well done putting that together, exactly what I meant by keeping the brightness of the galaxy the same for a better comparision

[Lee_P]

12 minutes ago, CloudMagnet said:

Very interesting! Huge difference from 2-4-6 hours then a steady drop in difference as we would expect. Think that the practical limit is still around that 10hr mark as before.

Main difference I can see from 10 hours onward is just around the very faint outer arms near the top left. Some aren't even visible in the early pictures but do show up later on. Might be helped by the steady darkening of the background-I assume this is the noise cancelling out with longer integration.

Well done putting that together, exactly what I meant by keeping the brightness of the galaxy the same for a better comparision

Having consistency in brightness was a good shout, it does make it easier to see the differences. I've tried to do the same with the original Iris Nebula data.

 

Any thoughts on why the noise tests a few posts above show no real improvements after four hours? That's got me scratching my head.

[CloudMagnet]

2 minutes ago, Lee_P said:

Any thoughts on why the noise tests a few posts above show no real improvements after four hours? That's got me scratching my head.

Might just be down to the Bortle 8 skies. You are just reaching the maximum limit of what that setup in that location is capable of. Someone in Bortle 2/3 would likely hit that signal to noise ratio in a fraction of the time. I'm sure someone with more knowledge can work out mathematically, but that would be my assumption. I have deliberately avoided the Iris nebula from my Bortle 6/7 skies as I’m sure I read that somewhere that it would take me around 40 hours to reach the same SNR as someone taking 4 hours in a proper dark sky zone.

You must just be tending towards the natural minimum noise possible and no more dithering can remove noise beyond that limit. Only easy option to improve further would be to run the camera at a cooler temperature. Anything beyond that is tweaking at the edges. Such as using more dark frames or using dark flats (if not already), looking at gain settings and exposure to minimise read noise.

[Lee_P]

4 minutes ago, CloudMagnet said:

Might just be down to the Bortle 8 skies. You are just reaching the maximum limit of what that setup in that location is capable of. Someone in Bortle 2/3 would likely hit that signal to noise ratio in a fraction of the time. I'm sure someone with more knowledge can work out mathematically, but that would be my assumption. I have deliberately avoided the Iris nebula from my Bortle 6/7 skies as I’m sure I read that somewhere that it would take me around 40 hours to reach the same SNR as someone taking 4 hours in a proper dark sky zone.

You must just be tending towards the natural minimum noise possible and no more dithering can remove noise beyond that limit. Only easy option to improve further would be to run the camera at a cooler temperature. Anything beyond that is tweaking at the edges. Such as using more dark frames or using dark flats (if not already), looking at gain settings and exposure to minimise read noise.

Ah, I calibrated them with Flats and Darks, but not Dark Flats. Something else to read up about!

[CloudMagnet]

Good thing about this hobby is you never stop learning -there is always more you can do

[Ouroboros]

11 hours ago, Lee_P said:

Yep, they were dithered.

I'll have a crack at making the galaxies the same brightness.

As you say the pattern in the noise does seem to remain the same. By how much did you dither?  Should we conclude by not enough?  Or are these variations in light and dark real?  Possibly due to more distant galaxies and light absorbing dust clouds. 

[Lee_P]

On 04/05/2021 at 08:15, Ouroboros said:

As you say the pattern in the noise does seem to remain the same. By how much did you dither?  Should we conclude by not enough?  Or are these variations in light and dark real?  Possibly due to more distant galaxies and light absorbing dust clouds. 

I'm using an ASIAIR PRO, with the dither options being 1, 2, 3, 5, or 10 pixels. I've been using 3. Sounds like another test is called for, with a higher number.

The variations being real is something I hadn't even considered 🤯

[Lee_P]

@Ouroboros @CloudMagnet I've made a few more noise tests: increasing the dithering, and trying Dark Flats. Nothing seems to make an appreciable difference in the end. So either I'm getting something wrong in the image acquisition / pre-processing stages, or my Bortle 8 skies are forever cursed to have this level of background noise. I've been staring at these images for hours though, so maybe some fresh eyes can spot something I'm missing! 🥴

[Ouroboros]

@Lee_P Mmmmmm ... The background noise doesn’t get much better with increasing dither does it?  I doubt the graininess is down to your Bortle 8  skies though.  Light pollution wouldn’t have that much structural detail in it would it?   It would just be a slowly varying background fog across the skies surely. 

Those variations in background structure are consistent too.  They get more noticeable the more frames you average. So they’re either inherent variations in the sensitivity of your image sensor or they are in deep space aren’t they? 

I wonder whether 10 pixels dither on your image camera is enough. Or is that 10 pixels on your guide cam, which I think works out at 30+ pixels on your image camera (looking at your system specs)?

I also wonder how you’re stretching the images.  If you’re using automatic stretch is this somehow making the background noise appearing to not get much better as you average more frames?   

Edited May 7, 2021 by Ouroboros

[CloudMagnet]

Ok, just to check what exposure times are you using? It could be down to read noise creating a minimum floor that dithering cannot help with. This would be worse with multiple short exposures (as you are reading more often)

Looking at the specs on your camera, there is a step change downward in read noise from using >100 gain. What gain are you using?

Very interesting to see that dark flats are making no visible difference. Thanks again for putting this together

[Lee_P]

37 minutes ago, CloudMagnet said:

Ok, just to check what exposure times are you using? It could be down to read noise creating a minimum floor that dithering cannot help with. This would be worse with multiple short exposures (as you are reading more often)

Looking at the specs on your camera, there is a step change downward in read noise from using >100 gain. What gain are you using?

Very interesting to see that dark flats are making no visible difference. Thanks again for putting this together

Lights are integrations of two-minute exposures. Flats taken using my ASIAIR PRO's auto flat exposure, which came out as 2.7s. Dark Flats 2.7s too, to match. Gain 100.

[Lee_P]

1 hour ago, Ouroboros said:

@Lee_P Mmmmmm ... The background noise doesn’t get much better with increasing dither does it?  I doubt the graininess is down to your Bortle 8  skies though.  Light pollution wouldn’t have that much structural detail in it would it?   It would just be a slowly varying background fog across the skies surely. 

Those variations in background structure are consistent too.  They get more noticeable the more frames you average. So they’re either inherent variations in the sensitivity of your image sensor or they are in deep space aren’t they? 

I wonder whether 10 pixels dither on your image camera is enough. Or is that 10 pixels on your guide cam, which I think works out at 30+ pixels on your image camera (looking at your system specs)?

I also wonder how you’re stretching the images.  If you’re using automatic stretch is this somehow making the background noise appearing to not get much better as you average more frames?   

I reckon it's 10 pixels on the guidecam. That's the highest possible setting, so I'd be surprised if it weren't enough.

Re: stretching, I used EZ Stretch in Pixinsight. 

🤔🤔🤔

[Lee_P]

@CloudMagnet @Ouroboros @ Anyone Else That's Interested   
Could we try ruling out user error? Seems that me having messed up the pre-processing or integrations is a possible cause for the noise pattern. I've uploaded the files from the latest test here: https://drive.google.com/drive/folders/1simtZOsVAgwuqIT44uMkJ_a_jzGvhE3n?usp=sharing Any chance you could try some integrations and see if you get the same effect? No worries if that's too much hassle for you!

FYI the Lights folder contains the 10px drizzled data. The file that begins "REF" is what I used for my reference file. If you wanted to reproduce the exact field of view of my tests, the location ("Region of Interest") in PixInsight) is:
Left: 2973
Top: 1756
Width: 400
Height: 400

I then Rescaled to 1000px x 1000px to make it easier to see.

What are the odds that we'll see that same noise pattern again? 🤔

[Ouroboros]

@Lee_P Oooooh! Might take me a while. I’ll be travelling tomorrow and getting ready today to get away, and I’m slow at processing at the best of times!   If no one else has had a go within the next few days I’ll see what I come up with. 

[CloudMagnet]

On 08/05/2021 at 07:37, Lee_P said:

@CloudMagnet @Ouroboros @ Anyone Else That's Interested   
Could we try ruling out user error? Seems that me having messed up the pre-processing or integrations is a possible cause for the noise pattern. I've uploaded the files from the latest test here: https://drive.google.com/drive/folders/1simtZOsVAgwuqIT44uMkJ_a_jzGvhE3n?usp=sharing Any chance you could try some integrations and see if you get the same effect? No worries if that's too much hassle for you!

FYI the Lights folder contains the 10px drizzled data. The file that begins "REF" is what I used for my reference file. If you wanted to reproduce the exact field of view of my tests, the location ("Region of Interest") in PixInsight) is:
Left: 2973
Top: 1756
Width: 400
Height: 400

I then Rescaled to 1000px x 1000px to make it easier to see.

What are the odds that we'll see that same noise pattern again? 🤔

Seem to be having a problem downloading them, think its my computer doesnt like it

I dont think it would be user error anyway, what program and settings are you using for stacking?

[Lee_P]

26 minutes ago, CloudMagnet said:

Seem to be having a problem downloading them, think its my computer doesnt like it

I dont think it would be user error anyway, what program and settings are you using for stacking?

You might need to be signed into a Google account? https://drive.google.com/drive/folders/1simtZOsVAgwuqIT44uMkJ_a_jzGvhE3n?usp=sharing

I used PixInsight. I'm a relative newbie, so wouldn't be surprised if I were doing something dumb that's causing the effect. Here are my settings:

 

Edited May 9, 2021 by Lee_P