Walking noise

[bomberbaz]

I know the best cure for walking noise is not getting it in the first place. Dithering and good PA being the two methods as far as I am aware.

I also read that taking your subs over several nights would, when stacked even out the walking noise.

However, I took my subs over three nights of M108/97 and the walking noise was awful, however I did not dither. Currently taking some more subs with dithering to add to the others to quieten it down although I doubt it will cure it altogether.

Gmic had a useful tool to dampen it down (denoise smooth) and other tinkering in Gimp. 

Wondered if anyone else has other ideas for removing noise. 

[ONIKKINEN]

Walking noise is really difficult to get rid of if you let it build up to the point of it being a nuisance in an image. The best way to reduce it, you guessed it, not having it! If you keep getting more data and you "overwhelm" the earlier sessions with well dithered subs you will get rid of it.

But software solutions are all a pain, your best bet is to try and selectively desaturate the walking noise with masks first before trying to denoise it. Often there is a single colour that is most visible in the walking noise pattern that once removed takes much of the problem out with it (like purples for many DSLRs, maybe greens for some OSC cameras). Of course easiest to do after star removal with Starnet/StarXterminator so that the stars are not in the way. Then once desaturated you can try to denoise the grain but honestly its not something that can be trusted to do the job and you will likely end up with an image that looks like a bit of a painting if and when the denoise goes too far.

[900SL]

I manually dither every 5 frames when using my Fornax Lightrack and DSLR. You need at least 10 dithers. Longer exposures also help from what I can tell?

Bit of a PITA but it works. I have angular scales on RA and Dec, I release each axis and reset. I use live view at the same time to get a chosen star back in approximately the same place. Means you have to babysit but it does a good job

On my go-to mount I just dither. Depending on exposure every 1 to 3 frames. Never been able to remove walking noise without significant loss of detail 

[Stuart1971]

I have never dithered and never suffered with walking noise, and I use a modern QHY268m and also the colour version, many people don’t use darks with these cameras, but I do as it cures any issues that I would get due to walking noise…so are you using a good stack of darks, dark flats and flats….??

[bomberbaz]

6 hours ago, Stuart1971 said:

I have never dithered and never suffered with walking noise, and I use a modern QHY268m and also the colour version, many people don’t use darks with these cameras, but I do as it cures any issues that I would get due to walking noise…so are you using a good stack of darks, dark flats and flats….??

I am using the zwo 183mc pro camera and yes, I have a a library of D, F and DF's. They are recent too. 

I restacked some data that had shown bad walking noise using DSS but this time I activated darks optimisation which made a noticeable difference. The noise was still there but definately improved upon the previous version.

I now have some additional data which has been dithered but not had chance to add to the original data.

I also noticed after posting this there is also a dither tickbox in Siril - background extraction. Not sure how or if it works but it's something else to try and improve things. 

Edited February 16, 2023 by bomberbaz

[bomberbaz]

17 hours ago, ONIKKINEN said:

Walking noise is really difficult to get rid of if you let it build up to the point of it being a nuisance in an image. The best way to reduce it, you guessed it, not having it! If you keep getting more data and you "overwhelm" the earlier sessions with well dithered subs you will get rid of it.

But software solutions are all a pain, your best bet is to try and selectively desaturate the walking noise with masks first before trying to denoise it. Often there is a single colour that is most visible in the walking noise pattern that once removed takes much of the problem out with it (like purples for many DSLRs, maybe greens for some OSC cameras). Of course easiest to do after star removal with Starnet/StarXterminator so that the stars are not in the way. Then once desaturated you can try to denoise the grain but honestly its not something that can be trusted to do the job and you will likely end up with an image that looks like a bit of a painting if and when the denoise goes too far.

That is precisely what is occuring. I am ending up using so much process to remove walking noise and severe gradiant that the end results are losing the natural look. 

Very frustrating but I am stuck with my garden as I cannot go elsewhere that is darker to gather data for other reasons. 

[ONIKKINEN]

Not familiar enough with the 183 to say if its something that can do without dithering, but as mentioned above the newest very clean IMX571/533 cameras can work without dithering, provided that the polar alignment is competently done, darks are well matched (and maybe that there is no significant cone error).

The Siril background tool dither option is there to dither the background samplers in case you end up with a posterized result with obvious harsh transitions between areas of different illumination. Cant recall if i have ever needed it, but safe to say it does not cure walking noise nor is it intended to do so.

On the topic of the background tool, make sure you have cropped the incomplete edges, dont place too many samplers and that none of the samplers are on stars or nebulosity. Put the preview mode to Histogram and negative to get the best view on faint stuff. Dont trust the automatic sampler placement!

[pipnina]

11 minutes ago, ONIKKINEN said:

Not familiar enough with the 183 to say if its something that can do without dithering, but as mentioned above the newest very clean IMX571/533 cameras can work without dithering, provided that the polar alignment is competently done, darks are well matched (and maybe that there is no significant cone error).

For my 571, the hot pixels are there but get removed by the default settings in pixinsight. I do dither though on every 3rd sub.

I also found that for my much noisier Nikon D3200 which had super bright hot pixels, that the hot pixels would overpower the signal even when dithering and the hot pixels would trail all over my images. However the ultimate savior for me was the Sigma-clipping stacking method. It does a phenominal job of removing hot pixels if you dither your subs.

Here is the rejection map for a 1.5 hour stack of 17 second exposures taken on my google Pixel6 on the main camera, when using sigma clipping in pixinsight. My goodness!

And the final stack, once I finished processing... Those hot pixels are gone!

Ok, the image isn't exactly "clean", but we are dealing with a very tiny and poor quality sensor, plus optics that have a very complicated distortion and abberation pattern which makes it impossible to work with in a desirable way for astro. But if sigma clipping can make a phone produce an image this good and remove all that gunk from it, it should work just as beautifully on a much higher quality astrocam!

[bomberbaz]

59 minutes ago, ONIKKINEN said:

Not familiar enough with the 183 to say if its something that can do without dithering, but as mentioned above the newest very clean IMX571/533 cameras can work without dithering, provided that the polar alignment is competently done, darks are well matched (and maybe that there is no significant cone error).

The Siril background tool dither option is there to dither the background samplers in case you end up with a posterized result with obvious harsh transitions between areas of different illumination. Cant recall if i have ever needed it, but safe to say it does not cure walking noise nor is it intended to do so.

On the topic of the background tool, make sure you have cropped the incomplete edges, dont place too many samplers and that none of the samplers are on stars or nebulosity. Put the preview mode to Histogram and negative to get the best view on faint stuff. Dont trust the automatic sampler placement!

Thank you for that, some very useful information there. I am either not,  over or under doing certain parts within the underlined statement by the sounds of it.

I shall try again and see if I can further improve results. 

[Elp]

You MAY be able to address some of it with image subtraction, but I'd need to see an example of what you're dealing with.

[bomberbaz]

I have three images below, all from same data with one slight difference, dither during background extraction. I followed the advice below and got a much tidier image after background extraction.

On 16/02/2023 at 16:47, ONIKKINEN said:

Not familiar enough with the 183 to say if its something that can do without dithering, but as mentioned above the newest very clean IMX571/533 cameras can work without dithering, provided that the polar alignment is competently done, darks are well matched (and maybe that there is no significant cone error).

The Siril background tool dither option is there to dither the background samplers in case you end up with a posterized result with obvious harsh transitions between areas of different illumination. Cant recall if i have ever needed it, but safe to say it does not cure walking noise nor is it intended to do so.

On the topic of the background tool, make sure you have cropped the incomplete edges, dont place too many samplers and that none of the samplers are on stars or nebulosity. 

However as you can see below, the first image is a horrible looking thing and the second one much tidier. Difference being I used the dither during background extraction which evened things out nicely. Both top two have had siril auto-histogram stretch

Things are still very noisy though, walking noise still there in them but less prominent and gradiant removal has been far more successful than previous tries.

I tried saving without using the siril auto-histogram and then tinkering in gimp and that gives the bottom image.

I think I can get better but I am very satisfied in getting to this stage and shall continue to work at it. Very tired now though so will try again some more anotheer day.

Edited February 19, 2023 by bomberbaz

[Elp]

As a test, I took the top image and applied an image subtraction to it. Subtracting it 100% looks unnatural as by default an image needs to have noise in it to look "normal", so left it around 70% opacity (you can always use a noise filter to put back in much finer noise than that in the image originally). This is only around 2 minutes manual work, with a bit more blending work (and colour balancing) especially with the stars it can look more natural:

[bomberbaz]

On 22/02/2023 at 12:42, Elp said:

As a test, I took the top image and applied an image subtraction to it. Subtracting it 100% looks unnatural as by default an image needs to have noise in it to look "normal", so left it around 70% opacity (you can always use a noise filter to put back in much finer noise than that in the image originally). This is only around 2 minutes manual work, with a bit more blending work (and colour balancing) especially with the stars it can look more natural:

Thanks @Elp, that's quite impressive. What programme do you use for this and is there a walkthrough anywhere, I am all for learning new techniques.

[Elp]

Manual, PS. It should be possible in GIMP but I haven't worked out how to do it yet.

1. Copy your original image and paste into a new document,

2. Paint black with a soft outline brush anything you don't want to be altered during the subtraction, so in this case I painted out the galaxy and the nebula,

3. We now want to remove all the stars as leaving them in will effectively cancel them out during the subtraction. Goto select > colour range, select "highlights" in the drop down menu. It will select all your bright objects (stars we want here),

4. We now want to expand the star selections so they capture the halos around them. Goto select > modify > expand and insert a small figure around 4-10 pixels depending on your range of small and large stars, larger stars will mean larger pixel value but you don't want to go too large around your stars otherwise you'll end up with black circles in the image,

5. We now want to feather the star selection so it's a soft blend and not a harsh edge, goto Select > feather and input a value half of that used under (4),

6. Now just delete. All your stars should disappear and leave transparent pixels where they were, make sure your layer has right click "layer from background" applied prior, otherwise your deleted stars will be a background colour, not transparent. Tbh I think having black circles where the stars were doesn't matter but I haven't tried it,

7. Now the magic. In your original image file goto image > apply image. This allows you to do combinations to images with other images (it might be the same as having a layer above another and changing the blend operation but I haven't tried it). From the drop-down menu select your edited image filename (it doesn't even have to be saved) and the operation to "subtract". Magically everything will be flat as it's subtracted what was visible in your edited image, from the original. If you leave the opacity to 100% it will be completely flat with no detail, as mentioned it sometimes looks a bit unnatural so you can adjust this to make some of the defect still visible, or add in some noise afterward. You can also adjust the offset which I think alters the midtone level.

That's it basically. You can also do this on comet images to remove troublesome star trails on your comet stack.

With two more steps I also use this process to make synthetic flats when calibration hasn't flattened the field, or if I'm using OSC filters which have caused issues.

Every day's a school day, and I much prefer doing this stuff manually than relying on software trickery to do the work for you automatically as where's the fun in that?

[bomberbaz]

Further to all the above I have carried out the following procedures and I am far happier with what I have been able to achieve. 

first stacking in DSS. I have increased the number of calibration frames used and added 2x drizzle. Interesting to note that in doing this there was no sign of posterizing in the image prior to back ground extraction.

The only Auto/semi auto process used in Siril was background extraction and green noise removal.

In gimp nearly all other processing was manually done and I have to say, I felt more connected as it were to what I was doing.

I do love that gimp now has the workflow running down the left side to allow you to look back easily without constantly having to add and name layers, far more intuitive.

As always I know I am not there yet, it's a little overdone in some areas, notably the stars but happy to be making some kind of progress. The biggest plus is my stacking and processing appears to be in better control of the walking noise. (I will dither from now on)

 

Edited February 24, 2023 by bomberbaz