Mure denoise and cmos camera - help needed!

[MartinB]

PI has an uncanny capacity to make me feel really stupid!  I hear that Mure Denoise is a very good tool to be applied as a first processing procedure to an integrated image.  As usual I have to set some correct parameters in the boxes and that is the challenge.  I would be very grateful for help with what looks like a great tool.

I have a ZWO ASI 1600mm pro camera and have a calbibrated, aligned and combined image from 50 lights in Ha.  

I put 2 darks (300secs matching my exposure times) into the DarkBiasNoise estimator and this came up with an offset of 800.00 DN and Temporal noise 40.48 DN

moving on to the Mure Denoise script the combination count is easy - 50 (or is that too easy!), interpolation method I'm guessing at Lanczos-3 (I used auto and didn't know to check at the time).

Now for the gain.  I was using a gain setting of 150 which corresponds with an actual gain value of 0.9 e-/ADU but the unit in PI is e-/DN (I know DN is a unit of force but don't know whether that corresponds with e-/ADU)

Gaussian noise - from the noise estimator tool I have a value for "temporal noise" of 40.48 DN.  I don't know if this is the figure I should enter under gaussian noise but this is what I have entered

Offset - the noise estimator gives a value of 800 so this is what I have entered

Variance scale - I am leaving this at 1 since I didn't create a log of the integration.  The subs were very consistent so should be ok.

Cycle spin count - left at the default of 8

At some point a gaussian noise value of 1.3 got entered as a value but I'm not sure how this happened!

After running the script I get an image of the noise and my light image has an undo arrow showing that the process has been applied.  However, when I undo and redo the image doesn't change even when highly zoomed.  There is noise clearly visible but it hasn't been affected by the process.

I am sure I am making some very basic mistakes and would be grateful for guidance!

[wimvb]

AfaIk, DN and ADU are the same, a digital unit. Otherwise, I can't help, I've never come to grips with Mure denoise. 

[MartinB]

Thank you Wim, that is actually very helpful.  So DN (in this context) simply means digital number and corresponds with ADU.  Still not there with this but I have found a set of numbers to imput which seems to very effectively reduce the background noise whilst leaving signal intact and free of artefacts.  More study needed to fine tune but I am now refining the questions I need to ask!

[jimjam11]

28 minutes ago, MartinB said:

Thank you Wim, that is actually very helpful.  So DN (in this context) simply means digital number and corresponds with ADU.  Still not there with this but I have found a set of numbers to imput which seems to very effectively reduce the background noise whilst leaving signal intact and free of artefacts.  More study needed to fine tune but I am now refining the questions I need to ask!

What numbers did you land on? There is a tutorial on the ip4ap website which explains the usage of the tool, but when I followed the instructions I got the same resul as you; namely a nice noise image but the integrated file looked identical...

If you run image integration prior to loading the MureDenoise script you can then hit the 'load variance scale' which populates the values for combination count and variance scale.

I think offset should be 0; your integrated stack will be fully calibrated? Conversely the dark bias noise estimator script comes up with an offset because you feed it 2x uncalibrated darks?

 

[MartinB]

Thanks JimJam.  Yes, I fianlly worked out that the offset needed to be zero, more by trial and error than anything but I understand now.  I gather that you can get a good idea of detector gaussian noise by dividing read noise by detector gain (based on the ZWO graph for the camera).  This gives me a value of 1.9 however this only produced a very minor effect.  FlatSNRestimator reported a gain value of 0.054  which doesn't seem to make sense.  I'll have to double check but I'm pretty sure I was using an original flat with this tool rather than one which was dark calibrated.  The calculated detector gaussian noise figure from this value was 3.15.  This appears to have worked a like a charm.  The background is nicely smoothed with no impact on faint stars or faint wisps of nebulosity.  I can see nothing untoward going on in the background.  So I don't understand the values but I do understand the evidence of my eyes!

I really appreciate your help JimJam, thank you

[jimjam11]

I wonder if this could actually be a difference between DN and ADU because the ASI1600 has a 12bit ADC:

https://www.cloudynights.com/topic/573886-sub-exposure-tables-for-asi-1600-and-maybe-qhy163/?p=8068575

I was testing on a ha sub captured at gain 200, so 0.48 e/ADU. When I entered this I got the same result as you; no change. Decreasing this to 0.03 as measured by flatsnresttimator and I am starting to see results.

I then tested the gaussian noise value. A small value (measured/10) got minimal/no change. A big value (measured * 5) got a weirdly over-smoothed result.

 

Edited November 6, 2019 by jimjam11

[wimvb]

7 hours ago, jimjam11 said:

I wonder if this could actually be a difference between DN and ADU because the ASI1600 has a 12bit ADC:

https://www.cloudynights.com/topic/573886-sub-exposure-tables-for-asi-1600-and-maybe-qhy163/?p=8068575

I was testing on a ha sub captured at gain 200, so 0.48 e/ADU. When I entered this I got the same result as you; no change. Decreasing this to 0.03 as measured by flatsnresttimator and I am starting to see results.

I then tested the gaussian noise value. A small value (measured/10) got minimal/no change. A big value (measured * 5) got a weirdly over-smoothed result.

 

That actually makes sense, since with a 12 bit ADC (zwo cameras) the 12 bit output is stored in the high end of a 16 bit word; 1 ADU = 16 DN. (16 = 2^4 where 4 =16 - 12.) 

0.48 e/ADU = 0.48/16 e/DN = 0.03 e/DN

Good thing I wrote "afaIk" before, because I didn't then, but now I know farther. Always nice to learn new stuff. 

[MartinB]

You two are my friends for life!

[Jon2070]

On 07/11/2019 at 09:17, MartinB said:

You two are my friends for life!

Have you had any joy with Mure Martin? I also have the ASI1600MM and have had the same problem that Mure doesn't seem to do much, although many people hold it in high regard which is why I would like to use it. 

I tried dividing my Gain by 16 but the result was way over-smoothed.

Also as a side note the script loaded the metadata successfully upon first attempt including the Gain, Gaussian Noise and Offset, but I'm not able to repeat this. It will however load the combination count and interpolation method repeatedly.

[MartinB]

1 hour ago, Jon2070 said:

Have you had any joy with Mure Martin? I also have the ASI1600MM and have had the same problem that Mure doesn't seem to do much, although many people hold it in high regard which is why I would like to use it. 

I tried dividing my Gain by 16 but the result was way over-smoothed.

Also as a side note the script loaded the metadata successfully upon first attempt including the Gain, Gaussian Noise and Offset, but I'm not able to repeat this. It will however load the combination count and interpolation method repeatedly.

Yes... I think so.  I used in on a Veil Complex image  and it seemed to work nicely.  

I'm now working on data from Simeis 147 and the same settings shown above ie gain 0.056  and gaussian noise 3.15.  These figures were derived from the flatSNRestimator. I think the issue is as described by Wim above re working with 12 bit data.  

 

[Jon2070]

17 hours ago, MartinB said:

Yes... I think so.  I used in on a Veil Complex image  and it seemed to work nicely.  

I'm now working on data from Simeis 147 and the same settings shown above ie gain 0.056  and gaussian noise 3.15.  These figures were derived from the flatSNRestimator. I think the issue is as described by Wim above re working with 12 bit data.  

 

Thank you Martin, a very nice image by the way!

I have just used flatSNRestimator for my G139 O21 Ha flats to get the figures for my Ha data.

It gives me Gain 0.061 e-/DN and temporal noise of 715.11 DN.

If I leave the Gain as 0.061, but divide the temporal noise by 16 to get 44.69 and put these values into Mure I get what looks like a nice smooth result. If I divide the gain by 16 then I get this horrible granular result, likewise if I don't divide the temporal noise by 16.

So is that what you are doing? Leaving the gain given by flatSNRestimator as it is but dividing the temporal noise by 16 to give you the gaussian noise? Are temporal noise and gaussian noise one and the same, the difference being between 12 bit and 16 bit here?

I just left the Offset at 0, despite using an offset during capture, I'm not sure whether this is correct or not?

Also have you left variance scale and cycle spin-count the same?

Edited December 8, 2019 by Jon2070

[MartinB]

I seem to remember flatSNRestimator came up with a gaussian noise of 3.15 and I don't recall dividing by 16 but I'm 100% certain now!  The offset has been removed by calibration so it isn't the offset you have applied for the data capture.  It should be zero.  Yes, use the variance scale as shown.  You can adjust the cycle spin count to suit, the higher the number the better the accuracy and the longer the process.  8 seems to be a good default  I think the key is to get a result that looks good.  I have zoomed in a long way to check faint details aren't being blurred.  It might not be quite the PI way but I guess you can tweak the gain / noise values.

[Jon2070]

3 hours ago, MartinB said:

I seem to remember flatSNRestimator came up with a gaussian noise of 3.15 and I don't recall dividing by 16 but I'm 100% certain now!  The offset has been removed by calibration so it isn't the offset you have applied for the data capture.  It should be zero.  Yes, use the variance scale as shown.  You can adjust the cycle spin count to suit, the higher the number the better the accuracy and the longer the process.  8 seems to be a good default  I think the key is to get a result that looks good.  I have zoomed in a long way to check faint details aren't being blurred.  It might not be quite the PI way but I guess you can tweak the gain / noise values.

Thanks, I still don't fully understand offset, but that helps to know that calibration removes it.

I have attached a before (bottom image) and after, one with variance at 0.8 (top) and the other at 1.0 (middle), I have used a heavy crop as well.

Edited December 8, 2019 by Jon2070

[Jon2070]

An update which hopefully will prove useful, I have been in touch with Mike Schuster on the PixInsight forums throughout the day and he pointed me towards his new script MureDenoiseDetectorSettings (which initially was not showing up in PI even though I have the latest version and all of the repository updates and it had even downloaded but for reasons unknown was not visible in PI under MURE, but after adding it in manually it works great).

Using two uncalibrated flats and darks it gives the gain and gaussian noise that you need for MURE, no need to divide by 16, they work straight out of the box and provide a very pleasing result. So for the data above, the gain is correct and the Gaussian noise is 27.71DN.

[ampleamp]

missed that update Jon, thanks; 

[MartinB]

Brill, thanks Jon, will check that out.

[kirkster501]

The DarkBiasNoiseEstimator script will punch out the numbers if you feed it two darks at the same temp and length as the lights.  You plug the numbers it returns into Mure Denoise tool.  MD does not work on drizzle images.