PI integration going nuts

[Datalord]

I'm processing data from multiple nights with about 50*180s and 15*600s data. I register, normalize all the usual in PI. Then I stack. This all worked fine for LRG. In blue, though, the situation below happens and I have no clue why. I have redone the entire process one time over just for blue and still this mayhem occurs. 

Any ideas?

[wimvb]

Are you stacking the 180 s and 300 s subs together in one go? The normalization may be messed up. If you have enough and about the same number of subs of each exposure, you can also integrate seperately and then integrate the two masters without pixel rejection.

[Datalord]

Yeah, integrating in one go, but calibrating separately.

Maybe I should try integrating the two separately, as you suggest.

[wimvb]

As an alternative, don't use normalization, but integrate all subs in one go. Select noise evaluation as the weight factor.

Personally, I'd try both methods and keep the image that looks best.

[Datalord]

Noise evaluation sorted the problem. Very odd that only blue decided to give me grief with this.

EDIT: With a closer look, the non-normalized blue integration is much noisier than the others. I don't think I can mix those without problems. I really wish I could comprehend why the blue is giving me this grief...

[Datalord]

Final update. I found the magic, lesson learned. I changed Linear Fit high from 2.5 to 5.0. Solved the problem after a few hours of headache.

[Barry-Wilson]

On 13/03/2019 at 01:16, Datalord said:

Final update. I found the magic, lesson learned. I changed Linear Fit high from 2.5 to 5.0. Solved the problem after a few hours of headache.

Why are you using Linear Fit?

Have you tried processing without this step?

[Datalord]

22 minutes ago, Barry-Wilson said:

Why are you using Linear Fit?

Have you tried processing without this step?

It's the algorithm. I use it mostly because Lightvortex tutorials recommend it with more than 15 images.

[wimvb]

22 minutes ago, Barry-Wilson said:

Why are you using Linear Fit?

Have you tried processing without this step?

I think the OP is using linerar fit pixel rejection, not the linear fit calibration routine.

Confusing names.

[Datalord]

Incidentally, I also use the routine. Should I not?

[Barry-Wilson]

Ah, understood.

Linear Fit as rejection algorithm in Image Integration: try using Winsorised Sigma Clipping instead.

Linear Fit as adaption tool between channels: I’ve been using PixInsight for many years and the two times I used this ended in awful results. I have never found a need to use this tool at all.

Barry

[coatesg]

If you have >15 images, I'd def use Windsorized Sigma clip - should result in a lower noise in the final integration.

[Big Jim Slade]

33 minutes ago, coatesg said:

If you have >15 images, I'd def use Windsorized Sigma clip - should result in a lower noise in the final integration.

My recollection is that the docs suggest WSC for > approx. 10 images, and Linear Fit for more than 15 images. My policy is to run both on any given set of data and use what works best - it will vary I think. The best bit of PI wisdom I've learned is that there is no one single recipe that works on everything.  

Blue will always be the noisiest channel, there's just less of it. As far as integrating different length subs goes, a better approach would be to move to using the Subframe Selector to do the weighting but this is more than a little fiddly! It's my default workflow now, but it took a while to get comfortable with it.

[coatesg]

It does depend, but of the tests I've run with this, especially for lower SNR data, WSC comes out better than the others.  For larger data sets, the differences are marginal, and may go to other way.  

The good thing about PI, is that it offers the tools to try it out and measure/compare the results and use the most suitable one. Of course, the rejection parameters also make a difference! 

[Datalord]

32 minutes ago, Big Jim Slade said:

Subframe Selector

I'm using that as well, but it doesn't have anything to do with the Pixel rejection algorithm, does it? 

[Hallingskies]

Just loving the jargon-fest here...?????

[Big Jim Slade]

5 minutes ago, Datalord said:

I'm using that as well, but it doesn't have anything to do with the Pixel rejection algorithm, does it? 

Well, there's an interaction with all the steps here. If the weighting/noise evaluation of the images isn't working (for example giving too much weight to the shorter subs), then your pixel rejection has a much harder job to do.  Fundamentally, it's the pixel rejection algorithm's job to get rid of statistical outliers, so if the weights favour the longer, lower noise subs, you should have less trouble tuning the pixel rejection. More signal, less noise.

[Big Jim Slade]

10 minutes ago, Hallingskies said:

Just loving the jargon-fest here...?????

Hehe, indeed. The problem with PI is you can start out trying to get a nice piccy and end up thumbing through tomes like "Image Processing and Data Analysis - The Mutliscale Approach" looking for tips. (Sez the man who has barely mastered calculus)   

[wimvb]

8 hours ago, Barry-Wilson said:

Linear Fit as adaption tool between channels: I’ve been using PixInsight for many years and the two times I used this ended in awful results. I have never found a need to use this tool at all.

Hear, hear.

Background neutralization does a much better job. (I don't even use DBE to neutralize the background.)

(Throwing in other participants into the jargon-fest.)