PixInsight ABE sending background jet black?

[x6gas]

Hi all.

I'm trying patiently to learn how to use PixInsight and posting this question about a problem I'm having with background neutralisation in the hope that someone can shed some light on it and / or it may help others...

Basically when I was trying to process my data of the Whale Galaxy I was having no end of trouble stretching the data.  Now admittedly I was doing the stretching in PS and I did get slightly better results if I did a histogram transformation in PI but nonetheless... what was happening was ABE (DBE too as it happens) seems to be clipping the black point on the data.  This means (I think) that in order to get a natural background colour I was having to stretch a clipped background which resulted in a lot of background noise.  It was driving me crazy and it took ages for me to figure out that PI's background normalisation was to blame.  I've never struggled so much with basic processing and while I wasn't happy with the result (see the last four images I posted here) I waived the white flag in a fit of exasperation.

I don't suffer much vignetting so even without flats if I run the same stacks, sans ABE, through my normal workflow a much better image drops out with no fuss at all (see below).  So, OK, I could just dispense with PI's background neutralisation altogether but I am curious: am I missing a step or doing something wrong here?

My PI workflow was: calibration with darks only (of the correct length); cosmetic correction using a master dark and auto detect; sub frame selection; registration; stacking using sigma clipping, weighted by noise evaluation; dynamic crop (and I changed camera angle half way through data acquisition so there was a fair bit of cropping); deconvolution, gentle noise reduction using MLT; then ABE (or not) channel combine; (sometimes I'd do photometric colour calibration here but it makes no difference to this issue); export as a TIF for further processing.  I've tried doing a linear fit before the channel combination but that too makes no difference.

When I open the ABE version in PS the background values are zero.  Without ABE they are 3 and that makes a HUGE difference.

Any thoughts?

Best wishes, Ian

Image without ABE (35 x 300s per channel)

[MarkAR]

ABE can be a bit hit and miss and DBE can be tricky to get right.

Try it directly after cropping. 

[x6gas]

20 minutes ago, MarkAR said:

Try it directly after cropping. 

Thanks for the suggestion Mark, but that makes no difference.  Still get background values of zero...🤒

[jimjam11]

12 hours ago, x6gas said:

My PI workflow was: calibration with darks only (of the correct length); cosmetic correction using a master dark and auto detect; sub frame selection; registration; stacking using sigma clipping, weighted by noise evaluation; dynamic crop (and I changed camera angle half way through data acquisition so there was a fair bit of cropping); deconvolution, gentle noise reduction using MLT; then ABE (or not) channel combine; (sometimes I'd do photometric colour calibration here but it makes no difference to this issue); export as a TIF for further processing.  I've tried doing a linear fit before the channel combination but that too makes no difference.

Abe (or dbe) would typically be done immediately after stacking and prior to decon, mlt etc? If edge stacking artefacts cause trouble you could crop prior to abe/dbe. Do you have the warren Keller book, it has some excellent processing workflows in it...

[jimjam11]

The image looks superb by the way! 

Did you run through preprocessing manually or did you use BPP/WBPP? 

[MarkAR]

Having another close look at the non ABE image above, I don't think there's really any background worth removing. It is pretty flat and dark as it is.

It's almost like you're telling PI to take some more out hence the clipping.

Just don't bother with background extraction and carry on with it. 

[x6gas]

3 hours ago, jimjam11 said:

Abe (or dbe) would typically be done immediately after stacking and prior to decon, mlt etc? If edge stacking artefacts cause trouble you could crop prior to abe/dbe. Do you have the warren Keller book, it has some excellent processing workflows in it...

Thanks for the reply JimJam.  Yes I have the latest edition of the book which I am finding quite useful and seems to correct / contradict some of the popular online tutorials.

As I said when replying to Mark I did try ABE immediately after cropping the image but the same thing happens and the background is clipped to zero...

3 hours ago, jimjam11 said:

The image looks superb by the way! 

Did you run through preprocessing manually or did you use BPP/WBPP? 

Thanks very much.  The data is pretty good given that it's not getting astronomically dark (though this is using only around half the subs I captured).

I did preprocessing manually.  I am really keen to try to understand what I am doing in PI rather than just follow a recipe - it's uphill work as the documentation is, frankly, awful in many places - so I've experimented a lot with different stacking algorithms, subframe weighting, normalisation etc. and have settled on what seems to give the best results for my set-up.

Thanks for the replies!

[x6gas]

2 hours ago, MarkAR said:

Having another close look at the non ABE image above, I don't think there's really any background worth removing. It is pretty flat and dark as it is.

It's almost like you're telling PI to take some more out hence the clipping.

Just don't bother with background extraction and carry on with it. 

Yeah you're right Mark there isn't a lot there (though there is some as I didn't calibrate with flats) in fairness so perhaps that is the problem.  I guess I need to try another data set from the same set-up.

Do you output to PS at all or do all your processing in PI?

[jimjam11]

8 minutes ago, x6gas said:

Thanks for the reply JimJam.  Yes I have the latest edition of the book which I am finding quite useful and seems to correct / contradict some of the popular online tutorials.

As I said when replying to Mark I did try ABE immediately after cropping the image but the same thing happens and the background is clipped to zero...

Thanks very much.  The data is pretty good given that it's not getting astronomically dark (though this is using only around half the subs I captured).

I did preprocessing manually.  I am really keen to try to understand what I am doing in PI rather than just follow a recipe - it's uphill work as the documentation is, frankly, awful in many places - so I've experimented a lot with different stacking algorithms, subframe weighting, normalisation etc. and have settled on what seems to give the best results for my set-up.

Thanks for the replies!

As an experiment it would be worth using the wbpp script with your data and then trying abe/dbe again. I have always found manual preprocessing a nightmare so always use bpp/wbpp even if I run image integration manually afterwards (the main part where you want to tinker). If you get the same result it would prove your manual preprocessing steps were correct and probably indicate an issue with your calibration files?

I use lightroom/ps as the last step in processing, normally adjusting levels down to get the background where i want it.

 

[MarkAR]

2 hours ago, x6gas said:

Do you output to PS at all or do all your processing in PI?

Only use PI. Haven't found the need for PS.

[wimvb]

In ABE and DBE, check "normalization" in the corrections tab. This maintains the median background value.