Can you make my Whale look better? (Lots of data available)

[NigeB]

Hello All,

Yesterday I posted the results of a recent imaging attempt on NGC4361 (The Whale Galaxy) over in the Imaging - Deep Sky section. Here's the original thread with a description of my method and what I'm not happy with:

https://stargazerslounge.com/topic/313366-thar-she-blows/

In the post I asked for comments/criticism and some suggestions on how to get a better result. One of the suggestions (thanks @Allinthehead) was to make my data files available for others to work on - which sounds like a great idea. So I've posted the (~1.5 GB) of raw lights, flats, darks and bias frames on my DropBox area. I've also posted four fits files, which are corrected, aligned and stacked L, R, G, and B frames, in case anyone wants to have a go at processing but doesn't want to start from the very beginning. (Of course, these aligned and stacked frames may also contain errors from my processing method - so it's better to start with the raw files if you can!)

So if anyone is looking for something to do when the nights are cloudy, I'd be really interested to see what others can do with this, because I'm well aware that my processing technique leaves much to be desired. I'm particularly keen to hear from people using Pixinsight, which is the system I decided to invest in. If you can make this look good, please post, preferably with a summary of your processing steps, so that I can better understand how to improve my technique.

Here is the link to my Dropbox area containing the files - the zip files contain the raw data:

https://www.dropbox.com/sh/th6595cdneqr24u/AADdUywsfzkARZ-917O_gHFta?dl=0

Thanks!

Nigel 

(My own attempt shown below).

 

 

 

 

[geordie85]

I've had a quick play and I must say it was a joy to work with such good data. I've also attached a screen print of the processing steps I took in photoshop (easier than listing everything I done, but also ran a high pass filter at the end)

Thank you for allowing me (well, everyone who want to) have access to your data. it really was such a pleasure.

[MrsGnomus]

Hi Nigel.  Thanks for uploading the data.  I had a play with it - since we are getting nothing from home at the moment.  My workflow seems to be significantly different to yours.  I have copied what you did from your earlier thread and added my steps in yellow.  

Here are the main processing steps I applied:

1) Dark/Bias/Flatfield correction, alignment and stacking in Pixinsight using Winsorized sigma clipping.

2) Linear fit of R, G and B using L as the reference.

I/We don't do Linear fit at all and I certainly wouldn't use Lum as the reference.  When I used to do Linear Fit, I'd choose the 'darkest' of the colour frames as the reference. 

3) Applied a histogram stretch to all frames (transferring the Screen Transfer Function settings to make these permanent)

This doesn't seem right.  I would combine the RGB without stretching the individual stacks (using Channel Combination).  I would try to get the colours right before stretching.  I would do DBE to the RGB combination.  I would then do a Background Neutralization and Color Calibration.  I might do some SCNR at this stage.   I would then do the stretch.  With the RGB we tend to do a 'Masked Stretch' with the Background number set around 0.07 or 0.08.   I am just looking for nice colour at this stage, the brightness will come from the Lum.  At this point, I would check that the peaks of the histograms for the RG and B are aligned, and if not I would tweak the individual channels to achieve this.   This gives me a more balanced colour at the end.

4) Performed Dynamic Background Extraction, subtracting the resulting background frames from images to remove some gradients and a few irritating flat field issues (some remain)

Do the DBE before stretching - see above

5) LRGB Combination performed

I did a DBE on the Lum then stretched it using Histogram Transformation.  I did this in small stages.  I always do repeated iterations - drives Gnomus mad - watching the Preview - trying to keep the star sizes from getting away from me.  Once happy, I add the stretched Lum to the stretched RGB using LRGB Combination with only the Lum box ticked.  I dragged the saturation slider down to 0.36 - this boosts the saturation.  I ticked the Chrominanace Reduction box.

6) Increased saturation across the spectrum to bring out some colour. 

With the combined LRGB,  I check the histograms are still aligned and correct them again if necessary.  I do some more SCNR - the Boosted Autostretch button allows you to see if there is still some green in there.  I boosted saturation with Curves and with the Color Saturation tool - picking out the blue of the outer galaxy and the orange-yellow of the core.

7) Found that star sizes in the red image are larger than the blue and green. This was giving an excessive red tint to all stars. So I went back to the red frame, applied a star mask, followed by Morphological Transformation to reduce the red star sizes a little.

I didn't find a problem with this, since I didn't stretch the colours individually before combining them.

8) Re-did the LRGB combination + saturation increase. Found the red halo issue was helped, but not completely removed.

9) Applied a star mask and reduced saturation levels in the red channel for stars only. This removed the general red tint and brought out some colour in the stars.

10) Some adjustment of the R G and B curves and cutoffs in Histogram Transformation to try and get a colour balance in the galaxy which looked right (aesthetically).

11) Final tweak of overall RGB histogram to hide the worst of the background and achieve what looked like sensible brightness, contrast and colour balance.

I didn't do any of 8-11.  I did an HDRMultiscale Transform to a copy of the image.  Then I took the original and the HDR image into Photoshop and painted on the bits I liked.  I also did some Unsharp Mask in Photoshop.  I did find a problem with your Lum.  There was a large ring-shaped artefact extending a quarter of the way up the frame.  This is probably something to do with flats.  To be honest I didn't look at the individual subs you posted, I just used the calibrated stacks.  I 'fixed' the problem by placing the RGB only image over the LRGB one (in Photoshop again), changed the blending mode to Luminosity and painted over the artefact.         

I attach a screenshot of the odd Luminance artefact.  Then my finished image. 

 

Edited May 1, 2018 by MrsGnomus

[ollypenrice]

Although I get the hell out of Pixinsight as soon as I can (as Mrs and Mr Gnomus know ) I heartlily agree with Mrs G's advice and her result is lovely. I also like Geordie's but I prefer Mrs G's higher black point, lower saturation and stretch which keeps below the noise floor.

My PI approach to RGB is always very simple: combine the colours at equal weighting, edge crop, screen transfer function 'preview stretch' then DBE and SCNR green set as low as will do the job. (The original post was very green.) However, I've recently been trying PI's photometric colour calibration and it seems convincing. After that I head for Ps.

The background sky is a peach in Mr' G's rendition. Dead even, just the right brightness and just the right amount of grain with no sense of noise reduction. All I can think of to try on this rendition to take it a touch further (and it might not work) would be to put a little upward kink in the Curve just above the (pinned) background sky to pull out the faint outlying tidal structures and a slight lowering of the cyans in red (Ps, Selective Colour) to perk up the Ha components. Because Mrs G has done such a good job on the stretch my impression that there might be a tiny bit more left in the data could be an illusion. For me the perfect stretch does look as if there's a bit more left in there.

Olly

Edit: To emphasize a point already made, I would never stretch the colours individually. Combine first, then stretch.

Edited May 1, 2018 by ollypenrice

[NigeB]

Hi Geordie85, MrsGnomus, Olly,

Wow....

I'm astonished - your images are fantastic - thank you so much for your efforts. I have to say this is a revelation to me; I knew I was not getting the best out of the data, but I didn't expect to see such a big difference between my outputs and those of people who really know what they're doing! I guess I'm partly pleasantly surprised that the data I took was good enough to achieve these results; it's also sobering to see how far off my own processing approach is. I've just shown my wife, and she is similarly amazed.

Thanks also for setting out your processes in detail - this is an enormous help, and I'm going to go back and work through this from scratch this evening following your methods. MrsG... Yes, that's a flat field problem in the L frame - I took twilight flats on the first evening, but not subsequently - big mistake, because this feature appeared in the later nights. 

7 hours ago, geordie85 said:

I've had a quick play and I must say it was a joy to work with such good data. I've also attached a screen print of the processing steps I took in photoshop (easier than listing everything I done, but also ran a high pass filter at the end)

Thank you for allowing me (well, everyone who want to) have access to your data. it really was such a pleasure.

Not at all! Thank you for your comments and efforts!

Olly, am I right in thinking you're not a big fan of PI? Why is that? (I recall a post from you some years ago where you were talking about the usefulness of layers, which as far as I know, PI doesn't have).

Thanks again; it's going to take me a while to go through the information in your posts and properly understand the logic, but I'll post an updated version of my own once I've done that.

Much appreciated!

 

Nigel

 

[geordie85]

I've had another play around since its horrid and cloudy outside. Hopefully it's an improvement on my first attempt which I've used a similar approach but not so heavy handed this time.

[wimvb]

An "embossed" circular artefact is most likely a dust bunny that moved between taking lights and taking flats. As it only shows on the L master, it's located on the L filter, and probably moves a little when changing filters.

[wimvb]

Just to show what the "traditional" PI workflow will achieve:

• RGB combination of your colour masters, no linear fit
• DBE subtraction, checking the "normalize" box in corrections
• Background neutralization
• Colour Calibration using the galaxy as white reference
• Masked stretch, default values but target background reduced to 0.1

Saved as jpeg

[NigeB]

Hello Again,

I've had an entertaining evening reprocessing the original files following the methods described in this thread. The difference is quite amazing. My version is not up to the standards of those posted here - my stars are rather "loud" at the moment and they seem to pop out of the image. I think I can see how to tone them down - I'll work on this a bit more. I don't have PS, so I'm trying to do as much as possible in PI - although I do have Gimp...

I suspect I'm going to be working on this for several more days, but I feel like the image quality has already improved enormously thanks to your posts and is probably worth showing the delta here. So here are two versions from this evening.

Many thanks again - this has been a real revelation.

Nigel

Edited May 1, 2018 by NigeB
Replaced Tif with JPEG for forum

[wimvb]

Since your L master has a few nasty dust bunnies, I created a synthetic luminance in PI:

ImageIntegration of the R, G, and B masters. Average combination, no pixel rejection.

After applying a few passes of DBE (division as correction, because the only visible gradient has a circular pattern = vignetting), the background is flat and can take a lot of beating. I'm now doing deconvolution to reveal detail in the galaxy. Seems to work quite well.

[MrsGnomus]

6 hours ago, NigeB said:

I've had an entertaining evening reprocessing the original files following the methods described in this thread. The difference is quite amazing. My version is not up to the standards of those posted here - my stars are rather "loud" at the moment and they seem to pop out of the image. I think I can see how to tone them down - I'll work on this a bit more. I don't have PS, so I'm trying to do as much as possible in PI - although I do have Gimp...

I suspect I'm going to be working on this for several more days, but I feel like the image quality has already improved enormously thanks to your posts and is probably worth showing the delta here. So here are two versions from this evening.

Many thanks again - this has been a real revelation.

 

Both of the new versions are a huge improvement. I prefer the colours in the galaxy in the first one and the background in the second one. As you don’t have PS you could use Pixelmath to blend different versions of images - this would be one way of dialling down the effects of HDRMultiscale Transform, for example. You can make masks in PI, but it is so much easier to do it in PS that I/we rarely bother except for star masks. There are some good PI tutorials on Kayron Mercieca’s Light Vortex Astronomy site.

[ollypenrice]

13 hours ago, NigeB said:

Hi Geordie85, MrsGnomus, Olly,

Wow....

I'm astonished - your images are fantastic - thank you so much for your efforts. I have to say this is a revelation to me; I knew I was not getting the best out of the data, but I didn't expect to see such a big difference between my outputs and those of people who really know what they're doing! I guess I'm partly pleasantly surprised that the data I took was good enough to achieve these results; it's also sobering to see how far off my own processing approach is. I've just shown my wife, and she is similarly amazed.

Thanks also for setting out your processes in detail - this is an enormous help, and I'm going to go back and work through this from scratch this evening following your methods. MrsG... Yes, that's a flat field problem in the L frame - I took twilight flats on the first evening, but not subsequently - big mistake, because this feature appeared in the later nights. 

Not at all! Thank you for your comments and efforts!

Olly, am I right in thinking you're not a big fan of PI? Why is that? (I recall a post from you some years ago where you were talking about the usefulness of layers, which as far as I know, PI doesn't have).

Thanks again; it's going to take me a while to go through the information in your posts and properly understand the logic, but I'll post an updated version of my own once I've done that.

Much appreciated!

 

Nigel

 

I'm a fan of parts of PI but, in the important matter of layers versus masks, give me layers every time. Layers and masks don't resemble each other, on the face of it, but they serve the same purpose in allowing the imager to process selectively only one part of the image at a time. Masks have to be made to cover exactly what you want to be covered and this is very difficult. In Layers you modify the whole bottom layer and then simply erase, as fully or partially as you prefer, the top layer to let the modification into the result image. You can see the consequences of each touch of the eraser as you make it and can go back as you please. I also find the Ps colour selection tool a simple and powerful way of selecting certain parts of an image.

In the early days of PI its users were often tempted into gross excess, particularly in the use of HDR wavelets, but now there are plenty of nicely processed PI images. Check out Barry Wilson's, for instance, whose processing is always subtle and invisible.

Olly

[MrsGnomus]

I had another go at it this morning from scratch, trying to incorporate Olly's suggestion of pulling out more of the faint structures and also using Wim's idea of deconvolution.  This might be a slight improvement.  I still resorted to PS to get rid of the Lum artefact and for one or two other tweaks, like Unsharp mask.  

Edited May 2, 2018 by MrsGnomus
Typo

[wimvb]

6 minutes ago, MrsGnomus said:

I still resorted to PS to get rid of the Lum artefact

It's easier to use the combined r, g and b images as a synthetic luminance. The artefacts are only on the L image. The synthetic L is much cleaner and easier to process. And since the colour images weren't binned, they have the same level of detail.

Btw, I love those tight stars in your image.

1 hour ago, ollypenrice said:

In the early days of PI its users were often tempted into gross excess, particularly in the use of HDR wavelets, but now there are plenty of nicely processed PI images. Check out Barry Wilson's, for instance, whose processing is always subtle and invisible.

Yes, PixInsight is a power tool, and you need to learn how to tame it. As I've said before: it's the driver and not the car that gets you to your designation.

[gorann]

So, here is what I came up with today. I used artificial Lum from the RGB data as suggested by Wim (to avoid the dust bunnies). Now when I see Mrs Gnomus new magnificent version I may have another go a sharpening (deconvolution) so it is possible that I come back with a new version. Maybe I should look into what the Lum data looks like and possibly use it selectively on the galaxy so I no not have to deal with the bunnies. Al done in PS CS5 (using the files Nigel had stacked).

Thanks Nigel for this entertainment! If you do not mind I may put it on my Astrobin with full credits of course.

[tooth_dr]

What talent in this thread!  Just shows you no matter how good your scope/camera/skies are, if you cant process the data you'll end up with a lemon.

[MrsGnomus]

25 minutes ago, gorann said:

So, here is what I came up with today. I used artificial Lum from the RGB data as suggested by Wim (to avoid the dust bunnies). Now when I see Mrs Gnomus new magnificent version I may have another go a sharpening (deconvolution) so it is possible that I come back with a new version. Maybe I should look into what the Lum data looks like and possibly use it selectively on the galaxy so I no not have to deal with the bunnies. Al done in PS CS5 (using the files Nigel had stacked).

Thanks Nigel for this entertainment! If you do not mind I may put it on my Astrobin with full credits of course.

 

Thanks Gorann.  I think it is worth using the Lum for the galaxy itself.  Even though there was >6 hours of RGB, the false Lum taken from this wouldn’t be equivalent to 6 hours Lum, at least I don’t think it would.  However, the Lum artefacts were a nuisance.  I took the full LRGB image and the RGB only image into Photoshop.  I adjusted the RGB so that the background sky matched that of the LRGB image as closely as possible. I placed the RGB on top of the LRGB as a layer. I then changed the blending mode to Luminosity.  I think this is the same as extracting the Lum and using that.  I then applied a mask to the RGB layer and used the paintbrush tool on the mask to allow the RGB background to come through, whilst keeping the LRGB galaxies unchanged.  

Olly’s way of doing things is slightly different.  He does his adjustment on a layer underneath the main image and then uses the eraser tool to remove bits of the main image to allow the adjusted layer underneath to come through.  The result is identical to using a mask, but may be a bit quicker.  I am less confident about making catastrophic errors, even though there is PS ‘History’.  Using a mask makes it much easier for me to visualise what is going on.  A photoshop mask works identically to a PixInsight mask.  It allows you to choose the bits of an image that get adjusted.  The only difference between the PI and PS mask is that one is much easier to make than the other.

[MrsGnomus]

35 minutes ago, tooth_dr said:

What talent in this thread!  Just shows you no matter how good your scope/camera/skies are, if you cant process the data you'll end up with a lemon.

In the Gnomus household we think that the majority of the skill is in the processing and not in the mechanics of setting up the equipment.  Even I can tighten up bolts and plug in cables! We have had a few trips to Olly’s and watching him process his data (not all of which could be described as pristine ?) has helped us both enormously. I would go as far as saying that it was only after the first of those trips that our processing started to go in the right direction. I still have the copious notes that I made while Steve processed the Seagull image that I use as my avatar under Olly’s supervision. Each time we go we learn something new - for example on our last visit it was which brand of self destructing filter wheels to avoid!

[gorann]

42 minutes ago, MrsGnomus said:

Thanks Gorann.  I think it is worth using the Lum for the galaxy itself.  Even though there was >6 hours of RGB, the false Lum taken from this wouldn’t be equivalent to 6 hours Lum, at least I don’t think it would.  However, the Lum artefacts were a nuisance.  I took the full LRGB image and the RGB only image into Photoshop.  I adjusted the RGB so that the background sky matched that of the LRGB image as closely as possible. I placed the RGB on top of the LRGB as a layer. I then changed the blending mode to Luminosity.  I think this is the same as extracting the Lum and using that.  I then applied a mask to the RGB layer and used the paintbrush tool on the mask to allow the RGB background to come through, whilst keeping the LRGB galaxies unchanged.  

Olly’s way of doing things is slightly different.  He does his adjustment on a layer underneath the main image and then uses the eraser tool to remove bits of the main image to allow the adjusted layer underneath to come through.  The result is identical to using a mask, but may be a bit quicker.  I am less confident about making catastrophic errors, even though there is PS ‘History’.  Using a mask makes it much easier for me to visualise what is going on.  A photoshop mask works identically to a PixInsight mask.  It allows you to choose the bits of an image that get adjusted.  The only difference between the PI and PS mask is that one is much easier to make than the other.

Thanks Lis!

I think what I will do is essentially what you describe. It is just the order of the layers that may differ but the end resullt will be the same - just like in Olly's way of doing it. I will take the Lum data and process it with special empahsis on sharpness and then put it in as a lum layer (blend mode luminosity) and only use it for the galaxy by using the brush tool. The atrificial lum I now used was just a grayscale version of the RGB image but it allowed me to suppress colour noise in the sky. I am quite pleased with the sky and the stars so I don't want to mess that up. Actually, your sky and my sky looks very similar and if I get your method description right, you removed the Lum layer from the sky. I can see that you have a bit less noise when I zoom in on the skies. Did you use Gaussian blurr on it? At least I thought of doing that but decided to leave some noise in there.

Obviously a trip to Olly would be great in many ways and it is on my things to do! (if he will have me)

Need to do some handyman work in the garden now and then BBQ, so I may not do any more processing until tomorrow.

[MrsGnomus]

1 hour ago, gorann said:

Did you use Gaussian blurr on it?

I have no idea if I used Gaussian blur or not! What I did use is Noel’s Deep Space Noise Reduction. It’s entirely possible that Gaussian blur is one of the steps in that recipe.

[NigeB]

Hello Everyone,

I've been at a meeting in London all day so am just catching up now. This has been a fantastic thread... thanks to everyone for their responses. MrsG, I was going to ask you where you learned your processing skills, and you've answered that. In the BC (before children) years my wife and I used to tour France in our Caterham - I reckon I was within about 20 miles of Olly's place on at least 5 occasions over the years. I may need to suggest to Herself that we attempt a revival and book a stay at Les Granges... 

9 hours ago, gorann said:

Thanks Nigel for this entertainment! If you do not mind I may put it on my Astrobin with full credits of course.

Gorann, I really like what you did with the data - yes of course, please go ahead and use/post however you want (same for everyone else). I've added a note on my Astrobin entry that points back to this thread and credits the advice I've received from everyone here.

I'm also getting the feeling that perhaps an image is never finished - there's always the  temptation to go back and adjust processing. I know I'm going to be re-working the image (and revisiting some earlier targets) over the coming days.

Nigel

[wimvb]

Tried a variation of LRGB combination on this image: L*a*b* combination. Basically like this:

created a synthetic luminance (image integration of R, G, and B masters) and a standard RGB image.

Crop, DBE, BN, CC followed by masked stretch for the RGB image. Then decomposed into L, a, and b

Processed the synthetic luminance for maximum detail, using HDRMT, some star reduction, etc

Then combined the synthetic luminance and the a, and b images in L*a*b* combination.

Finished off with chroma noise reduction.

[NigeB]

20 minutes ago, wimvb said:

Tried a variation of LRGB combination on this image: L*a*b* combination. Basically like this:

created a synthetic luminance (image integration of R, G, and B masters) and a standard RGB image.

Crop, DBE, BN, CC followed by masked stretch for the RGB image. Then decomposed into L, a, and b

Processed the synthetic luminance for maximum detail, using HDRMT, some star reduction, etc

Then combined the synthetic luminance and the a, and b images in L*a*b* combination.

Finished off with chroma noise reduction.

Looks fantastic! Thanks for the process information - very useful indeed.

 

Nigel

 

[gorann]

12 hours ago, wimvb said:

Tried a variation of LRGB combination on this image: L*a*b* combination. Basically like this:

created a synthetic luminance (image integration of R, G, and B masters) and a standard RGB image.

Crop, DBE, BN, CC followed by masked stretch for the RGB image. Then decomposed into L, a, and b

Processed the synthetic luminance for maximum detail, using HDRMT, some star reduction, etc

Then combined the synthetic luminance and the a, and b images in L*a*b* combination.

Finished off with chroma noise reduction.

Looks great with a lot of fine detail Wim. What settings did you use in HDRMT to get that sharpness? I had a try but there are just so many settings to play with.

[gorann]

Here is my current version. Sharper than the previous one due to a sharpened layer of luminosity put on the galaxy (this time the real thing and not derived from the RGB data).

Edited May 3, 2018 by gorann