Correcting colour gradients

[wetcoaster]

I live  in a light polluted urban area near Vancouver BC.  Sky Quality Meter readings are typically 18.8 versus the stellar 21.5 that I get at my cottage. The best SQM reading possible is 22 by the way.

I have been trying to image portions of the Milky Way. I used an Astronomik City Light Suppression filter. I pre-processed in LightRoom to apply lens corrections and a few other small edits. Then I aligned and stacked in Pixinsight and followed up with Dynamic Background Extraction and MaskedTransfer. What I got looks like a badly designed quilt. See attached.

Since then I have tried to use HistogramTransformation to balance the colours but without success. I finally painstakingly edited the image in PhotoShop CC and managed to get a so-so image. I am wondering if there is some tool or technique that will colour balance this image. Or if it is Mission Impossible. The CLS filter chops out around 100nm of the visible spectrum.

BY the way, I have used a Custom WB on occasion but the problem will not go away.

Don

[Oddsocks]

Hello Don,

I will ask mods to move this post to the imaging section, it will get missed here in the lounge.

There are a couple of questions that need answering so that the imaging experts can help you.

1. What is the optical system you are using and how fast?

Astronomik say the CLS filter is good for between f1.3 to f15, there are some examples of images from DSLR camera's that show a darkened centre and coloured gradient ring with the CLS at f1.2.

2. Did you calibrate with darks and flats?

Your posted image shows hundreds of hot pixels, at this pixel scale with many stars also occupying a single pixel then darks or bias-used-as-darks together with a bad pixel map is going to be important. PixInsight DBE has two correction methods, correction by subtraction, used to remove sky gradients and correction by division used to remove vignetting. If your image contains both sky gradients and vignetting then DBE will not completely remove both in a single operation, if the CLS has introduced further gradients at a fast f ratio this will  further complicate things so flats are important, they will correct the vignetting and help with the CLS induced circular gradients.

For MW images at this scale where very little plain sky background exists DBE will have a hard job creating an accurate correction mask for any remaing gradients and you might achieve better results using Photoshop to blend a duplicate, gaussian-blurred and stretched layer by subtraction to correct the remaing gradients.

3. If you did take calibration frames did you apply these before any operation in Lightroom?

Calibration frames have to be applied while the image is still linear and unaltered, any difference between the original lights and the calibration frames will introduce new artefacts and not remove the original ones.

4. It would help the imaging experts if you could provide the original unaltered raw frames, can you place raw lights and calibration frames in a free Dropbox or Google drive folder so they can assess better any possible solution?

Hopefully, once the post gets moved to the imaging section and you reply to the above then the expert imagers will be able to assist.

William.

[wimvb]

Try doing the whole process in just one program. This will make it easier to follow what is done to the image.

DBE can be quite tricky. It is essential that you don't cover stars with the samples you place. It usually pays to spend some time to correct samples that were placed by DBE, or place your own samples.

Before applying the correction in DBE, examine the background model that DBE comes up with. If it isn't smooth and quite even, you probably put some samples over nebulosity or stars. Move or remove some samples in those areas and try DBE again. Usually one application of DBE will give you an image with an even background, but with really difficult gradients, you may have to reapply the process.

 

Good luck

[wxsatuser]

Gradients and colour balancing for widefield images in LP skies with LP filters, is an absolute nightmare.

I followed a lot off Roger Clark's stuff by using Camera Raw and then stacking in Registar.
Tony Hallas used a similar method to defeat colour noise.

I have in the past found that several passes of ABE in PI will kill most of the gradients, whats left I cropped out.
If you use DBE, as Wim pointed out, carefully place the points so that they sit on background only, look in the little
preview box in the controls, you should be able to see if your point is on any structure like stars.
Try to get some in each corner helps but you may find it just will not shift some of the tough stuff.

[wetcoaster]

On 10/3/2016 at 23:53, Oddsocks said:

Hello Don,

I will ask mods to move this post to the imaging section, it will get missed here in the lounge.

There are a couple of questions that need answering so that the imaging experts can help you.

1. What is the optical system you are using and how fast?

Astronomik say the CLS filter is good for between f1.3 to f15, there are some examples of images from DSLR camera's that show a darkened centre and coloured gradient ring with the CLS at f1.2.

2. Did you calibrate with darks and flats?

Your posted image shows hundreds of hot pixels, at this pixel scale with many stars also occupying a single pixel then darks or bias-used-as-darks together with a bad pixel map is going to be important. PixInsight DBE has two correction methods, correction by subtraction, used to remove sky gradients and correction by division used to remove vignetting. If your image contains both sky gradients and vignetting then DBE will not completely remove both in a single operation, if the CLS has introduced further gradients at a fast f ratio this will  further complicate things so flats are important, they will correct the vignetting and help with the CLS induced circular gradients.

For MW images at this scale where very little plain sky background exists DBE will have a hard job creating an accurate correction mask for any remaing gradients and you might achieve better results using Photoshop to blend a duplicate, gaussian-blurred and stretched layer by subtraction to correct the remaing gradients.

3. If you did take calibration frames did you apply these before any operation in Lightroom?

Calibration frames have to be applied while the image is still linear and unaltered, any difference between the original lights and the calibration frames will introduce new artefacts and not remove the original ones.

4. It would help the imaging experts if you could provide the original unaltered raw frames, can you place raw lights and calibration frames in a free Dropbox or Google drive folder so they can assess better any possible solution?

Hopefully, once the post gets moved to the imaging section and you reply to the above then the expert imagers will be able to assist.

William.

Thanks for looking at my problem William.

I used a Canon EF 100/2.8 USN Macro lens at f/3.5.

I do not use darks and flats. My Canon 6D has no dark current problems and is very low noise. In fact, noise is never an issue with my imaging. As for flats, I tried to create them for some of my lenses but it went badly. I now use the lens correction functions in PhotoShop and LightRoom. I can also use Canon DPP as well.

The CLS filter for full frame often creates a shadow on the bottom of the frame. At one time I had problems with it loosening but after chatting with Gerd Neumann, I bent the frame a bit and it now remains seated.

There are no hot pixels, just blown out stars.

Here is a Dropbox link to the PI Integration image <https://www.dropbox.com/s/y9bsiw2rjhjyrjz/160928 integration ET%2BNA v1.tif?dl=0>

I received some good advice in the digital_astro Yahoo group and will use that to correct later captures. However, I have decided to leave this one be because it is never going to look good enough to publish.

I did a series of 3 tests using my 6D with and without the CLS filter, and my A7R with a Hutech LPS-V4 clip in filter (it works but vignettes). All runs were taken with the 100/2.8 macro lens at f/3.5. I went after the area around Deneb and Sadr so captured a number of red emission nebulas. The filtered images turned out reasonably good but the unfiltered one is not so great. I used the editing process championed by Roger Clark (clarkvision.com) but didn't get the results he achieved. That may be due to his superior editing skills although I used his recipe.

 

Don

 

 

[wetcoaster]

On 10/4/2016 at 15:01, wimvb said:

Try doing the whole process in just one program. This will make it easier to follow what is done to the image.

DBE can be quite tricky. It is essential that you don't cover stars with the samples you place. It usually pays to spend some time to correct samples that were placed by DBE, or place your own samples.

Before applying the correction in DBE, examine the background model that DBE comes up with. If it isn't smooth and quite even, you probably put some samples over nebulosity or stars. Move or remove some samples in those areas and try DBE again. Usually one application of DBE will give you an image with an even background, but with really difficult gradients, you may have to reapply the process.

 

Good luck

Thanks Wim.

I try to use Pixinsight but relying solely on it means missing the lens corrections for example. And I have found that ImagesPlus has a couple of features that are superior to what I can do in PI. Star size reduction and removing halos around stars are examples of that. perhaps PI can do them well but I haven't been able to find the tool that works for me.

I have tried dozens of parameter combinations if DBE with limited success most of the time when I really need it, like for this image. It's trial and error non-stop.

 

Don

[wetcoaster]

On 10/4/2016 at 21:29, wxsatuser said:

Gradients and colour balancing for widefield images in LP skies with LP filters, is an absolute nightmare.

I followed a lot off Roger Clark's stuff by using Camera Raw and then stacking in Registar.
Tony Hallas used a similar method to defeat colour noise.

I have in the past found that several passes of ABE in PI will kill most of the gradients, whats left I cropped out.
If you use DBE, as Wim pointed out, carefully place the points so that they sit on background only, look in the little
preview box in the controls, you should be able to see if your point is on any structure like stars.
Try to get some in each corner helps but you may find it just will not shift some of the tough stuff.

Amen to your first comment. Much of my work is with really wide angle lenses ranging from 14mm through 35mm. I have some nifty captures of the Orion Complex that I still haven't edited to my satisfaction. But I am hopeful that what I have recently learned will make a difference when I go back to it.

I haven't tried multiple passes of ABE and will do so. Given how wide angle my shots are I find cropping is the best way. The problem with this image is that the two primary nebulas are near the edges of the FOV.

With DBE, I find that the samples often don't get put in the area that needs fixing. In this case, the right side of the image. I bumped up the samples per row to 40 and even higher and it still din't cover the image as expected.

 

Don

[ollypenrice]

The best tool is DBE and it can defeat pretty will anything, but it needs to be used to its advantage.

A large number if samples is not an advantage. You want it find a gradient, not details. A gradient is just a general drift in colour. The ultimate minimalist sample set would be one in each corner and one in the middle. Something with a bit more than that sould be better but lots and lots of markers become counter productive. The 'threshold' value should be very high if you have bad LP gradients. Keep raising it from default till it does the job.

Olly

[wimvb]

I downloaded the tif file and applied one iteration of DBE, manually placing only samples in the corners, and along the edges, plus a bit towards the center. Less than 20 samples in total.

Then applied back ground neutralisation and colour calibration, a histogram stretch, resampled (50 %) and saved as PNG. There's still some gradient left in the lower left and right corners. Due to the number of stars, it's difficult to place samples. I used a sample radius of 9, and set tolerance to 1.95, minimum sample weight = 0.4. Correction method subtraction with normalisation.

Hope this helps

[Oddsocks]

The thing I find puzzling with this image, the original and Wim's is the strong shift in star colour across the image. 

Towards the edges of the frame the star colours look more normal but in the centre of the frame they become really very red and not at all what you might expect from a normal star field. I can't help wondering if something has gone wrong in the image integration phase and wonder how a single capture looks in comparison to the integrated version.

I have come across a few web postings with users testing various combinations of LP filters and DSLR lenses that show circular colour gradients with one optical design of lens at a fixed f ratio and length where another optical design of an exact same f ratio and length shows no colour gradients, all unable to explain why but able to demonstrate the effect. I do wonder if it has a lot to do with the shape and diameter of the final base optic in the lens assembly, i.e a small diameter final element that has a steep ray angle to cover the full size detector against a lens group where the final element has a larger diameter and therefore has a correspondingly shallower ray angle to cover the same size detector. If using a clip in LP filter rather than a front-of-lens filter I would expect this might explain where these circular gradients originate. I notice that all the LP filter manufacturers quote max f ratio useable based on telescope optics where the ray angle is directly related to the objective or mirror and no modification occurs as the beam travels from objective, through filter to detector. But in a DSLR lens it is a different story, the ray angle is bent many times as it passes through the lens groups and the final exit ray angle will have little to do with the overall f ratio of the group as a whole. To the clip-in filter a small diameter final lens element with a steep ray angle would appear as a very fast f ratio and could be far outside the makers designed max f ratio.

Purely conjecture and probably just a lot of waffle, unfortunately I don't have the means to test the theory.

( I have downloaded the source image and will have a play after I get back home at the weekend )

William.

[wimvb]

The tif image showed stars with a very strong red halo, near the center of the image. The op also mentions preprocessing having been done on the raw images. We don't know what effect that may have had on the stack. I only addressed the gradients.

[Oddsocks]

Well I had a quick go operating my home PC remotely since it seems we are staying away from home for a few more days, the weather has been so-so the last week and it has kept us indoors but with good weather forecast for the next four days we thought why waste it coming back to the UK with nothing particular to do?

Anyway,

I didn't run all the steps on this I wanted as it is just too painful trying to remote in to my observatory PC with a lot of lag.

Steps were in PixInsight

1: ABE with correction by subtraction using a box size of 50 pixels and separation of 25 pixels, no normalisation. 

2: DBE with correction by division with 20 regular placed samples.

3: STF applied "unchained" and then STF settings transferred to the Histogram Transformation Tool.

4: Image saved.

5: Moved to Photoshop CS6 and Lens Correction tool used to remove small remaining vignette.

6: Haloed red stars selected using "Select" tool and "Colour Range" using sample-in-image to pick up only the saturated bright red stars, selection expanded four pixels and feathered by two pixels.

7: Selected stars adjusted using Image > Adjustments > Hue and Saturation >  to reduce star halo intensity, hue shifted to pale orange and saturation reduced.

8: Boosted background-foreground difference using a pinned curve.

9: Added a little noise reduction.

Called it quits here as remote desktop connection became too laggy to do anything sensible!

Steps to finish might include a masked deconvolution of select areas in PixInsight to attack the coma at the corners, earlier in the process, at the haloed star selection stage, I could have used the replace colour tool first to sample a reference star from one of the online catalogues and replace the halo colour with a true colour sample before adjusting saturation and intensity but otherwise with the source data I would not go too much further.

William.

 

 

[wimvb]

The gradients cleaned up nicely. I have found that stopping down a lens about 2 stops, can dramatically improve the shape of stars. It does make the lens slower of course, but sometimes it can be worth that price.