Processing guidance for M31 & M33 with lots of moonlight

[keramos]

Morning Everyone,

So last week was my first ever proper imaging experience, got two lovely nights and managed to take image of M31 & M33.

Idea was to get the setup of the gear correct and practise polar alignment and guiding...anything more would be added bonus.

Took 30 subs @ 120s with bias and darks, didn't take any flats (for both M31 and M33)

I recognise that I need to improve polar alignment and guiding, those are things to work on over time, though was pleased with data I gathered with promise of improvement. 

So equipment, 

• WO Redcat 51
• ASIAIR pro 
• Nikon D3300 unmodded
• ES iexos 100 mount and stock tripod (going to swap the tripod for EQ5)
• IDAS LPS - D2 filter
• ASI174MM as guide camera, ASI mini guide scope

However, I need Help! with the following:

Pre processing the data. 

• At this point I have only stacked the images in Deep Sky Stacker using the recommended settings as a start (FITs file saved by ASIAIR).
• DSS debayers the image (I've used generic RGGB for FITs), resulting image has greenish tint hardly any colour even though its a colour DSLR.  is this normal?

Moonlight

• Problem was lots of Moonlight, and to make matters worst I picked target close to moon glow 🤣.
• I've read that LP filter will not help with moonlight but can this be fixed in post processing?

Processing Software

• Lastly any recommended tutorials for either Pixinsight or Photoshop, something for a dummy with detailed steps.
• I'll probably get trials for both for the moment and then decide.

 

I've attached the stacked Tif file for M31, M33 also attached, this is much worst as it completely washed out by the Moon

Any help advice would be welcome, obvious one would be don't choose targets close to the moon 😆

M31_27102020.tif M33_28102020.tif

[ultranova]

Hi, I am unable to download and look at the pictures at the moment,

But I can offer some advice,

In General broadband targets when the moon has come up and is less than

a 1/4 you can get some good results, if you pick a target facing as far away

from the moon as possible, some broadband light pollution filters, will help a little

as well, but not by much.

• DSS debayers the image (I've used generic RGGB for FITs), resulting image has greenish tint hardly any colour even though its a colour DSLR.  is this normal?

This is quite normal, The color Bayer matrix in your camera by its very definition RGGB, ( this is a filter that sits on top of your pixels )it has only one red, one blue but two green covering each pixel, , consequently,  your camera is more green sensitive than any other color.

If the moon is out, the data you collect, the signal from the target needs to be stronger than glow coming off the sky glow from the moon,

otherwise the sky glow will wash out the image, or you will get bad gradients in your image.

If you live in a area, where the light pollution is low, and the moon is less than 1/4, you should get some fair results,

but if you live in a moderate to high area of light pollution, even with the moon not out it will seriously affect your image

chuck the moon in the mix as well with broadband  imaging, its a real struggle to get a decent image.

I have been there and done that so to speak.

possibly go down the route of getting your camera modified, at least this would enable you to use narrow band

imaging, to a certain degree.

Unfortunately, no amount of processing can get the detail out of a image if it is drowned in sky glow,

or has a really bad gradient,

Don't get me wrong,, I am only talking about really bad sky glow, caused by the moon, or street lighting,

you will be amazed to what information can be drawn out of a image with software,

On you tube there are loads of good tutorials , try this one for a starting point

https://www.youtube.com/watch?v=W66Okjh4P2g

Patience and persistence, is what is required, when i get back later i shall look at your images. 

hope this helps a little

Paul

 

 

 

[wornish]

Had a very quick go with your M31 image. Processed in Pixinsight and tweaked in PS

 

Stars are good shape.

 

 

[keramos]

2 hours ago, ultranova said:

Hi, I am unable to download and look at the pictures at the moment,

But I can offer some advice,

In General broadband targets when the moon has come up and is less than

a 1/4 you can get some good results, if you pick a target facing as far away

from the moon as possible, some broadband light pollution filters, will help a little

as well, but not by much.

• DSS debayers the image (I've used generic RGGB for FITs), resulting image has greenish tint hardly any colour even though its a colour DSLR.  is this normal?

This is quite normal, The color Bayer matrix in your camera by its very definition RGGB, ( this is a filter that sits on top of your pixels )it has only one red, one blue but two green covering each pixel, , consequently,  your camera is more green sensitive than any other color.

If the moon is out, the data you collect, the signal from the target needs to be stronger than glow coming off the sky glow from the moon,

otherwise the sky glow will wash out the image, or you will get bad gradients in your image.

If you live in a area, where the light pollution is low, and the moon is less than 1/4, you should get some fair results,

but if you live in a moderate to high area of light pollution, even with the moon not out it will seriously affect your image

chuck the moon in the mix as well with broadband  imaging, its a real struggle to get a decent image.

I have been there and done that so to speak.

possibly go down the route of getting your camera modified, at least this would enable you to use narrow band

imaging, to a certain degree.

Unfortunately, no amount of processing can get the detail out of a image if it is drowned in sky glow,

or has a really bad gradient,

Don't get me wrong,, I am only talking about really bad sky glow, caused by the moon, or street lighting,

you will be amazed to what information can be drawn out of a image with software,

On you tube there are loads of good tutorials , try this one for a starting point

https://www.youtube.com/watch?v=W66Okjh4P2g

Patience and persistence, is what is required, when i get back later i shall look at your images. 

hope this helps a little

Paul

 

 

 

Hi Paul, 

Thanks for the reply, its reassuring that nothing is wrong with the equipment at one point I thought that might be the case compounded by the skyglow.

I'm in Bortle 5 skies according to Clear Outside but then add the street lights (LED) and ambient light from the houses etc...it gets worst.

Tbh, after I set up the rig, first thing I went for was M31...and the moment I saw the smudgy image (which looked rather nice on ASIR Air pro app on iPad) I was eager to image...my excitement got the better of me..

I look forward to further feedback and will definitely look at the youtube video after work.

 

[keramos]

1 hour ago, wornish said:

Had a very quick go with your M31 image. Processed in Pixinsight and tweaked in PS

 

Stars are good shape.

 

 

Hi Dave,

That is rather impressive, I'm glad/pleased and reassured that I'm heading in the right direction.

I reckon lots for me to learn in terms of image processing, do you have predefined scripts in pixinsight that run through the stacked image?

I found this earlier https://www.lightvortexastronomy.com/tutorial-pre-processing-calibrating-and-stacking-images-in-pixinsight.html

Do you think its a good tutorial for beginner? any recommendations?

Thank you for the processing, trust me its very reassuring.

Quote

Stars are good shape.

Earlier I mentioned that Polar Alignment and Guiding needed some work..

PA -

• Total error as reported by ASIAIR app was 27", I read somewhere or saw in a video that about 10" for a set like mine (WO Red cat 51 f4.9) should suffice.
• Having said that, the tripod that comes with Explorer scientific iEXOS 100 is rubbish for many reasons, now I've got the mount on EQ5 triop and its rock solid, heavier to carry to the location (car to open space) from where I image but will pay dividends.

Guiding

• Total RMS error was around 2.5", with dithering it was all over the place.
• I've read sub 1" is good
• M31 was with dithering
• M33 dithering switched off as it would complain about settle time.

Quote

Processed in Pixinsight and tweaked in PS

Lastly if you don't mind me asking, which bits are done in Pixinsight and PS

 

[vlaiv]

Only major issue that I found with this data is lack of flats.

This is processing that I did in ImageJ and Gimp 2.10.

If you wish, I can do a step by step guide, but I'm not sure if you'll be interested in that since you use PI and PS.

[keramos]

43 minutes ago, vlaiv said:

Only major issue that I found with this data is lack of flats.

This is processing that I did in ImageJ and Gimp 2.10.

If you wish, I can do a step by step guide, but I'm not sure if you'll be interested in that since you use PI and PS.

Hi Vlaiv,

That would be very much appreciated, I still don't have PS or Pixinsight installed (trial).

And I'm sure it would be helpful and beneficial to other beginners like me.

I didn't take flats as I disturbed the focus on the first night while tearing down the equipment, then did the exact same thing on the second night.

ps: like how to you the background rather dark, excellent.

[wornish]

2 hours ago, keramos said:

Hi Dave,

That is rather impressive, I'm glad/pleased and reassured that I'm heading in the right direction.

I reckon lots for me to learn in terms of image processing, do you have predefined scripts in pixinsight that run through the stacked image?

I found this earlier https://www.lightvortexastronomy.com/tutorial-pre-processing-calibrating-and-stacking-images-in-pixinsight.html

Do you think its a good tutorial for beginner? any recommendations?

Thank you for the processing, trust me its very reassuring.

Earlier I mentioned that Polar Alignment and Guiding needed some work..

PA -

• Total error as reported by ASIAIR app was 27", I read somewhere or saw in a video that about 10" for a set like mine (WO Red cat 51 f4.9) should suffice.
• Having said that, the tripod that comes with Explorer scientific iEXOS 100 is rubbish for many reasons, now I've got the mount on EQ5 triop and its rock solid, heavier to carry to the location (car to open space) from where I image but will pay dividends.

Guiding

• Total RMS error was around 2.5", with dithering it was all over the place.
• I've read sub 1" is good
• M31 was with dithering
• M33 dithering switched off as it would complain about settle time.

Lastly if you don't mind me asking, which bits are done in Pixinsight and PS

 

 

The Light Vortex Tutorials are very good

 

I have attached my Pixinsight crib sheet built up over quite a while.  Its far from ideal but the post processing steps on Page 12 are what I follow usually.  There are a few people who have also shared their process steps on this  forum so if you do a search you can find them.  Anyway this is in PDF format so hope you can read it.

 

I didn't follow all the steps on my crib sheet on Pixinsight, I missed out he noise reduction ones. Then the only thing I did in PS was flip the image horizontally and vertically as that's the way I like to show it.  Then I reduced the noise and colour noise using the Topaz DeNoise AI plug-in its the best I have found. I also increased the contrast a little to darken the background.  The one that Vlaiv did above had a much darker background than mine but that is always down to individual taste.

 

Have fun.

Pixinsight Process Steps.pdf

Edited November 3, 2020 by wornish

[keramos]

2 hours ago, wornish said:

 

The Light Vortex Tutorials are very good

 

I have attached my Pixinsight crib sheet built up over quite a while.  Its far from ideal but the post processing steps on Page 12 are what I follow usually.  There are a few people who have also shared their process steps on this  forum so if you do a search you can find them.  Anyway this is in PDF format so hope you can read it.

 

I didn't follow all the steps on my crib sheet on Pixinsight, I missed out he noise reduction ones. Then the only thing I did in PS was flip the image horizontally and vertically as that's the way I like to show it.  Then I reduced the noise and colour noise using the Topaz DeNoise AI plug-in its the best I have found. I also increased the contrast a little to darken the background.  The one that Vlaiv did above had a much darker background than mine but that is always down to individual taste.

 

Have fun.

Pixinsight Process Steps.pdf 284.59 kB · 2 downloads

Thanks Dave,

Does the process change a lot between DSLR and OSC e.g. ZWO cameras?

[Aramcheck]

When starting out I found Richard Bloch's youtube videos on Basic Processing and Stacking useful. Also Mitch's 12 video series. I've subsequently made more use of Warren A. Keller's "Inside Pixinsight" book, which I find indispensible. Light Vortex is good too - particularly like his bit on SubFrameSelection.

Had a quick play with your M33 data - it looks to me like the focus was slightly off. Did you use the Bahtinov mask to focus?

Cheers
Ivor

[keramos]

1 hour ago, Aramcheck said:

When starting out I found Richard Bloch's youtube videos on Basic Processing and Stacking useful. Also Mitch's 12 video series. I've subsequently made more use of Warren A. Keller's "Inside Pixinsight" book, which I find indispensible. Light Vortex is good too - particularly like his bit on SubFrameSelection.

Had a quick play with your M33 data - it looks to me like the focus was slightly off. Did you use the Bahtinov mask to focus?

Cheers
Ivor

Hi Ivor,

Thank you for looking at M33, I'm must say all you guys are wonderful and very helpful.

One generally gets to see end results and doesn't know what the initial data looks like...I was in despair before I started this topic but now I can't wait to get my hands on some of the softwares (at least Gimp is free) also can't wait to get out again.

• I did use the Bahtinov mask to focus before PA, guiding with this was not particularly the best and dithering was refusing to settle (I still need to find my way with the parameters in ASIAIR pro) and was a bit breezy too, I'm sure that would affect the subs.
• There are some circular artefacts, which don't appear on the unprocessed tif file (they are quite well defined circles), I reckon this could be eliminated using flats?
• Could it be dew?
• To get more detail in M33 = more subs?

Still waiting for trial license for pixinsight, been more than 24hrs, got the initial email still waiting for verification.

Looks like I'm going to do my bit during the lockdown by going through the tutorials 😃

[vlaiv]

Ok, I'll now do a step by step processing "tutorial" so I apologize in advance for long posts and plenty of them. Hopefully people will learn something from this.

Tools used - Gimp 2.10 and Fiji latest version (Fiji Is Just ImageJ - one of those recursive acronyms - just ImageJ loaded with plugins. Probably stock ImageJ will work as well as I use pretty much basic things). You'll also need a plugin that I wrote. I'll include it for download:

Extract_Background.class

This should be placed in Plugins/Astro folder inside Fiji/ImageJ installation directory (create Astro folder inside Plugins folder as it won't exist by default). Restart ImageJ/Fiji in order for plugin to load.

Part one - data processing part.

1. Open M31.tif in Gimp

2. Use Color / Components / Decompose menu item. Select RGB model and uncheck decompose to layers. This will create three additional monochromatic images - each channel will be as mono image, there will be red, green and blue channel image.

3. Export As each of those channel images - as fits files (Gimp 2.10 handles fits files)

4. Open all three channel fits in ImageJ (or Fiji, but I'll just write ImageJ from now on ...). Open red then green then blue (keep that order as will matter later on)

5. Use Image / Stack / Images to stack menu command, use default settings (just click ok). This should create a stack of images.

6. Use Image / Adjust / Brightness&Contrast menu item to display Brightness and Contrast window. This is "screen transfer function" window - it won't change data just display range, so you can move sliders to display different part of histogram of the image.

7. Use rectangle selection tool to make a crop section (first icon in main program window / toolbar). There are some stacking artifacts on the left side and on the bottom and we need to remove those. Make larger rectangular selection of almost whole image except dozen or so pixels at each side (top, bottom, left and right).

8. Right mouse click on image and select "Duplicate" from popup menu. Select duplicate whole stack check box. This will make duplicate of selected area as new stack.

9. Close original stack as we won't be needing it any more

10. Use Image / Transform / Bin menu item. Use default settings (X and Y factor is 2, Z factor is 1, method is average). This will bin data x2. This image came from OSC sensor and there simply is no point in having interpolated resolution which is empty resolution. If you want to get some more SNR at this point, you can bin by factor of x3 - just change X and Y value to 3. This will create smaller image but higher SNR.

11. Image / Stack / Stack to images menu command - this should split the stack again into individual images.

I will describe next section on one image / channel, but it needs to be performed on all three images.

12. Select image that you are working on and run Plugin / Astro / Extract background menu command. Uncheck produce background map (you can leave it checked if you want to see what pixels algorithm considers to be background) and leave produce gradient map checked. Leave default parameters of 3 and 10. This command will produce new image that will be just gradient image.

13. Hover mouse over gradient image and examine pixel values - displayed in main window. If gradient image values in the corners, where gradient starts and ends are very small values - like something e-7 or something e-8, you will be done with this procedure, if not, you'll have to repeat it. Depending on type of gradient, it usually takes something like 5-6 round of removal to get nice background.

14. Process / image calculator. Select first image to be image you are working on (red channel for me at the moment of writing) and select second image to be gradient image (gradient image will have same title with "Gradient map of" prepended). Select operation subtract and uncheck create new window and check 32bit precision. this will subtract gradient from our image (it will also change how image looks if you set certain range on screen transfer function so you can inspect image by selecting it again and changing sliders in brightness and contrast).

15. Close gradient map of .... image. If at step 13 you found that gradient is still significant, select again the same image and repeat procedure (go to step 12). Else, proceed with step 16.

It took 4-5 rounds to get this value for red channel for me. I can now proceed with next step.

16. Again select image that you are working on and do Plugin / Astro / Extract background. This time uncheck produce gradient map and check produce background map. Leave other parameters the same. This will create black and white image where "foreground" pixels will be white and "background" pixels will be black (255 and 0 values - nothing in between)

17. Select the image that we work on and run Analyze / Measure command. This should bring window with basic stats for the image. Important ones right now are Mean, Min and Max. If one of those is missing, run Analyze / Set Measurements and check ones that are missing then run Analyze / Measure again. Note minimum value in the image (Min). for me, it is -0.0017009....

18. Process / Math / Add and enter any number that is slightly larger than this - I'll go with 0.01. We want all values in our image now to be positive, so we want any number that is larger (by value) than minimum value (negative one) in the image. You can run Analyze / Measure again to see the change:

and close results window now ...

19.  Process / image calculator. First image is our image that we work on, second image is background map of that image. Operation is add, check create new window, check 32bit precision. This will create new image. This image will be our artificial flat. You can right click on it and rename it to something meaningful like "working_image_flat" - in my case "m31_red_flat". Close background map of ... image now.

20. On this new flat image run Process / Math / Macro and type this line in:

Quote

if(v>1) v=NaN;

like this:

This will set all values that are "foreground" pixels to non existing value (Not a Number).

21. Run Process / Noise / Remove NaNs and enter value of 100 on this flat image. This will take some time to complete. It just fills in missing values with median of surrounding pixel values (in circle of 100 pixels).

22. run Process / Filters / Median with value of 10 on this flat image.

23. Run Analyze / Measure on this image and note max value

24. Process / Math / divide and use max value that you measured in step 22

25. Select image that we work on and Process / Image calculator. First image is our working image (red channel), second image is our flat image. Operation is divide. 32bit precision and uncheck new image. This will apply our fake flat. After that you can close flat

26. Again do one round of background removal (steps 12 - 14) to remove offset that we added for artificial flat to work properly.

Now repeat above with two remaining channels (starting at step 12). Just a note - if Process / Noise / Remove NaNs does not "close" the image completely - run it again until there are no "missing" pixels in the image.

27. Image / stack / images to stack - this will create stack again.

28. Find star with color index 0.48 in Stellarium .

29. Find matching star in your stack and make circular selection around it - like this:

30. Image / Stack / Measure stack, you should get something like this:

31. Unselect and Image / Stack / Stack to images to split images

32. Divide each channel image with corresponding measured value (Process / Math / divide and then enter corresponding number)

33. Image / Stack / Images to Stack and then Image / Stack / Measure stack

34. Process / Math / Divide - take the least out of Max values. Process all three slices when asked

35. Again measure stack, and note highest of min values

36. Process / Math / Max - set to 1, Process / Math / Min - set to highest of last measured min values (in my case -0.02878031 as it is highest of the three).

37. If you measure once more after this you should get same min / max values for all three channels like this:

38. Save each as fits ...

Part one done

[vlaiv]

Part 2 - simple gimp composing and stretch.

1. Load all three fits saved in previous part.

2. Do Color / Channels / Compose with RGB model and assign each image to corresponding channel

3. Do simple levels stretch - Color / Levels

Move top slider until you see that M31 core is starting to saturate. Move middle slider to left, like this:

4. Do another round of levels, this time bringing left most slider at the base of histogram hill like this:

5 Do a round of Curves like this:

6. Due to artificial flats - there is distinct red cast on background everywhere except for galaxy. Select it and remove it by curves on red channel.

And voila, here is resulting image:

[keramos]

Hi vlaiv,

This is brilliant, thanks for the step by step guide, this is very helpful.

I'm sure will be helpful for lot of people here. 👍