Some help with processing(Andromeda)

[msacco]

Hi, so I got some pictures of Andromeda yesterday, but I'm having some troubles getting some good results out of it.

I wonder if someone could maybe help me a bit with that.

My issue mostly is that I simply have too much background noise strips which I simply don't manage to get rid of.

This is a picture for example:

Another example after automatic background extractor in PI:

As you can see, there is A LOT of noise strips, I'm using pixinsight, also tried photoshop and gimp, but I don't manage to get much improvements.

I have uploaded all the frames here: https://drive.google.com/drive/u/2/folders/10GIeiJ-CYRV9sOaxlDfUc294t4LrIejI which includes:

Bias, Dark, and lights.

The folders are Lights good which is all the good frames in my opinion, and the Lights folder is simply all the other light frames which in my opinion are a bit worse(doesn't mean they're that bad, and maybe it actually would've been better to use them as well).

Anyway, if someone could give me some assistance, that would be really really useful, as I believe I could get some better results out of it.

General information - The pictures has been taken from a REALLY dark place, around bortle 1-2 I'd say, though Andromeda is fairly low in the horizon at the moment, so that could make a bit of a difference, exposure time - 20 seconds, ISO - 1000.

Thanks a lot!

Edited July 28, 2019 by msacco

[Gerry Casa Christiana]

I think the issue is more the length of your exposures. 20 seconds is not enough to get the detail you need. The background in the first picture is too bright. In photoshop try to get the black levels to around 30 that will help to see more detail in the galaxy. 

Kind regards

Gerry

[Scooot]

Hi msacco,

Firstly I'm by no means an expert but I downloaded your good lights, Bias and darks and had a go at this. As you said your sky is very dark & mine isn't so I was curious.   Flats would have helped and I agree more data would help but it isn't too bad considering.

 

This is the workflow, all processed in Pixinsight.

Integrated Bias, Integrated Darks, Calibrated Lights,

Next Cosmetic Correction, and Debayer

Then I used subframe selector to choose & weight the best lights. The first few weren't so good so I ended up using 69 of your images, so a total Light integration time of about 14 minutes. When I used blink to view your lights I wondered whether you were imaging through a thin layer of high cloud particularly the early ones but as the night went on they improved.

Anyway, after this , Star Aligned, then Drizzle Integrated the lights.

On the Stack, Background Neutralisation and Colour Calibration. (Couldnt get Photometric Colour Calibration to work for some reason).

There was a lot of vignetting because you had no flats so I used Dynamic Background Extraction to remove some of it. Using Divide instead of subtract. Here are the settings.

I ran it twice, the above are the settings for the second run. I then cropped the corners a bit with Dynamic Crop.

After this I applied some Noise Reduction with Multiscalelineartransform using a grey duplicate mask. Followed by a little SCNR as it had a green colour cast.

Next was a masked stretch, followed by more noise reduction, darkening the background and contrast tweaks with curves.

Hope I haven't missed anything too important and this helps. The Light Vortex Tutorials are a good aid.

Edited July 29, 2019 by Scooot

[msacco]

1 hour ago, Gerry Casa Christiana said:

I think the issue is more the length of your exposures. 20 seconds is not enough to get the detail you need. The background in the first picture is too bright. In photoshop try to get the black levels to around 30 that will help to see more detail in the galaxy. 

Kind regards

Gerry

Well, I really don't expect much, what scoot posted is pretty much what I expect, and I think it's decent for starting up.

29 minutes ago, Scooot said:

Hi msacco,

Firstly I'm by no means an expert but I downloaded your good lights, Bias and darks and had a go at this. As you said your sky is very dark & mine isn't so I was curious.   Flats would have helped and I agree more data would help but it isn't too bad considering.

 

This is the workflow, all processed in Pixinsight.

Integrated Bias, Integrated Darks, Calibrated Lights,

Next Cosmetic Correction, and Debayer

Then I used subframe selector to choose & weight the best lights. The first few weren't so good so I ended up using 69 of your images, so a total Light integration time of about 14 minutes. When I used blink to view your lights I wondered whether you were imaging through a thin layer of high cloud particularly the early ones but as the night went on they improved.

Anyway, after this , Star Aligned, then Drizzle Integrated the lights.

On the Stack, Background Neutralisation and Colour Calibration. (Couldnt get Photometric Colour Calibration to work for some reason).

There was a lot of vignetting because you had no flats so I used Dynamic Background Extraction to remove some of it. Using Divide instead of subtract. Here are the settings.

I ran it twice, the above are the settings for the second run. I then cropped the corners a bit with Dynamic Crop.

After this I applied some Noise Reduction with Multiscalelineartransform using a grey duplicate mask. Followed by a little SCNR as it had a green colour cast.

Next was a masked stretch, followed by more noise reduction, darkening the background and contrast tweaks with curves.

Hope I haven't missed anything too important and this helps. The Light Vortex Tutorials are a good aid.

First of all thank you so much for taking time into this, I really appreciate it!

Second, that's some really decent results for the amount of frames I have, so that's already something, do you think there's anything that can be done with the noise strips? I think that kinda ruins the picture, and also why this happens? Is that common for DSLR camera? Or that's more noise than usual?

One last question, for a reason my pixinsight started to produce only Gray images, for example if I integrate the bias/darks/etc, everything comes up as Gray like so:

Do you know why that happens? I tried looking at the settings but didn't see anything related..I also don't think I changed anything.

Thanks again for all the help!!!

[Scooot]

Not sure what noise strips you can see. I can’t see any, not noise anyway. You’re not referring to the dust bands are you? I thought the stack was quite clean. Perhaps there’s some banding, which you can get with Canon Dslrs and there’s a script called canonbanding you can run to remove them but it didn’t seem to need it to me.

Re the grey darks etc, are the individual cr2 images showing that grey as well when you use the screen transfer tool?

Edited July 29, 2019 by Scooot

[Scooot]

I think you’ve just changed that integration image. It looks ok to me, just stretched by the screen transfer a bit too much maybe.

[msacco]

15 minutes ago, Scooot said:

Not sure what noise strips you can see. I can’t see any, not noise anyway. You’re not referring to the dust bands are you? I thought the stack was quite clean. Perhaps there’s some banding, which you can get with Canon Dslrs and there’s a script called canonbanding you can run to remove them but it didn’t seem to need it to me.

Re the grey darks etc, are the individual cr2 images showing that grey as well when you use the screen transfer tool?

Hmm the noise I was referring to is like so:

Noise strip that goes through the whole height of the picture.

I will try that

As for the gray images, I guess it was because I changed to pure raw, and when doing that I first need to debayer right?

8 minutes ago, Scooot said:

I think you’ve just changed that integration image. It looks ok to me, just stretched by the screen transfer a bit too much maybe.

What do you mean by that?

[Scooot]

Oh yes I see what you mean, some sort of vertical streaking. I don't know how to remove that. It's not on the individual subs but there is a lot of differing noise pattern on each one so I guess this is the result of stacking it. The differing noise pattern, if it is that is why I wondered whether you were imaging through a thin layer of cloud. Or perhaps your ISO was too high? What camera are you using?

This is a screen image (with Iphone) of my Dark Integration. I've used the screentransfer tool to view it otherwise it would look totally black.  

Edited July 29, 2019 by Scooot

[msacco]

18 minutes ago, Scooot said:

Oh yes I see what you mean, some sort of vertical streaking. I don't know how to remove that. It's not on the individual subs but there is a lot of differing noise pattern on each one so I guess this is the result of stacking it. The differing noise pattern, if it is that is why I wondered whether you were imaging through a thin layer of cloud. Or perhaps your ISO was too high? What camera are you using?

This is a screenshot of my Dark Integration. I've used the screentransfer tool to view it otherwise it would look totally black.  

Yes that is what I was referring to. I don't think there were clouds really, the weather was really amazing, being in one of the darkest spots in my country as well, no wind, no dew, no clouds(at least not in any weather app I checked or visible to the eye..).

Anyways, I'm using canon eos 6D, it's really possible that the ISO was too high, I don't really know, this is pretty much the first image I've taken(I also took M8, but it was like total 15 mins exposure from my backyard).

Would you recommend me trying to use lower ISO next time?

Edited July 29, 2019 by msacco

[Scooot]

7 minutes ago, msacco said:

Anyways, I'm using canon eos 6D, it's really possible that the ISO was too high, I don't really know, this is pretty much the first image I've taken(I also took M8, but it was like total 15 mins exposure from my backyard).

Would you recommend me trying to use lower ISO next time?

Now I know your camera no, up to ISO 1600/3200 should be fine. Have a look here. http://dslr-astrophotography.com/iso-values-canon-cameras/

If you had great weather maybe it is just noise showing up because I've stretched the image too much for the data. I don't know I'm afraid. Maybe someone else can help.

Thanks for posting your data, it was good to have a go with some data from a dark sky area.   

[msacco]

13 minutes ago, Scooot said:

Now I know your camera no, up to ISO 1600/3200 should be fine. Have a look here. http://dslr-astrophotography.com/iso-values-canon-cameras/

If you had great weather maybe it is just noise showing up because I've stretched the image too much for the data. I don't know I'm afraid. Maybe someone else can help.

Thanks for posting your data, it was good to have a go with some data from a dark sky area.   

Well it is noise I believe, but shouldn't the bias and darks help with that? Maybe flats would've been very helpful here as well?

Anyways, thanks a lot for all the help! And if it was fun for you processing this, you should really try processing other things from people in my country, they're really amazing.

Edited July 29, 2019 by msacco

[Scooot]

2 minutes ago, msacco said:

Well it is noise I believe, but shouldn't the bias and darks help with that? Maybe flats would've been very helpful here as well?

Anyways, thanks a lot for all the help! And if it was fun you processing this, you should really try processing other things from people in my country, they're really amazing.

Darks wouldn’t remove thermal noise. Was it hot?

[msacco]

1 minute ago, Scooot said:

Darks wouldn’t remove thermal noise. Was it hot?

Are you referring to the weather or the camera?

The weather was very good, nice chill with short shirt(short shirt is funny..), and comfortable with long shirt.

As for the camera, I'm not really sure, haven't felt it, but it didn't feel very hot, though I can't really determine.

[Scooot]

5 minutes ago, msacco said:

Are you referring to the weather or the camera?

The weather was very good, nice chill with short shirt(short shirt is funny..), and comfortable with long shirt.

As for the camera, I'm not really sure, haven't felt it, but it didn't feel very hot, though I can't really determine.

Thermal noise is usually generated by the continuous use of the camera, but it would be worse in hot weather. One of the disadvantages of DSLRs although some people modify them to cool them during use. Btw I’m not saying it is thermal noise, just that darks  wouldn’t remove it if it is.

Edited July 29, 2019 by Scooot

[msacco]

2 minutes ago, Scooot said:

Thermal noise is usually generated by the continuous use of the camera, but it would be worse in hot weather. One of the disadvantages of DSLRs although some people modify them to cool them during use. Btw I’m not saying it is thermal noise, just that darks  wouldn’t remove it if it is.

I understand, are there any frames that should handle noise then?

[Scooot]

4 minutes ago, msacco said:

I understand, are there any frames that should handle noise then?

Sorry, I don’t understand your question? Frames that handle noise?

[msacco]

1 minute ago, Scooot said:

Sorry, I don’t understand your question? Frames that handle noise?

Well I guess that if that was the case you would've probably told me that before, but I was referring to bias, darks, flats which might help more with my situation, but I guess not

[Scooot]

4 minutes ago, msacco said:

Well I guess that if that was the case you would've probably told me that before, but I was referring to bias, darks, flats which might help more with my situation, but I guess not

Oh I see. No nothing else. Flats will help with the vignetting.

[msacco]

1 minute ago, Scooot said:

Oh I see. No nothing else. Flats will help with the vignetting.

Ok, thanks a lot!!!

One more question(sorry 😶) - I'm trying the workflow with my pixinsight, and it's so incredibly slow....I really don't know why.

I don't have an amazing PC, but that should be at least decent, I have i5 3470 OC to 3.6 ghz, 12 GB RAM, fast SSD. I mean, I don't expect it to fly, but each process such as integration, registration, etc simply takes around 10-15 minutes.

So I'm spending like 2-3 hours on the stacking process, and that simply feels like it doesn't go anywhere.

Is that the same to you as well?

[Scooot]

3 minutes ago, msacco said:

Ok, thanks a lot!!!

One more question(sorry 😶) - I'm trying the workflow with my pixinsight, and it's so incredibly slow....I really don't know why.

I don't have an amazing PC, but that should be at least decent, I have i5 3470 OC to 3.6 ghz, 12 GB RAM, fast SSD. I mean, I don't expect it to fly, but each process such as integration, registration, etc simply takes around 10-15 minutes.

So I'm spending like 2-3 hours on the stacking process, and that simply feels like it doesn't go anywhere.

Is that the same to you as well?

It is quite slow. Calibrating the lights will take a while as will stacking them.

There’s a batch process which I believe is quicker but I’ve never used it.

[msacco]

2 minutes ago, Scooot said:

It is quite slow. Calibrating the lights will take a while as will stacking them.

There’s a batch process which I believe is quicker but I’ve never used it.

I've mostly heard that it's not as good to use the batch process. I'll just try and see what I can do to improve it, thanks, and sorry for all the dumb questions

[Scooot]

2 minutes ago, msacco said:

I've mostly heard that it's not as good to use the batch process. I'll just try and see what I can do to improve it, thanks, and sorry for all the dumb questions

No probs, there’s no dumb questions  

[msacco]

Just now, Scooot said:

No probs, there’s no dumb questions  

Just dumb people?

[wimvb]

Regarding the banding:

They are a result of the camera read noise showing through, and indicate under exposure. If they also show in your calibrated darks, you could test stacking without darks altogether. Use cosmetic correction to remove excessive hot pixels. 

The CBR script will only remove horizontal bands. Quick rotate your image 90 degrees, use the script, turn back. If the bands aren't exactly horizontal, the script won't work properly. 

[wimvb]

I've also downloaded your data set and am playing around with your image. Impressions so far:

• You have a full fram camera, and I'm not sure that the scope will illuminate the whole chip.
• Do you use a coma corrector? Even that would need to illuminate the whole chip. Vignetting can only be corrected up to a certain level.
• Calibration frames (darks, bias, flats) can never decrease the random noise in images. Only averaging during stacking can do that.
• The integration process is so slow because you have such large images (20 Mp). You can try increasing the memory settings in PI, but in all honestly, I don't think that will improve things much.
• Your unstretched master doesn't show any stars. This and the vertical banding are tell tale signs of under exposure. The bands are the read pattern of the sensor.

If you want to fully utilise the dynamic range of the camera, you need to increase the exposure time. At a dark site, even at ISO 1000, you should be able to use an exposure time of several minutes. A general rule for DSLR imaging is to have the peak of the histogram at 1/4 to 1/3 from the left edge of the display. But at a truly dark site, that may be impossible. You should at least have the brightest stars at full intensity. If you want to keep star colour even in the brightest stars, you can decrease the exposure somewhat once you've determined which exposure starts to blow out star cores.

On 29/07/2019 at 20:16, msacco said:

One last question, for a reason my pixinsight started to produce only Gray images, for example if I integrate the bias/darks/etc, everything comes up as Gray

Calibration images should be gray scale. Bias, darks and flats are never deBayered, because they correct the light frames pixel by pixel. If these calibration images were deBayered, their pixel values would change.