Processing Help with noise & gradients

[CraigD1986]

Hi All.

I had a go at imaging M45 with my Canon 5D Mk IV, Sigma 150-600 (500mm @ f5.6) and Star Adventurer. I captured about 60mins of data and used dark, flat and bias frames. I stacked in DSS and edited in Adobe Photoshop. Light frames were 60 seconds, f5.6, ISO 800.

I have to do a lot of stretching to bring out the detail but can't figure out a way to prevent the excessive colour noise, banding and gradients. Can anyone advise as to how I can improve on this?

Thanks in advance!

 

Edit-1.tif

Edited December 27, 2021 by CraigD1986
image file type change

[bottletopburly]

More data will definetley help nearer 4 hrs will help noise ,also if you can dither between frames will help with walking noise ,if posting the tif best post the file straight from DSS no editing done 

[CraigD1986]

Thanks for your advice. I had about double the amount of data originally but the batteries in my star adventurer ran out half way through the session so I lost half of the lights immediately.

I've since purchased an ASiAir Plus, guide camera and scope so hope to be able to increase exposure time and reduce ISO next time with the aid of tracking. Obviously, it's been cloudy since it arrived though. I believe I will be able to dither with the ASiAir too but that's something I still need to learn about.

Attached is the TIFF straight out of DSS if anyone is interested.

PleiadesAutosave.tif

[Stu Wilson]

Had a quick go for you.

Need alot more data to get rid of noise though.

Hope ive done it justice.

Cheers.

[CraigD1986]

That looks great, much sharper than my attempt.

I’ll have another go at getting some more data the next time the clouds are gone.

I’ve just been reading bout dithering with a star adventurer. Am I right in thinking I don’t need dark files if I dither? Darks really chip into the imaging time but what I read suggests that the dithering process would also waste a fair bit of time.

Would i be better trying to get longer exposures with a lower ISO with tracking, or just more 60 second exposures?

[bottletopburly]

Dithering does not take long and benefits outweigh lost time , you can dither every other sub , 

[bottletopburly]

Had a play in startools

[Richard_]

2 hours ago, CraigD1986 said:

Thanks for your advice. I had about double the amount of data originally but the batteries in my star adventurer ran out half way through the session so I lost half of the lights immediately.

I've since purchased an ASiAir Plus, guide camera and scope so hope to be able to increase exposure time and reduce ISO next time with the aid of tracking. Obviously, it's been cloudy since it arrived though. I believe I will be able to dither with the ASiAir too but that's something I still need to learn about.

Attached is the TIFF straight out of DSS if anyone is interested.

PleiadesAutosave.tif 263.5 MB · 7 downloads

The ASI Air will be a game changer for you! I bought the Pro version last year and it really helped make things easier. 

You can enable dithering in the ASI app and you can choose how often to do this (eg after every X number of images). I'd recommend starting with every 3rd image and decrease if you feel its not working for you. There is a timer to wait for stability, I'd set this to the minimum time of 30 seconds (eg if you don't achieve the desired stability by this point, it will start imaging). When this was not enabled, I'd be waiting minutes before it would start shooting again, now its 30 seconds each time. 

The amount of dither really depends on camera and telescope. For my Redcat (250mm focal length) and ASI533 camera I dither about 10", whereas for my FLT120 (780mm focal length) this is way too much, so I set it to 2" or 3". You can have a play and see how much your images "move" when you scan through each sub. 

Some other nice benefits of the ASI Air, you can power the star Adventurer and Canon with it. The star Adventurer has a USB-mini port which you can plug into the ASI USB 2.0 port. You can also buy a dummy battery for Canon cameras on Rother Valley Optics website (below). You can plug this directly into the ASI Air using a 12v DC cable. So if you power the ASI Air from mains electric, you never have to worry about batteries running out again for your mount or camera 🙂

https://www.rothervalleyoptics.co.uk/pegasus-battery-couplers.html

[vlaiv]

4 hours ago, CraigD1986 said:

I captured about 60mins of data and used dark, flat and bias frames.

Well, your calibration is not very good. I can deal with noise in various ways - but I can't fix poor calibration.

When I stretch image hard - I get this:

Here - look, almost no noise:

but calibration issues remain.

[Stu Wilson]

This was my version with 3.5 hours. Was done with SW200p with no coma corrector.

Just to show difference more time makes, but there is still alot of noise in it.

Edited December 27, 2021 by Stu Wilson

[Swillis]

Well they all look better than my effort so far. Looking forward to getting more data once the clouds clear. @Stu Wilson love the Look of the diffraction spikes

[Stu Wilson]

5 minutes ago, Swillis said:

Well they all look better than my effort so far. Looking forward to getting more data once the clouds clear. @Stu Wilson love the Look of the diffraction spikes

newtonian reflector spikes

 

[CraigD1986]

2 hours ago, Richard_ said:

The ASI Air will be a game changer for you! I bought the Pro version last year and it really helped make things easier. 

You can enable dithering in the ASI app and you can choose how often to do this (eg after every X number of images). I'd recommend starting with every 3rd image and decrease if you feel its not working for you. There is a timer to wait for stability, I'd set this to the minimum time of 30 seconds (eg if you don't achieve the desired stability by this point, it will start imaging). When this was not enabled, I'd be waiting minutes before it would start shooting again, now its 30 seconds each time. 

The amount of dither really depends on camera and telescope. For my Redcat (250mm focal length) and ASI533 camera I dither about 10", whereas for my FLT120 (780mm focal length) this is way too much, so I set it to 2" or 3". You can have a play and see how much your images "move" when you scan through each sub. 

Some other nice benefits of the ASI Air, you can power the star Adventurer and Canon with it. The star Adventurer has a USB-mini port which you can plug into the ASI USB 2.0 port. You can also buy a dummy battery for Canon cameras on Rother Valley Optics website (below). You can plug this directly into the ASI Air using a 12v DC cable. So if you power the ASI Air from mains electric, you never have to worry about batteries running out again for your mount or camera 🙂

https://www.rothervalleyoptics.co.uk/pegasus-battery-couplers.html

Thanks. That’s some really great info. I didn’t realise the ASiAir doesn’t come with a power adapter so I had to get one from Amazon but it won’t be here until Wednesday. Will have a play with dithering and guiding as soon as I can. Great to hear about the power options for the mount and camera so I’ll look at that too.

2 hours ago, vlaiv said:

Well, your calibration is not very good. I can deal with noise in various ways - but I can't fix poor calibration.

When I stretch image hard - I get this:

Here - look, almost no noise:

but calibration issues remain.

Please can you expand on what you mean by not very good calibration? How can you tell? I just took the exposures, loaded them into DSS and let it do it’s thing. I’m still a beginner so have a lot to learn.

[vlaiv]

6 minutes ago, CraigD1986 said:

Please can you expand on what you mean by not very good calibration? How can you tell? I just took the exposures, loaded them into DSS and let it do it’s thing. I’m still a beginner so have a lot to learn.

You have uneven background due to vignetting even after calibration and that is a sign that calibration was not successful.

Point of calibration of the data is to remove those imperfections.

There could be several sources of this:

1. Improper calibration procedure performed

2. Proper calibration performed but with calibration files that are themselves flawed somehow.

As far as first point - you'll hear very different opinions on how to perform calibration of DSLR data. Some will say - use darks, some will say don't use darks, some will say use bias and so on...

I'll write about what is proper calibration and why it is hard to do with DSLR - and the way to do it with DSLR.

Proper calibration is one that uses:

- darks matching lights (in exposure, temperature and capture settings like ISO and so on). These are taken in total absence of light

- flats. With flats you have to be careful that they are properly exposed - enough signal but no so much that you end up with saturation / clipping

- flat darks. same as darks but matching flat exposure (same exposure length, iso and so on as flats)

Problem with DSLR is that you can't guarantee matching darks because you don't have control over temperature of the sensor. There is no set point cooling, and even small change in temperature can have significant impact on dark current (dark current doubling temperature is about 6°C - so even degree or two will change dark current by 10-20%).

Luckily there is way to compensate for this (and it is mostly successful) - and that is "dark optimization". It works only if you have good bias subs for your DSLR and if your bias behaves properly. With some cameras that is not the case.

This technique relies on math and computer to determine multiplier for dark current - in reality to do a best guess on it so that dark current in dark subs matches dark current in light subs even if they are taken at different temperature.

In order to do it in DSS - you need darks (which you have), bias (which you have), flats (which you have) and flat darks (which you don't have). If your flat exposures were reasonably fast (less than 1-2s) and were taken at same ISO setting as lights and darks - then you can use your bias subs as flat darks - although that is not proper calibration procedure (but it will have to do in this case, unless you can shoot another set of flat darks at same temperature as flats - again problem of set point).

We can only hope that bias for your camera is proper bias.

You need to check dark optimization check box in DSS stacking settings. Use simple Average stacking method for darks, bias and flats.

In the end - I'll address briefly point 2 from the beginning - proper calibration but with improper calibration files.

This means that you follow proper calibration procedure but something in your calibration files is wrong and that results in wrong calibration.

You can have light leak while taking darks for example (even IR leak). Always cover viewfinder of your DSLR when taking bias and darks. There is small plastic cover for it that came with your camera.

Make sure you match ISO settings and exposure length between lights / darks and flats / flat darks. Don't over or under expose your flats. Make sure you have good flat panel. If you have mechanical shutter - take longer flats exposure so that motion of mechanical shutter does not produce artifacts in flats (gradient from closing shutter).

 

 

[Richard_]

1 hour ago, CraigD1986 said:

Thanks. That’s some really great info. I didn’t realise the ASiAir doesn’t come with a power adapter so I had to get one from Amazon but it won’t be here until Wednesday. Will have a play with dithering and guiding as soon as I can. Great to hear about the power options for the mount and camera so I’ll look at that too.

Yeah I was a bit miffed about no power supply too! What amperage power supply did you order? 12V @ 2A should work  fine for a DSLR and Star Adventurer, but if you add in dew heaters and replace the DSLR with a cooled camera in future, 2A may not be enough juice (see extract from manual below). I've been using a 12V @ 5A for the past year and it was enough to power the Star Adventurer, two dew heaters and the cooling on my ASI533 camera. The warning in the manual about powering mounts is aimed  towards heftier GoTo mounts like the Skywatcher EQ6-R as these have higher power requirements than the Star Adventurer. 

Sorry, my replies are starting to stray from the original topic! There's a few nuances with the ASI Air and some less obvious tips which I wish I knew when I started with it. Feel free to send me a PM if you have any questions which Google or YouTube aren't answering as I'm happy to share my experiences where I can 😊

[alacant]

On 27/12/2021 at 15:29, CraigD1986 said:

banding and gradients. Can anyone advise

Hi

Lovely shot. Some nice detail emerging. A few bits of hands on advice...

5D iv noise? Bands? So lose the dark frames. Use flat and bias frames only. Add some dithered -say about 12 pixels- frames next time out. process the whole lot of them again without the dark frames and stack using a clipping algorithm. Most of the noise will be gone. Instead of DSS, try Siril.

Lastly, add sheets of paper between light source and lens to adjust the exposure for the flat frames to at least 1 second and take them in a dark room.

(Had a go at the stars. Maybe have a look at polar alignment?)

Cheers and HTH

 

Edited December 28, 2021 by alacant

[ONIKKINEN]

24 minutes ago, alacant said:

Had a go at the stars. Maybe have a look at polar alignment?

Looks like the elongation is along the RA axis, which means the exposure was too long and affected by periodic error.

30s exposures would probably be more reliable for unguided. The only way to get rid of this limitation is to guide.

[CraigD1986]

Thanks so much to everyone for the replies. As a novice, this kind of help is invaluable and I really appreciate it. There is a lot to take in here so I'll try and address everything one by one so I'm better prepared for the next clear night.

13 hours ago, vlaiv said:

You have uneven background due to vignetting even after calibration and that is a sign that calibration was not successful.

Point of calibration of the data is to remove those imperfections.

There could be several sources of this:

1. Improper calibration procedure performed

2. Proper calibration performed but with calibration files that are themselves flawed somehow.

As far as first point - you'll hear very different opinions on how to perform calibration of DSLR data. Some will say - use darks, some will say don't use darks, some will say use bias and so on...

I'll write about what is proper calibration and why it is hard to do with DSLR - and the way to do it with DSLR.

Proper calibration is one that uses:

- darks matching lights (in exposure, temperature and capture settings like ISO and so on). These are taken in total absence of light

- flats. With flats you have to be careful that they are properly exposed - enough signal but no so much that you end up with saturation / clipping

- flat darks. same as darks but matching flat exposure (same exposure length, iso and so on as flats)

Problem with DSLR is that you can't guarantee matching darks because you don't have control over temperature of the sensor. There is no set point cooling, and even small change in temperature can have significant impact on dark current (dark current doubling temperature is about 6°C - so even degree or two will change dark current by 10-20%).

Luckily there is way to compensate for this (and it is mostly successful) - and that is "dark optimization". It works only if you have good bias subs for your DSLR and if your bias behaves properly. With some cameras that is not the case.

This technique relies on math and computer to determine multiplier for dark current - in reality to do a best guess on it so that dark current in dark subs matches dark current in light subs even if they are taken at different temperature.

In order to do it in DSS - you need darks (which you have), bias (which you have), flats (which you have) and flat darks (which you don't have). If your flat exposures were reasonably fast (less than 1-2s) and were taken at same ISO setting as lights and darks - then you can use your bias subs as flat darks - although that is not proper calibration procedure (but it will have to do in this case, unless you can shoot another set of flat darks at same temperature as flats - again problem of set point).

We can

a is proper bias.

You need to check dark optimization check box in DSS stacking settings. Use simple Average stacking method for darks, bias and flats.

In the end - I'll address briefly point 2 from the beginning - proper calibration but with improper calibration files.

This means that you follow proper calibration procedure but something in your calibration files is wrong and that results in wrong calibration.

You can have light leak while taking darks for example (even IR leak). Always cover viewfinder of your DSLR when taking bias and darks. There is small plastic cover for it that came with your camera.

Make sure you match ISO settings and exposure length between lights / darks and flats / flat darks. Don't over or under expose your flats. Make sure you have good flat panel. If you have mechanical shutter - take longer flats exposure so that motion of mechanical shutter does not produce artifacts in flats (gradient from closing shutter).

 

 

Here is my procedure for calibration frames:

DARKS: Lens cap on and eyepiece cover on, everything exactly the same as the lights and at the end of the imaging session or in-between if clouds start to roll in. Just set the intervalometer off again and get as many as I have time for.

The whole DSLR sensor temperature thing has always bothered me but I've always read that dark files are absolutely essential. I'm sure the sensor temperature must fluctuate, even just as it gets colder outside so this has never really made sense to me.

I've just looked in Adobe bridge and realised that the lights were shot at f5.6 and the darks at f6.3! Argh! The lights were also shot at 374mm and the darks at 388mm. As this lens has a variable max aperture, I bet the lens slipped and caused the aperture to change.

BIAS: Lens cap stays on, shutter to 1/8000, same ISO and aperture.

FLATS: I use an artists light board to cover the end of the lens (with lens hood on), switch to AV mode & get the peak of the histogram about 1/3from the left. In this case, the flats were shot at 374mm, f5.6, 1/800 and ISO 800.

I've uploaded one each of the light, dark, flat and Bias frames in case this is useful.

13 hours ago, Richard_ said:

Yeah I was a bit miffed about no power supply too! What amperage power supply did you order? 12V @ 2A should work  fine for a DSLR and Star Adventurer, but if you add in dew heaters and replace the DSLR with a cooled camera in future, 2A may not be enough juice (see extract from manual below). I've been using a 12V @ 5A for the past year and it was enough to power the Star Adventurer, two dew heaters and the cooling on my ASI533 camera. The warning in the manual about powering mounts is aimed  towards heftier GoTo mounts like the Skywatcher EQ6-R as these have higher power requirements than the Star Adventurer. 

Sorry, my replies are starting to stray from the original topic! There's a few nuances with the ASI Air and some less obvious tips which I wish I knew when I started with it. Feel free to send me a PM if you have any questions which Google or YouTube aren't answering as I'm happy to share my experiences where I can 😊

 

I picked this power supply up, which has just been delivered so fingers crossed it works and I can start playing:

https://www.amazon.co.uk/gp/product/B01G9YS4NU/ref=ppx_yo_dt_b_asin_title_o00_s00?ie=UTF8&psc=1

I still have so much to learn so I'm happy to go off track a little. There seem to be so few clear skies that I need to make sure I'm well prepared to capture some good data whenever I can.

I hope to upgrade to a ZWO ASI294MC Pro or similar at some point but think I've still got a lot to learn with what I have first.

0N3A0091.CR2 0N3A0260.CR2 0N3A0354.CR2 0N3A0010.CR2

[CraigD1986]

1 hour ago, alacant said:

Hi

Lovely shot. Some nice detail emerging. A few bits of hands on advice...

5D iv noise? Bands? So lose the dark frames. Use flat and bias frames only. Add some dithered -say about 12 pixels- frames next time out. process the whole lot of them again without the dark frames and stack using a clipping algorithm. Most of the noise will be gone. Instead of DSS, try Siril.

Had a go at the stars. Maybe have a look at polar alignment?

Cheers and HTH

I've just had a little look at Siril and see that it's Mac compatible so I might give that a go. I hope to be able to achieve a better polar alignment next time as I'll have the ASIAir working. Do you recommend not using darks at all then for DSLR shooting? Most resources say they are essential but some say not so it can be confusing at times.

59 minutes ago, ONIKKINEN said:

Looks like the elongation is along the RA axis, which means the exposure was too long and affected by periodic error.

30s exposures would probably be more reliable for unguided. The only way to get rid of this limitation is to guide.

Next time out I'll be able to guide so hopefully can improve there.

[vlaiv]

5 minutes ago, CraigD1986 said:

I've just looked in Adobe bridge and realised that the lights were shot at f5.6 and the darks at f6.3! Argh! The lights were also shot at 374mm and the darks at 388mm. As this lens has a variable max aperture, I bet the lens slipped and caused the aperture to change.

That actually should not be a problem as darks don't receive any light. F/ratio and focal length are not important for darks, but they are important for flats - you need to take flats with same F/ratio, focal length and focus position as you had when taking lights.

6 minutes ago, CraigD1986 said:

I've uploaded one each of the light, dark, flat and Bias frames in case this is useful.

That can be very useful in diagnosing the problem. I'll take a look at them as soon as I get some time to do it.

[CraigD1986]

1 minute ago, vlaiv said:

That actually should not be a problem as darks don't receive any light. F/ratio and focal length are not important for darks, but they are important for flats - you need to take flats with same F/ratio, focal length and focus position as you had when taking lights.

That can be very useful in diagnosing the problem. I'll take a look at them as soon as I get some time to do it.

Thanks so much. I really appreciate you taking the time to help.

[alacant]

1 hour ago, CraigD1986 said:

Do you recommend not using darks at all then for DSLR shooting? Most resources say they are essential but some say not so it can be confusing at time

The theory may predict that dark frames are essential and although Siril's dark optimisation comes close, so far no algorithm we've found is capable of compensating for the artefacts dark frames introduce.

That's what we found by trying it. Not only with a 5d, but most other DSLRs which have passed through our hands.

Best to do a comparison and decide yourself.

Summary: light, bias and flat frames. Dither between the former. Stack with a modern clipping algorithm.

Cheers

 

[vlaiv]

I'm not really sure what is going on, but there are some issues with calibration files and possibly with lights?

First flats.

I'm surprised to see flats looking like this. This is montage of flat you posted separated by colors - I think this is BGGR order (blue top left, then green top right, then again green bottom left and red in the bottom right).

Flat intensity has been normalized to 100% for each channel here. Red and Green look very similar but blue looks much "flatter". Red and green fall of to 75% in the corners while blue falls off only down to 85%.

This indicates that system transmission and QE depending on angle varies with wavelength as well. This also means that flat correction will never work 100% - as you can't tell what sort of wavelengths you'll be getting. If you for example shoot Ha which is 656nm and falls into red part of spectrum - you won't be sure if your flat produce with your flat panel will have same fall off as flat panel is not producing exactly that frequency.

I don't know if this is due to lens and coatings on the lens or is it down to sensor.

Second issue is with dark / bias. Camera adjusts data with arbitrary offset so bias is useless and there is no chance of doing dark optimization.

Here is measurement of bias and dark you posted. Bias should have lower mean value than dark simply because dark contains both bias signal and dark current signal that builds up during exposure. In this case bias has higher signal than dark, and both are interestingly close to 2048 - which means that camera firmware adjusts data so that mean level of dark regardless of exposure is 2048.

We simply can't figure out true bias level nor dark current value because of this (not sure if there is a way to recover this artificial offset number).

In the end - I don't think that gradients in the final image are due to dark after all nor due to flat calibration. I think it is some sort of weird signal present in the image itself. It could be consequence of light pollution and the fact that lens or camera has different sensitivity depending on angle / wavelength thing - similar to flats above, but I'm not 100% sure.

This is very stretched bias:

and it looks like bias - no surprises there.

This is dark - and again - no surprises there, in fact - absence of any sort of gradient is a surprise there. It looks rather nice and flat.

In the end - here is what each color components looks like after just flat fielding (both light and flat was subtracted with 2048):

This is after LP gradient removal. This strange glow looks different in different color components.

I guess that your best bet in processing this sort of image is to use advanced background removal tool to make background nice and flat as I don't really think it is due to calibration but something else - and I can't really tell what it is.

Is there a lot of light sources near you when you image? Maybe some strange reflection on lens or something like that?

 

[CraigD1986]

37 minutes ago, vlaiv said:

I'm not really sure what is going on, but there are some issues with calibration files and possibly with lights?

First flats.

I'm surprised to see flats looking like this. This is montage of flat you posted separated by colors - I think this is BGGR order (blue top left, then green top right, then again green bottom left and red in the bottom right).

Flat intensity has been normalized to 100% for each channel here. Red and Green look very similar but blue looks much "flatter". Red and green fall of to 75% in the corners while blue falls off only down to 85%.

This indicates that system transmission and QE depending on angle varies with wavelength as well. This also means that flat correction will never work 100% - as you can't tell what sort of wavelengths you'll be getting. If you for example shoot Ha which is 656nm and falls into red part of spectrum - you won't be sure if your flat produce with your flat panel will have same fall off as flat panel is not producing exactly that frequency.

I don't know if this is due to lens and coatings on the lens or is it down to sensor.

Second issue is with dark / bias. Camera adjusts data with arbitrary offset so bias is useless and there is no chance of doing dark optimization.

Here is measurement of bias and dark you posted. Bias should have lower mean value than dark simply because dark contains both bias signal and dark current signal that builds up during exposure. In this case bias has higher signal than dark, and both are interestingly close to 2048 - which means that camera firmware adjusts data so that mean level of dark regardless of exposure is 2048.

We simply can't figure out true bias level nor dark current value because of this (not sure if there is a way to recover this artificial offset number).

In the end - I don't think that gradients in the final image are due to dark after all nor due to flat calibration. I think it is some sort of weird signal present in the image itself. It could be consequence of light pollution and the fact that lens or camera has different sensitivity depending on angle / wavelength thing - similar to flats above, but I'm not 100% sure.

This is very stretched bias:

and it looks like bias - no surprises there.

This is dark - and again - no surprises there, in fact - absence of any sort of gradient is a surprise there. It looks rather nice and flat.

In the end - here is what each color components looks like after just flat fielding (both light and flat was subtracted with 2048):

This is after LP gradient removal. This strange glow looks different in different color components.

I guess that your best bet in processing this sort of image is to use advanced background removal tool to make background nice and flat as I don't really think it is due to calibration but something else - and I can't really tell what it is.

Is there a lot of light sources near you when you image? Maybe some strange reflection on lens or something like that?

 

Thanks for taking the time to look into this for me. Most of what you said above is way above my understanding though. That said, I can see that the blue channel from the separated flat file does look way out of place. It doesn't sound like I can do anything about it though if this is something the camera/lens is causing.

I'll have a look through some Youtube videos about using the 5D Mk IV for astrophotography to see if there are any setting I should change.

I don't think there are any odd reflections or anything. I've been imaging from the back garden, which is pretty dark (Bortle 5). I leave the lens hood on and make sure to not turn any lights on.

It sounds like the conclusion is that the calibration frames could be doing more harm than good at this stage though.

[CraigD1986]

3 hours ago, alacant said:

The theory may predict that dark frames are essential and although Siril's dark optimisation comes close, so far no algorithm we've found is capable of compensating for the artefacts dark frames introduce.

That's what we found by trying it. Not only with a 5d, but most other DSLRs which have passed through our hands.

Best to do a comparison and decide yourself.

Summary: light, bias and flat frames. Dither between the former. Stack with a modern clipping algorithm.

Cheers

 

Thanks for the great info. I'll do a re-stack without the dark files and see if there is an improvement