IC 1805 Processing Help

[SyedT]

Hi all,

Looking for a bit of help with figuring out what's going on. I'm using an ASI 1600MM-C on a WO GT81 with 0.8x reducer in place. This is a stack of 182*120s + 11*300s images at Gain 300 Offset 50 in OIII. All images in the stack were taken on moonless nights. The images have been calibrated with darks and bias (no flats as I use 2 inch filters)registered in Pixinsight. What I keep seeing are these black areas everywhere on the image (can see them particularly well in top left. Any ideas what these might be? Would appreciate any insight. The same problem doesn't appear on my SII/Ha frames, or at least not to this extent. I've tried cosmetic correction, doesn't really help.

Cheers

[PhotoGav]

Are the darks fresh, i.e. done recently or are they old?

[swag72]

This is definitely the sort of thing I see when my darks and / or bias need to be redone.........

[SyedT]

12 hours ago, PhotoGav said:

Are the darks fresh, i.e. done recently or are they old?

 

11 hours ago, swag72 said:

This is definitely the sort of thing I see when my darks and / or bias need to be redone.........

Darks have just been done a couple of weeks ago, I did a fresh set for both 120s and 300s exposures; bias were only done a month or so ago too. I thought that's what might be going on but not sure any more. Could this be a CMOS thing?

[wimvb]

I think that with gain 300, your offset of 50 may be a bit low. You need to make sure you don't clip the data. Also, if you don't use flats, you don't need to use bias frames, since the bias signal is contained in the darks. You could try restacking with uncalibrated darks.

With my ASI174mm-cool I don't use bias, but I do use darks, flats, and dark flats. I also make sure that the darks have the exact same settings as the lights for exposure time, temperature, gain and offset. And I don't optimize my master dark during image calibration of the light frames.

[SyedT]

21 hours ago, wimvb said:

I think that with gain 300, your offset of 50 may be a bit low. You need to make sure you don't clip the data. Also, if you don't use flats, you don't need to use bias frames, since the bias signal is contained in the darks. You could try restacking with uncalibrated darks.

With my ASI174mm-cool I don't use bias, but I do use darks, flats, and dark flats. I also make sure that the darks have the exact same settings as the lights for exposure time, temperature, gain and offset. And I don't optimize my master dark during image calibration of the light frames.

Thanks Wim. I read up a lot about the 1600MM-C and optimum gain/offset and exposure values etc, and I couldn't see it being suggested to go beyond offset 50 for a gain of 300. Could you explain it in a bit more detail please? I'm having this issue with only some of my images and not others, even with same settings, which is puzzling to say the least!

[SyedT]

Interestingly, running cosmetic correction in PI worked this time, but I've had to use quite an aggressive cold pixel removal value to do so.

[wimvb]

46 minutes ago, SyedT said:

Thanks Wim. I read up a lot about the 1600MM-C and optimum gain/offset and exposure values etc, and I couldn't see it being suggested to go beyond offset 50 for a gain of 300. Could you explain it in a bit more detail please? I'm having this issue with only some of my images and not others, even with same settings, which is puzzling to say the least!

In a cooled CMOS cameras such as ASI and QHY cameras, you can set both gain and offset. The gain increases sensitivity, but also noise (same as high ISO in a DSLR). The lower tail of the noise curve can get clipped, meaning that the bell shaped part of the histogram stretches below zero, but is clipped at zero. If this clipping is excessive, it will show up as black pixels or patches in an image. Increasing the offset moves the entire bell curve to higher values, and you need to use a value where no pixels register as completely black (0). At the same time, using a too high offset value will lower the dynamic range of the camera.

The analogy in post-processing is the black point slider in curves stretching. A too high black point value corresponds to 0 offset, and clips dark values. Increasing the offset is like moving the black point to the left (decreasing it), which moves the histogram to the right. In an image this decreases the tonal range. In a camera it decreases the dynamic range.

25 minutes ago, SyedT said:

Interestingly, running cosmetic correction in PI worked this time, but I've had to use quite an aggressive cold pixel removal value to do so.

Cosmetic correction in PI removes black clipped pixels, by filling in the average of the surrounding pixels. If you need aggressive cold pixel removal, this means that the data is clipped; the offset is too low. Btw, since this noise is completely random, you can't use "cold pixel maps", as you would use defect maps for hot pixels.

When I measured the characteristics of my ASI174MM-cool, I found that at the highest gain setting of 400, I needed to use a very high value for the offset in order to avoid clipping alltogether. I used very short exposures in these tests. When I use the offset during image capture, with longer exposure times, I inevitably get images that are too bright, and the histogram is always away from the left edge. My conclusion from this is that the best offset value depends on exposure time. For longer exposures the sky background or dark current of the camera does the same as the offset; it moves the histogram to higher values.

Here's how I did my test:

Took a dark sub at the gain setting and the shortest exposure time that I would normally use. Used offset = 0.

Examined the pixel values of the entire image. (I used image statistics in PI for this.) If the lowest pixel value was 0, increased the offset and took a new exposure.

Repeated this until there were no pixels that were clipped (all pixel values > 0) Increasing the offset value beyond this would increase the pixel value of the darkest pixel in the image, and decrease the dynamic range.

[SyedT]

1 hour ago, wimvb said:

In a cooled CMOS cameras such as ASI and QHY cameras, you can set both gain and offset. The gain increases sensitivity, but also noise (same as high ISO in a DSLR). The lower tail of the noise curve can get clipped, meaning that the bell shaped part of the histogram stretches below zero, but is clipped at zero. If this clipping is excessive, it will show up as black pixels or patches in an image. Increasing the offset moves the entire bell curve to higher values, and you need to use a value where no pixels register as completely black (0). At the same time, using a too high offset value will lower the dynamic range of the camera.

The analogy in post-processing is the black point slider in curves stretching. A too high black point value corresponds to 0 offset, and clips dark values. Increasing the offset is like moving the black point to the left (decreasing it), which moves the histogram to the right. In an image this decreases the tonal range. In a camera it decreases the dynamic range.

Cosmetic correction in PI removes black clipped pixels, by filling in the average of the surrounding pixels. If you need aggressive cold pixel removal, this means that the data is clipped; the offset is too low. Btw, since this noise is completely random, you can't use "cold pixel maps", as you would use defect maps for hot pixels.

When I measured the characteristics of my ASI174MM-cool, I found that at the highest gain setting of 400, I needed to use a very high value for the offset in order to avoid clipping alltogether. I used very short exposures in these tests. When I use the offset during image capture, with longer exposure times, I inevitably get images that are too bright, and the histogram is always away from the left edge. My conclusion from this is that the best offset value depends on exposure time. For longer exposures the sky background or dark current of the camera does the same as the offset; it moves the histogram to higher values.

Here's how I did my test:

Took a dark sub at the gain setting and the shortest exposure time that I would normally use. Used offset = 0.

Examined the pixel values of the entire image. (I used image statistics in PI for this.) If the lowest pixel value was 0, increased the offset and took a new exposure.

Repeated this until there were no pixels that were clipped (all pixel values > 0) Increasing the offset value beyond this would increase the pixel value of the darkest pixel in the image, and decrease the dynamic range.

Thanks for the detailed explanation, much appreciated! I've taken a screenshot of a dark frame at 15s (shortest exposure time I use), Gain 300 Offset 50 , and have pulled up the statistics. Am I correct in assuming that the 'minimum' value is what you are referring to when you mention the lowest pixel value?

[wimvb]

Yes, minimum is lowest. In this image, it's clear from clipping. The highest value (maximum) is due to hot pixels. The statistics look normal for this exposure.

[SyedT]

29 minutes ago, wimvb said:

Yes, minimum is lowest. In this image, it's clear from clipping. The highest value (maximum) is due to hot pixels. The statistics look normal for this exposure.

Just for my understanding; if I were to look at the stats for each light frame I've taken, and ensure that the minimum value for each is above zero, then I can safely say that there is no clipping present in that particular frame?

[wimvb]

That should be correct. But if you subtract dark frames that don't match light frames, you can (theoretically) get clipping. That's why you should take darks at the same settings as lights, and not rely on the PixInsight optimization function. You also need to make sure that your darks are really dark. Just using a plastic lens cap/cover is not enough.

[SyedT]

On 30/11/2017 at 19:08, wimvb said:

That should be correct. But if you subtract dark frames that don't match light frames, you can (theoretically) get clipping. That's why you should take darks at the same settings as lights, and not rely on the PixInsight optimization function. You also need to make sure that your darks are really dark. Just using a plastic lens cap/cover is not enough.

I've tried it without optimization as well, not really getting anywhere. I'm using the scope cap + a black rubber cap + a black hat + a thick towel to cover the objective. 

Ran a test on NGC6888 OIII subs in the following sets to see if it was the gain/sub length causing issues:

100x60s @ Gain 300, Offset 50
20x300s @ Gain 139, Offset 21

Both have been integrated without any calibration, using Winsorized Sigma Clipping in PI. Images here: https://drive.google.com/open?id=1H6JPMMc8C3C6iUxHBGIhcFquPnEtmzhl. It's clearly visible that the 60s integration has black lines visible throughout the image, whereas the 300s stack has no such issues. I'm a bit lost for an explanation. Any ideas?

[wimvb]

The 60 second stack definitely shows walking noise, but as dark pixels and not the usual bright pixels. Did you save the rejection maps that were created during stacking? Sometimes these can show the problem.

Have a look here for a method to get rid of walking noise:

http://pixinsight.com.ar/en/info/processing-examples/19/crux-southern-cross.html

If this doesn't work, as a last resort (something which I have never tested), ImageCalibration can add a pedestal to calibrated images to avoid negative values. Could be worth trying, I think.

Here's a description:

https://pixinsight.com/forum/index.php?topic=2182.0

[SyedT]

21 hours ago, wimvb said:

The 60 second stack definitely shows walking noise, but as dark pixels and not the usual bright pixels. Did you save the rejection maps that were created during stacking? Sometimes these can show the problem.

Have a look here for a method to get rid of walking noise:

http://pixinsight.com.ar/en/info/processing-examples/19/crux-southern-cross.html

If this doesn't work, as a last resort (something which I have never tested), ImageCalibration can add a pedestal to calibrated images to avoid negative values. Could be worth trying, I think.

Here's a description:

https://pixinsight.com/forum/index.php?topic=2182.0

Thanks Wim. Had a long session yesterday; upped my dithering to Extreme in SGP, which solved the issue completely. Here is a 50x180s stack of uncalibrated NGC 2264 subs: https://drive.google.com/open?id=1uidbpCaGdxnMOw6kpqH_cMnwwXM6XYeV. No walking noise visible at all.