Asi533 calibration problem

[Craig a]

Ok so for the last 4 nights I’ve been imaging the deer lick galaxy with the said camera, I was using 3 min subs and have a average pixel value of around 5000, now I have taken dark frames and flat frames and used them for calibration but after they were applied nebulosity says I now have a min pixel value of 0 yes 0? What the hell is going on? 

Edited September 22, 2020 by Craig a

[PeterCPC]

Have you tried stacking in DSS?

Peter

[Craig a]

1 hour ago, PeterCPC said:

Have you tried stacking in DSS?

Peter

Yes dss won’t even register the subs it just crashes, the subs don’t look dark in neb yet the min pixel value is 0, could a single black pixel cause neb to read the min value of 0? 

Edited September 22, 2020 by Craig a

[symmetal]

7 hours ago, Craig a said:

Yes dss won’t even register the subs it just crashes, the subs don’t look dark in neb yet the min pixel value is 0, could a single black pixel cause neb to read the min value of 0? 

Yes a single black pixel would mean the min value can be reported as 0. It's also an indication that the offset value set in the camera is too low as especially at the black end of the histogram where all the stretching takes place you don't want values from the camera to be outside the range of the A-D converter and so be reported as zero.

I've found with ASI cameras I've used, the minimum is often clipped to 1 ADU (at the camera resolution) rather than 0 just to avoid the dreaded zero value which will always give a black pixel no matter how much it's stretched.

The 533 is a 14 bit camera so ADU steps will increase in steps of 4 when viewed as a 16 bit histogram. Normally taking a bias frame and examing the histogram, (FitsLiberator gives a good histogram for viewing the black end of the histogram) will show the bias noise bump and the ADU values. As some CMOS cameras can apply different internal processing between short and long exposures, to be on the safe side take darks of around 1 second or so and examine the histogram. Increase the offset until you're well away from the zero minimum value, or 4 minimum value if it clips to 1 ADU at 14 bit. I'd suggest a min value of 12 or 16 ADU at least.

This allows image calibration to be carried out with more accuracy as the noise spectrum is better represented, and avoids dark spots in your images. 🙂

Alan

[Craig a]

I was using unity gain 100 and the offset was 70 as zwo have preset, I’m not sure nebulosity is working correctly with this cam because my 3min subs with no lp filter at f4.5 was yielding a mean pixel value of 5000ish I was expecting more than that to be honest, that seems low compared to my canon 1100d that yields 20000 mean pixel value, it’s either neb not working with the cam properly or it’s the camera itself my money is on the former to be honest, maybe I should ditch neb and get the demo version of apt to try with this cam and see if there’s any difference 

[symmetal]

The 533 pixel size is 3.76um compared to the 5.19um pixels of the 1100d so the 1100d pixels will collect twice the number of photons as the 533 pixels in the same exposure time so the average ADU would be twice for the 1100d on pixel size alone. As the 1100d is uncooled the dark current will also be significantly higher at 3 min exposure giving a higher average ADU for the 1100d darks, which are added on to the pixel value. The relative gain settings between the cameras will also give a difference in average ADU value. If the 1100d at your ISO setting has effectively twice the gain (in electrons /ADU) than that used on the 533 then the average ADU will be doubled too.

I found that the ZWO standard offset value is too low on the 1600 and you do get black clipping. The ASI071 and 6200 standard offsets though seem to be OK in that respect.

Some cameras have a column of 'black' pixels down one side presumably when the sensor perhaps has an odd number of pixels in one direction. I know one of my cameras does but I can't rember which one. If it's set to black after the A-D converter then it will always be black and you have to ignore it. Or clip the column off in processing to see the true black value. As long as the actual image pixels are above 0 ADU your're OK.

Alan

Edited September 22, 2020 by symmetal

[Craig a]

3 hours ago, symmetal said:

The 533 pixel size is 3.76um compared to the 5.19um pixels of the 1100d so the 1100d pixels will collect twice the number of photons as the 533 pixels in the same exposure time so the average ADU would be twice for the 1100d on pixel size alone. As the 1100d is uncooled the dark current will also be significantly higher at 3 min exposure giving a higher average ADU for the 1100d darks, which are added on to the pixel value. The relative gain settings between the cameras will also give a difference in average ADU value. If the 1100d at your ISO setting has effectively twice the gain (in electrons /ADU) than that used on the 533 then the average ADU will be doubled too.

I found that the ZWO standard offset value is too low on the 1600 and you do get black clipping. The ASI071 and 6200 standard offsets though seem to be OK in that respect.

Some cameras have a column of 'black' pixels down one side presumably when the sensor perhaps has an odd number of pixels in one direction. I know one of my cameras does but I can't rember which one. If it's set to black after the A-D converter then it will always be black and you have to ignore it. Or clip the column off in processing to see the true black value. As long as the actual image pixels are above 0 ADU your're OK.

Alan

Thanks a lot for explaining this to me It has helped, so in other words I should increase my exposure time to say 5 mins and see how it goes? And maybe increase my offset some more? What impact would increasing my offset to 80 from 70 have on my subs? Thanks again for your help the pixel size didn’t even enter my head to be honest......doh! I normally use iso1600 on the cannon which is quite high really as that iso gives the lowest read noise

Edited September 22, 2020 by Craig a

[symmetal]

1 hour ago, Craig a said:

Thanks a lot for explaining this to me It has helped, so in other words I should increase my exposure time to say 5 mins and see how it goes? And maybe increase my offset some more? What impact would increasing my offset to 80 from 70 have on my subs? Thanks again for your help the pixel size didn’t even enter my head to be honest......doh! I normally use iso1600 on the cannon which is quite high really as that iso gives the lowest read noise

No problem. 😀  Your average pixel values seem fairly high in both cases so I would go for a lower exposure. The average pixel value is pretty close to the level of the sky background and once the noise level from the sky background is significantly higher than the camera read noise, then the sky background is the dominant noise source. Further exposure gives little benefit and just reduces the dynamic range of your images as the left edge peak of the histogram approaches the right edge with increased exposure.

Looking at the read noise / gain etc. graphs for the 533 I assume you're using gain 100 which is unity gain, and where the HCG (high conversion gain) mode kicks in, giving lower read noise and increased dynamic range. If you look through this thread from last week you can see the details on calculating recommended average ADU values to just swamp the read noise by the sky background noise.

For the 533 at unity gain 100 and offset 70 (standard I believe) the optimal sky background ADU works out at just 370 ADU to make the camera read noise insignificant. Once the read noise is insignicant at say, 3 mins exposure there is no difference in stacking 10 subs at 3 min or 2 subs at 15 min so you may as well take 10 subs at 3 mins. Less chance of eggy stars due to alignment/tracking errors, or plane/satellite trails etc. and more subs means plane satellite trails are removed when using sigma stacking.  

For my 071 at unity gain my optimal sky background ADU is 526 ADU. It doesn't have a HCG mode so read noise is higher. I reach this ADU level (pretty much the same as the average pixel value as detailed in the above link) in 3 mins exposure, and that's the exposure I use all the time, (at unity gain). I'm in a bortle 3 zone which is pretty dark.

For you to get average pixel values of around 5000 at three minutes (assuming you're using nominal unity gain) implies you have significant light pollution where you are and it's better to take more exposures of a shorter duration to give your images a higher dynamic range and less clipping of stars. An exposure of 60 seconds would likely be better to use or even less if the average ADU is still well over 370.

To see whether the offset is too low or it's just a result of a column of added black pixels just try an exposure at maximum offset and see if the minimum pixel value is greater than 0. If it is then you can if you wish set the offset at a value where the min value is just above zero. If it stays at 0 then you probably have a fixed black column and the default offset, is most likely fine.

Hope that helps Craig and I haven't confused you further. 😄

Alan

Edited September 22, 2020 by symmetal

[symmetal]

For the above recommended 370 sky background ADU I assumed that 1 increase in offset gives one increase in ADU at the cameras bit resolution. To check if this is correct you can take an image (bias or dark, it doesn't matter) and look at the average ADU value (median average is better than mean average, but they should be very close anyway). Then take another image after increasing the offset by 10. If the average has incread by 40 then my ADU assumption is correct and the sky background ADU is right. 🙂 Converting from the cameras internal 14 bit to 16 bit, ( as the image statistics are usually displayed) you have to multiply the ADU value by 4 so 10 x 4 = 40.

Alan

[Craig a]

Thanks again, regarding the min pixel value of 0, I didn’t get that reading until I calibrated and debayered, the raw subs Never had that value. I normally dither my images so is it really worth using dark frames with such a low noise camera? As sigma clipping would get rid of any hot or cold pixels I never took darks with my cannon I used a bad pixel map, also if I go down with my exposure to say 60 seconds and it was on my target I did last week I would hardily see the galaxy in the subs I could only see the core on a 3 min sub the spiral arms were hardily visible would that be due to lp levels starting to wash out the spiral arms? Sorry I I’m sounding abit thick here I’m just struggling to get my head around these new cmos cameras coming from an uncooled ancient in tech terms dslr being used to seing a much brighter sub frame from it. According to clear outside I’m in Bortle 5 sky’s, I just carnt wrap my head around how you get faint details showing on galaxys with subs of 60 seconds, I suppose I’m just going to have to try it out and see what happens 

[symmetal]

Ah! I didn't realise that you only got the zero pixel values after calibrating. If the raw subs are not indicating 0 values then the offset value doesn't seem to be a problem and the default value is probably fine. 🙂 At least you got some info on offset if nothing else. 😄

Darks are usually used with CMOS cameras as they tended to suffer from 'amp-glow' which increased with exposure so darks were necessary to counteract this. Newer CMOS cameras have greatly reduced or even elimated amp-glow so darks are not as important. I've found with my 6200 that there is no amp glow visible even with a heavy stretch after 600s exposure although there are a number of clumps of pixels that seem to end up with ADU values around 10 ADU above the background level after the long dark exposure, so I continue using darks for that reason, although checking real images with and without dark calibration these clumps are not really apparant in the actual image.

There's an argument therefore for using bias instead of darks for the latest CMOS cameras, just as was done with CCD cameras. Some CMOS cameras do apply different internal processing for bias when using short exposures, less than 1 second or so, compared to long exposures so short bias didn't calibrate out properly, so a 1 or 2 second dark could be used as a bias in this case I suppose. With my 6200 I could find no difference in true bias or 2 second darks so I use bias to calibrate flats and use short exposure flats with no apparant problems.

Hot pixels left in your bias (or darks) would likely give a 0 pixel in your calibrated subs if subtracted in their entirety, though I'm not sure at how hot pixels are dealt with in bias/darks calibration procedure. I would hope the bad pixel map is applied to the darks/bias before calibrating. If so a bad pixel map may avoid your 0 pixels after calibrating.

It's quite normal not to see much detail in a single CMOS sub with the lower exposure times, but as the read noise is significantly lower now compared to CCD or older CMOS cameras, once the read noise is insignificant compared to sky background noise, it makes no difference as far as noise is concerned in taking many short subs over fewer longer subs. Short subs are less critical of tracking/alignment issues and will have less star bloat. 🙂

Are your raw subs exhibiting the 5000 ADU average pixel level, or just after calibrating? At a bortle 5 sky that seems pretty high for just a 3 min exposure. Maybe someone else in a similar sky can comment for comparison. You would reach the 370 ADU I calculated to render read noise insignificant in 10 seconds or so, so I'm a bit confused. 🤔 Whats the average pixel value of your bias frames or dark frames? My ASI071 dark frames have an average ADU of 284, and it takes 4 mins of exposure to increase the average ADU up to around 526, which is the calculated optimum sky background ADU for that gain/offset setting.

As a test you can take say 10 three minute exposures, and then 30 one minute exposures, and then stack the two results to compare how they look. I expect the detail will be the same in both, the 1 min exposure stack may even be sharper if you have some tracking/alignment issues, and the clipped stars should be smaller. 🙂

Alan  

Edited September 23, 2020 by symmetal

[Craig a]

The 5000 adu is as it came came straight from the camera so before calibration, my dark frame stack has a mean value of 2797 with a min of 2750 

[symmetal]

That's a high dark frame mean value and a corresponding high minimum. I can only assume that your offset value increases the camera native ADU by 10 for each increase of offset by 1, like my 6200. This calculates to an optimum Sky background ADU of 2823 which is closer to your readings but is only 26 ADU above your dark mean value. My 6200 sky background optimum ADU is 82 ADU above the dark mean value of 501. As my camera is not unity gain at that point that could account for the difference. 🤔

If you wish, but really to satisfy my curiosity, 😄 you can do the test I mentioned above of taking two darks, or bias, with one of then having the offset increased by 10, and see how much the mean pixel value increases between them. If it increases by 400 then my assumption above is right and the camera is meant to work with high offset values for some reason.

Alan

[Craig a]

Ok I just did 2 bias frames and the first came out at 2796 than I increased by 10 and that was 3196 so you are correct a 10 increase in offset is 400 in adu

Edited September 23, 2020 by Craig a

[Craig a]

The first was at unity gain and the zwo preset of 70 offset then increased the offset to 80 and 400 adu was added, so how do I get that 26 adu value above my dark mean to increase because judging my your reaction it’s low 😆 

Edited September 23, 2020 by Craig a

[symmetal]

Thanks for doing the test Craig. That's a relief. I was doubting all my calculations and that I was giving you false information. 😀

I don't know the reason Zwo has chosen such a high offset ADU for the 533 camera but there must be a reason, so I should leave it at the default value of 70. Perhaps at the extremes of gain settings, should you wish to use them, the high offset is necessary to avoid possible clipping of the blacks.

Anyway, your new optimum sky background ADU of 2823 is still way below your current 5000 ADU so I would at least try using a sequence of shorter exposures of 1 minute as in my test mentioned previously, and see if you get better results.

Also, next time you're out imaging, you could just see what exposure gives a mean ADU of around 2823. If it's only around 15 sec or so it implies that the read noise using the camera's HGC mode is low enough that it's insignificant at very low exposures. To save ending up with a 500 15 sec exposures to stack I'd probably stick to a minimum of a minute for trying out to avoid needing large hard disk storage and taking ages to stack. 😀  

Finally, to see if the 0 values go after calibration, you could try using a bad pixel map when calibrating. 

Alan

[Craig a]

Brilliant, thanks very much for taking time to go through all that info with me, I will try out 1min exposures on the next clear spell and report back, I shall make a bad pixel map from a stack of 1min exposures and see how that goes aswell 😀

Edited September 23, 2020 by Craig a

[Craig a]

Can I just ask would that sky background adu value you worked out for me work for any gain setting? For instance if I wanted to take advantage of the max dynamic range and full well of this camera? 

[symmetal]

No, if you alter the gain the level of read noise changes so the optimum sky background ADU will change as well. You can do the calculations as detailed in the other thread I mentioned above, or let me know what gain values you might use and I'll give you the new values. Changing the  offset also alters it as well, but it's easiest to keep the offset fixed and yours is such a high value anyway keeping it fixed won't be a problem.

Having the offset too high is not such a big deal. It just means you lose a bit of dynamic range as the whole histogram has been shifted to the right a bit.

The 533 HGC mode is turned off below gain 100 so the read noise is noticeably higher then, requiring longer exposures to 'neutralize' it. 🙂

Alan

[Craig a]

Ok if I did use a different gain at some point it would be 0 gain and as maths isn’t my strongest point could you give me the new values please? I will check what the preset offset is for highest dynamic range mode that zwo have preset 

[Craig a]

Right the ascom driver with the zwo presets are offset of 70 for all there modes so gain 0 is still 70 offset 🙂

[symmetal]

Hi Craig,

I just rechecked the maths for your normal gain 100 setting on the 533 and the optimum sky background ADU value is actually 2890. This is much more reasonable. 😀 This is 93 ADU higher than your dark frame mean ADU, and not the 26 ADU I said previously calculated. My 071 OSC camera for a similar gain setting has the sky ADU 205 above the dark. This is reasonable as it doesn't have the HCG mode so has significantly higher read noise. My 6200 which does have HCG has the sky ADU at 82 above the dark. All comparable figures. Sorry I got the figure wrong before, I had so many windows open I probably read the wrong graph at the time.

So, for gain setting 0 your sky background ADU is 2986 with the same offset 70. This is almost the same as at gain 100. This is because the high offset is contributing to the bulk of the value calculated. 2800 ADU is purely because of the offset of 70. At gain 100, the extra 90 ADU is to combat read noise. At gain 0, then an extra 186 ADU is needed to combat the read noise. When doing test exposures, it may be easier to subtract the offset 2800 ADU from your mean results, to get the actual change in ADU due to the exposure time itself, and you can then estimate the right exposure needed more easily.

Alan

Edited September 24, 2020 by symmetal

[Craig a]

Thank you again for that, I shall have a test at 0 gain aswell to see if my usual 3 min subs get me to that mean adu Purely  because of the space needed on the computer with 3 min subs compared to 3x the amount of subs for the same total integration  if using 60second subs, I shall still test out the 1 min subs at 100 gain too though, if I can get that 2986 with 3-4 mins at 0 gain I will stick with that so I can use the max dynamic range and full well of 50,000, roll on test night😀 I carnt thank you enough working this out for me I thought I had a duff camera and was close to throwing it over the neighbours house 

[symmetal]

No problem Craig. 😀 I use these calculations for all my deep sky cameras and they seem to give good results. My 071 OSC needs 270s exposures to reach the sky background value at bortle 3. At bortle 5 then 180s may well be optimum for you at gain 0. At gain 100 the optimum exposure may be too short, as you say, and you would end up with hundreds of subs. As I'm sure you know, it's the total integration time that matters and not the individual  exposure length, once the read noise has been made negligible. Report back once you've managed some tests. 🙂

Of course, these optimum exposures you work out, are for the same sky darkness, so if the moon's up then the exposure to reach the sky background ADU would be much shorter than on a moonless night. It's best to wait at least an hour after 'official' sky darkness on a moonless night to do the tests, as the sky will get darker for a period of time after it starts so you're calculating for a best average exposure throughout the night.

Alan

Edited September 24, 2020 by symmetal

[Phillyo]

I have nothing of value to really add to this but I just wanted to say that this thread has been a super interesting read (along with the linked thread for the calculations). I'm getting the 533 soon so it's also very applicable to my situation! Thanks Alan for taking the time and effort to explain all of this to us mere mathematician mortals! 

Phil