ZWO ASI294MC Pro - Bad calibration frames

[AweSIM]

Hi,

I recently purchased a ZWO ASI294MC Pro and was very happy with the results till a few days ago when all my processed images started turning up horrible. I've narrowed the problem to calibration frames not being correct. I've listed all problems with the calibration frames in this post, in hopes someone here can help me sort them out:

PS. For all frames, I kept the camera temperature at 0C, gain and offset at 10. (And while we're at it, what gain/offset values do you guys use and why? Is it just a personal preference or is there a way to determine what gain/offset should I use? Does it depend on target, visibility conditions, narrowband filters, etc?)

Also, I used the following exposure times for different calibration frames: BIAS (0s), DARKS (300s), FLATS (5s), LIGHTS (300s), DARK FLATS (5s).

Also, the frames were captured in SGP and processed in PI.

1. The bias frames are wierd, to say the least. Being a DSLR user till recently, I'm used to seeing the raw, bayered, stretched BIAS frame to be an image full of random noise with no discernible patterns. When I look at the ZWO ASI294MC Pro's BIAS frames, and stretch it, I can see the bayer grid pattern very clearly. The GREEN pixels have the lowest brightness while the BLUE pixels have the highest brightness. I took BIAS frames by detaching the camera from the OTA and putting its cap back on and turning off all lights in the room (to ensure no stray light leaked in).In the first attachment, you can see auto-stretched image of a BIAS i took with my earlier NIKON. The image is bayered but still the noise is random. In the same image, you can also see the auto-stretched image of the BIAS i took with ASI294. The bayer pattern is clearly visible. If I split the CFA based on each CFA channel (1, 2, 3, 4), and auto-stretch them, the noise is random. However, the clearly different bias of each channel in the CFA raw file is what's bothering me. I haven't been able to find samples of a BIAS frame online taken from this camera, so I'm hoping that someone here with the same camera can tell me what I'm doing or understanding wrong here.

2. I took 20 darks of 5m each. The uncalibrated DARKS show the same bayer pattern and auto-stretch barely shows any amp glow. However, when I calibrate them against the MASTER_BIAS and integrate them into a MASTER_DARK, and look at it after auto-stretching, I can see no bayer pattern and the amp glow is clearly visible. This tells me that what I'm doing is right so far. However, when I calibrate flat frames against the MASTER_BIAS and MASTER_DARK and integrate them, I see a grainy result that shows the vignetting clearly but the amp glow is visible as a dark patch in one corner. This is wrong, I think. The exact steps I'm doing are listed in the guide that I'm following: https://www.lightvortexastronomy.com/tutorial-pre-processing-calibrating-and-stacking-images-in-pixinsight.html

If it helps, I've uploaded 5 files of each type of frame (due to space constraints) on google drive that are available at https://drive.google.com/open?id=1fsxw_LIDH7B-DqJrN7WBXQQhOwB-ezKz if you want to take a closer look at the data. So far, I've tried a lot of combinations of settings in PI but haven't been able to find or come up with a workflow that reliably works for me. I would be really really really grateful if someone could help me out or point me in the right direction.

Finally, is it possible that I might've damaged the camera and that's why its not giving me good results now? I took superb images of the Orion and Rosette just days earlier and only now I am facing these issues. I lost raw frames for those targets so I cannot compare BIAS/DARKS of those images. I haven't dropped the camera, haven't gotten it wet (except when its cold and dew forms outside the casing. I cool it down via SGP down to 0C from 10C in about 4-5 mins. When warming it up, I sometimes ask SGP to warm it up to 15C in 5 mins but most of the time just stop the cooling aid so it warms up by itself. Is that the right way or a recipe for a disaster already happened or waiting to happen?

Thanks,

A very anxious member who's dreading something's massively wrong with his very expensive gear that he cannot afford again. =(

Asim

[vlaiv]

It's rather late over here so I'll be brief for now, but I'll take a closer look tomorrow in the morning.

If you are seeing bayer pattern in both bias and darks - it could be that you have a light leak.

It might seem to you that capping off camera and being in dark room is going to solve light leak problem, but you need to understand that your camera has only AR coated window and that you don't have any UV/IR cut filter on it. In fact here is what transmission of your anti reflexive cover window on camera looks like:

It will pass all wavelengths in IR part of spectrum (700nm and above). Another important thing to remember is that plastic is often transparent to IR wavelengths. So just putting plastic cover on your camera and being in dark room might not be enough to prevent IR leak - especially if you have heat sources near camera.

Best way to be sure you don't have any light or IR leak is to do following: put plastic cover on and then place camera "face down" on wooden table, or better yet - use aluminum foil to make additional layer on top of plastic cover (wrap camera in aluminum foil). I use first approach with wooden table, but some people do it like this:

I'll have a look at your subs tomorrow to see if I can figure out anything from them.

Btw, for ASI294 I would use gain 120 - as that puts camera into low read noise domain, and I would raise offset until I get very nice histogram separation from the left side on my bias subs. Not sure what that should be with ASI294, but with my ASI1600 I use offset of 64.

One way of doing it is to set gain that you are going to use, and then set offset at particular value - for example offset 10 that you already used. Then shoot a bunch of bias subs and see if there are pixels that are too low in value - something like 4ADU or similar and if histogram is well away from left side. If you find that it's not the case - raise offset and repeat. Once you find value that you are happy with - then prepare your darks with those settings (temp, gain, offset and wanted duration) and use same settings when you go out and gather actual data - lights.

Don't use bias in calibration though - it is not needed - darks contain both dark signal and bias signal so they alone will calibrate rights properly (also some CMOS sensors have issues with bias, and until you check your camera for bias issues - it's best to avoid using it in calibration).

[vlaiv]

Ok, here are my findings:

1. There is no light leak so you are ok with that.

What I suspect is going on with bayer pattern in the darks / bias subs is some sort of white balance in drivers. I'm not familiar with drivers of this particular camera, but if you turned on automatic white balance in drivers - you might want to turn that off.

I'm suspecting AWB because this camera is 14bit ADC camera - that means that all values in recorded raw sub should be divisible by 4 (16 bit total, 14 bit used - remaining two bits set to 0 - same as number being multiplied by 4). It is not case with your bias and dark files - there are odd values as well as even. This can happen only if values have been changed once they were read out of the camera - for example AWB process in drivers could explain that. AWB could also explain bayer patter. Since there is some offset to pixel value (pixels in bias are not 0, but rather positive value roughly equal for each pixel) - multiplying that with different factors for each color will create different values. Even values in bayer matrix show this could be so - since strongest is blue then red and green is the weakest. If you look at ASI294mc QE curves - it is other way around - green is strongest, red is second and blue is the weakest - which means that they have to be multiplied with certain factors to "even out" giving strengths of pixels in bias/dark that we see.

In any case - I would recommend turning off auto white balance in drivers. With that turned on, I can't judge if offset of 10 is good value because I can't see actual read out values of pixels but only WB corrected values.

Another thing - AWB should not mess up your calibration and everything should work fine.

2. Your subs indeed work fine. Here is comparison on one sub between proper calibration and raw sub without calibration:

This is the same sub - left has been calibrated by (ligth - avg_dark) / (avg_flat - avg_flat_dark) while right one is without calibration. Both subs were binned x8 to get enough SNR to show features of the image. As you can see - you get pretty decent sub with calibration - no vignetting and it looks clean in comparison to raw sub.

As you see - another confirmation that above is probably AWB, as that is not going to affect calibration, and indeed calibration works fine.

3. I would just recommend you in the future to use Gain 120, to turn off AWB and check if offset has good value - and continue taking nice images

HTH

[Dustspeakers]

10 hours ago, vlaiv said:

One way of doing it is to set gain that you are going to use, and then set offset at particular value - for example offset 10 that you already used. Then shoot a bunch of bias subs and see if there are pixels that are too low in value - something like 4ADU or similar and if histogram is well away from left side. If you find that it's not the case - raise offset and repeat. Once you find value that you are happy with - then prepare your darks with those settings (temp, gain, offset and wanted duration) and use same settings when you go out and gather actual data - lights.

Hi Vlaiv,

I'm also having issues with calibration frames (flats in my case) but I was interested in your comment above in trying to understand the whole calibration process. I have an ASI1600MM-Pro and I do see that raising the offset does shift the histogram from the LHS towards the RHS. For example on Bias, for Gain 200/Offset 10, I get a 'bell curve' of ADUs values between 0.78 - 12; for Gain 200/Offset 20, values between 6 - 18; for Gain 200/Offset 40, values between 19 and 29. So if I've understood you, if I shoot Darks (and Lights) with the latter value this is the correct way to proceed. I've just done a 300s Dark and it looks as I would expect with even random noise. So far so good (I think). But what is the action of increasing the offset to move the histogram to the right actually doing to benefit the images. Can you illuminate, so to speak.

Thanks

Steve

[vlaiv]

4 minutes ago, Dustspeakers said:

Hi Vlaiv,

I'm also having issues with calibration frames (flats in my case) but I was interested in your comment above in trying to understand the whole calibration process. I have an ASI1600MM-Pro and I do see that raising the offset does shift the histogram from the LHS towards the RHS. For example on Bias, for Gain 200/Offset 10, I get a 'bell curve' of ADUs values between 0.78 - 12; for Gain 200/Offset 20, values between 6 - 18; for Gain 200/Offset 40, values between 19 and 29. So if I've understood you, if I shoot Darks (and Lights) with the latter value this is the correct way to proceed. I've just done a 300s Dark and it looks as I would expect with even random noise. So far so good (I think). But what is the action of increasing the offset to move the histogram to the right actually doing to benefit the images. Can you illuminate, so to speak.

Thanks

Steve

Thing with cameras is that they can't produce negative values. One would think that because we are counting photons, we don't need negative values, and in perfect world that would be the case, but since there is read noise present, and read noise is gaussian type noise centered on 0 - meaning it can have negative values, subs that don't contain light can end up being negative due to noise (actual pixel value can't be negative, but pixel + noise can be negative).

Since camera can't produce those negative values, only positive, we need to make sure numbers that we read off from camera contain all the information - we do that by adding the offset. So if pixel values for example are in range of -10 to +10 (actually zero because there is no light, but read noise is such that it produces noisy values between -10 and +10) - we can say - let's add 20 to each pixel value and then we will have all the values to be positive -in range +10 to +30. We have all the same information if we later subtract that 20 - we will get our negative values.

Now if you imagine following - row of numbers -1,0,1,1,-1,-1,0,1 .... random equal spread of -1, 0 and 1 values and you decide to take average of that. Average value will be 0.

Let's now do the same, except this time do the "clipping" - we throw away all negative values and replace them with 0. Now we have 0,0,1,0,1,0,0,1,1 .... sequence. We take average of that value and it is no longer 0 - it must be higher than zero. Act of clipping changed some things - it changed our noise distribution and changed our mean value.

If clipping occurs when one is doing darks - it can happen (and most likely it will happen) that no clipping will occur when one is doing lights. There is Sky level (light pollution) that will create real offset and all those negative values will end up as positive values.

Since darks are clipped but lights are not - they no longer match and that leads to issues with calibration (and it even affects flat calibration) but also due to changed noise statistics - stacking is less effective in removing noise (stacking works when you have mix of poisson and gaussian distributions, or each one individually, but it will not work as good on clipped data).

This is why it is important to set offset properly - and you determine how to set offset properly by examining bias subs. If all pixel values are above lowest value - you are safe. There is exception to this rule - if you have "cold" pixels that don't have much bias signal in them and are always very low in value - but hopefully you will have no or very few of those (recently I've seen example of ASI1600 camera that has 12 cold pixels - it created confusion with above "make all pixels higher" rule).

What sort of issues with flat calibration do you have?

[Dustspeakers]

Ahhh. Great explanation - I understand that now. At the moment I'm going through the my whole setup to try and eliminate anything really inane that I may be doing, and increase my understanding of the whole calibration process so I'm doing some more testing and I'll post based on the results.

Thanks for the assistance.

Steve

[AweSIM]

3 hours ago, vlaiv said:

Ok, here are my findings:

1. There is no light leak so you are ok with that.

What I suspect is going on with bayer pattern in the darks / bias subs is some sort of white balance in drivers. I'm not familiar with drivers of this particular camera, but if you turned on automatic white balance in drivers - you might want to turn that off.

I'm suspecting AWB because this camera is 14bit ADC camera - that means that all values in recorded raw sub should be divisible by 4 (16 bit total, 14 bit used - remaining two bits set to 0 - same as number being multiplied by 4). It is not case with your bias and dark files - there are odd values as well as even. This can happen only if values have been changed once they were read out of the camera - for example AWB process in drivers could explain that. AWB could also explain bayer patter. Since there is some offset to pixel value (pixels in bias are not 0, but rather positive value roughly equal for each pixel) - multiplying that with different factors for each color will create different values. Even values in bayer matrix show this could be so - since strongest is blue then red and green is the weakest. If you look at ASI294mc QE curves - it is other way around - green is strongest, red is second and blue is the weakest - which means that they have to be multiplied with certain factors to "even out" giving strengths of pixels in bias/dark that we see.

In any case - I would recommend turning off auto white balance in drivers. With that turned on, I can't judge if offset of 10 is good value because I can't see actual read out values of pixels but only WB corrected values.

Another thing - AWB should not mess up your calibration and everything should work fine.

2. Your subs indeed work fine. Here is comparison on one sub between proper calibration and raw sub without calibration:

This is the same sub - left has been calibrated by (ligth - avg_dark) / (avg_flat - avg_flat_dark) while right one is without calibration. Both subs were binned x8 to get enough SNR to show features of the image. As you can see - you get pretty decent sub with calibration - no vignetting and it looks clean in comparison to raw sub.

As you see - another confirmation that above is probably AWB, as that is not going to affect calibration, and indeed calibration works fine.

3. I would just recommend you in the future to use Gain 120, to turn off AWB and check if offset has good value - and continue taking nice images

HTH

Hi Vlaiv!

First of all, thanks a lot for that in depth insight into the world of CMOS cameras. Thank you for looking at my subs and I'm amazed how well you were able to use just three frames of each type. No doubt, I'll be going over your post for a few days trying to understand everything you've mentioned. 😃

At the same time, thanks for clarifying the purpose of gain and offset, and I think you've pointed me towards the issue. In SGP, when you connect your camera, you have the option of specifying camera gain/offset, which by default is 10, 10. For my last good set of subs, I set it to 120, 10 but this time I used the default 10, 10. I've experimented a bit more about these and here are my findings:

1. I have no idea how to control the WB of this camera. I haven't changed anything in its driver, and honestly, I haven't even seen any GUI for the driver that lets you control anything about the camera. Just that there was a driver available from ZWO site and I downloaded and installed it and that's it. I'll dig around on the internet about how to control its WB and fix it there. I'll report back when I've figured it out.

2. If I set gain/offset to 10/0 or 120/0 and then take bias subs of 0s, and stretch it, then I get no bayer pattern. The entire stretched image appears almost black with just a few pixels of various shades of gray scattered about. Which is understandable after having read your posts that since I'm not adding any offset, all values should be around 0. If I set the gain/offset to ?/1 or ?/10 or ?/64, immediately I get a bayer pattern. I've attached FIT files for all of these cases as well. I can see that as I increase the offset, my histogram starts moving to the right. However there are two distinct peaks in the histogram and that's probably due to the WB you've mentioned. 😃

3. As you pointed out in your post, that probably having a bayer pattern in bias/dark subs is not an issue, since this is an offset added to all frames (bias/flats/darks/lights) and if calibration is done properly, this signal should be removed. Probably what mattered with my setup was that earlier on I was taking subs at 120/10 gain/offset and was very pleased with the result and then all of a sudden, I forgot to change it from the default value of 10/10 and got such a low signal in my frames that I got freaked out. 😃 I'll try shooting all the frames again as soon as the sky clears up at 120/64 and this time I'll pay special attention to the values inside the calibration frames as well. Your post has been very very very helpful in allowing me to understand how these cameras work. Thank you so much for the time you took to answer these questions! This forum has never once disappointed me in the past 10 years or so, and it's cuz of people like yourself who are willing to make time to answer naive/dumb questions of people like myself. 😃

4. Can you tell me how you inspected the values inside these FIT files? How can I check that none of the values inside my bias frames are not negative and adjust the offset accordingly?

Thank you so much for your insights. 😃

Asim

FIT_Gain1_Offset0.fit FIT_Gain120_Offset64.fit FIT_Gain1_Offset10.fit FIT_Gain1_Offset64.fit

[vlaiv]

13 minutes ago, AweSIM said:

1. I have no idea how to control the WB of this camera. I haven't changed anything in its driver, and honestly, I haven't even seen any GUI for the driver that lets you control anything about the camera. Just that there was a driver available from ZWO site and I downloaded and installed it and that's it. I'll dig around on the internet about how to control its WB and fix it there. I'll report back when I've figured it out.

This part got me worried as well. I could not find any reference of WB in ASCOM drivers online. In fact, I found one discussion on ZWO user forum that talks about WB settings - and there Sam stated clearly that ASCOM drivers don't have this setting.

I can think of only one possibility for changed White Balance. Have you used SharpCap by any chance at any point? I suspect that there might be issue between native and ASCOM drivers for this camera model. In native drivers you can set WB and it is set to something like 50/100 and 90/100. I remember that later is for blue, and former is perhaps for red channel, but I'm not 100% sure on that one. Next thing that I read in the same post on ZWO forum is that once you set those WB values - they will not change.

Maybe if you set WB values in native driver and ASCOM driver somehow works with native driver to produce images - it is quite possible that ASCOM results are impacted by native driver and remembered WB settings.

This is of course just a theory - maybe even a bit far fetched - but that is only explanation I can offer on this.

Above is taken from internet - it shows sharpcap with ASI294 camera and relevant controls have been marked with arrow.

21 minutes ago, AweSIM said:

4. Can you tell me how you inspected the values inside these FIT files? How can I check that none of the values inside my bias frames are not negative and adjust the offset accordingly?

You can't check for negative values - as you'll never get negative values out of camera. You can only conclude that there is histogram clipping to the left - by examining pixel ADU values and looking at the histogram.

Here are values for subs you posted:

1) Gain 1 offset 0

It has 0 as min - this is very strange. I think it is again consequence of that white balance thing (I have no other explanation for it). All other ASI cameras that I've seen behave like this - if they have less than 16 bit ADC - all ADU pixel values that come from camera will be multiplied with either 4 or 16 - depends if camera is 14bit or 12bit. This is because 12/14bit values are stored within 16bit word in MSB (most significant bit) with LSBs being 0. That makes number values - multiplied by 4 (2^2 for 14bit camera) or 16 (2^4 for 12bit camera). I have never seen 0 as ADU value.

On my ASI178 which is 14bit - lowest number I've seen is 4. On my ASI1600 which is 12bit - lowest number I've seen is 16.

In any case - histogram shows that there is clipping to the left and this combination of offset and gain is not good.

2) Offset 10 and Gain 1:

This clearly shows that histogram is separated from left side and minimum number is 476. Next to it is max number - which is odd number - 1533. You can't have odd numbers if every value is multiplied with 4. Again this shows that values have been altered from default format.

Now important bit - I can't tell if histogram is ok or not as it has been white balanced. I suspect it is ok - but look how far right is peak for blue. It is not that far right because it has that much offset - it is that far right because it has been multiplied with some value - white balanced. What if it was too left before multiplication.

Offset 64 is way too high for this camera - no need to go that high. Offset 10 is quite ok.

Use Gain 120 and offset 10 and I think you will be ok.

Big question remains - have you used SharpCap and if maybe WB settings were changed from that software?

 

 

[AngryDonkey]

Hello

The native ZWO driver has two properties for white balance: red (default 52) and blue (default 95) and as you say they are fixed values. The ASCOM driver does not have them and DOES NOT use the values from the native driver. I've just done two test shots, one with the native driver and one with ASCOM and the ASCOM one comes out totally green and the native one fairly balanced.

Mike

[AweSIM]

1 hour ago, vlaiv said:

This part got me worried as well. I could not find any reference of WB in ASCOM drivers online. In fact, I found one discussion on ZWO user forum that talks about WB settings - and there Sam stated clearly that ASCOM drivers don't have this setting.

I can think of only one possibility for changed White Balance. Have you used SharpCap by any chance at any point? I suspect that there might be issue between native and ASCOM drivers for this camera model. In native drivers you can set WB and it is set to something like 50/100 and 90/100. I remember that later is for blue, and former is perhaps for red channel, but I'm not 100% sure on that one. Next thing that I read in the same post on ZWO forum is that once you set those WB values - they will not change.

Maybe if you set WB values in native driver and ASCOM driver somehow works with native driver to produce images - it is quite possible that ASCOM results are impacted by native driver and remembered WB settings.

This is of course just a theory - maybe even a bit far fetched - but that is only explanation I can offer on this.

Above is taken from internet - it shows sharpcap with ASI294 camera and relevant controls have been marked with arrow.

You can't check for negative values - as you'll never get negative values out of camera. You can only conclude that there is histogram clipping to the left - by examining pixel ADU values and looking at the histogram.

Here are values for subs you posted:

1) Gain 1 offset 0

It has 0 as min - this is very strange. I think it is again consequence of that white balance thing (I have no other explanation for it). All other ASI cameras that I've seen behave like this - if they have less than 16 bit ADC - all ADU pixel values that come from camera will be multiplied with either 4 or 16 - depends if camera is 14bit or 12bit. This is because 12/14bit values are stored within 16bit word in MSB (most significant bit) with LSBs being 0. That makes number values - multiplied by 4 (2^2 for 14bit camera) or 16 (2^4 for 12bit camera). I have never seen 0 as ADU value.

On my ASI178 which is 14bit - lowest number I've seen is 4. On my ASI1600 which is 12bit - lowest number I've seen is 16.

In any case - histogram shows that there is clipping to the left and this combination of offset and gain is not good.

2) Offset 10 and Gain 1:

This clearly shows that histogram is separated from left side and minimum number is 476. Next to it is max number - which is odd number - 1533. You can't have odd numbers if every value is multiplied with 4. Again this shows that values have been altered from default format.

Now important bit - I can't tell if histogram is ok or not as it has been white balanced. I suspect it is ok - but look how far right is peak for blue. It is not that far right because it has that much offset - it is that far right because it has been multiplied with some value - white balanced. What if it was too left before multiplication.

Offset 64 is way too high for this camera - no need to go that high. Offset 10 is quite ok.

Use Gain 120 and offset 10 and I think you will be ok.

Big question remains - have you used SharpCap and if maybe WB settings were changed from that software?

 

 

Hi Vlaiv,

Thanks for getting back so quickly! I haven't used SharpCap before but I just downloaded to check it out and sure enough it does show what you showed. White balance set to R(52) B(95). I've tried fiddling with it in hopes of resetting the WB by setting both R and B to the same values (52/52) but the bayer pattern still shows. I'll look around on the internet for ways of reliably resetting the WB of this camera. Never thought it could give me this much of a headache! =P 😃

Also, thanks for identifying suitable gain/offset values for my camera. I wouldn't have figured that out myself. But now I'm starting to know how. 😃 Thanks.

Also, are you using FitsExplorer to inspect the FIT files?

PS. The only software that have ever connected to this camera are 1. SGP, 2. ASICAP (the capture software for ZWO). None of these software have any controls for WB so I'm not sure what or who set it. It could have been set from factory even. But I'll look into how I can go about resetting the WB. And I totally get now what you mean by that blue could have been too far left before being multiplied.

Thanks again!

Asim

[AweSIM]

13 minutes ago, AngryDonkey said:

Hello

The native ZWO driver has two properties for white balance: red (default 52) and blue (default 95) and as you say they are fixed values. The ASCOM driver does not have them and DOES NOT use the values from the native driver. I've just done two test shots, one with the native driver and one with ASCOM and the ASCOM one comes out totally green and the native one fairly balanced.

Mike

Hi Mike!

Can you please tell how I can try doing the same? How do I choose which driver to use? I use SGP and ASICAP and both I think use the native driver. But I don't understand where and how can I change the WB. Is there a manual you can point me towards?

Thanks!

Asim

[AngryDonkey]

Sorry I've not read through the entire thread yet but have you tried 'not' using bias frames at all? I don't own such a camera so not 100% sure but I think some people prefer not using bias frames due to the amp glow in some CMOS cameras. Instead they do flat frames, flat darks (to calibrate the flat frames) and darks. Might be worth a shot.

[AweSIM]

1 minute ago, AngryDonkey said:

Sorry I've not read through the entire thread yet but have you tried 'not' using bias frames at all? I don't own such a camera so not 100% sure but I think some people prefer not using bias frames due to the amp glow in some CMOS cameras. Instead they do flat frames, flat darks (to calibrate the flat frames) and darks. Might be worth a shot.

I can try that next. 🙂

[AngryDonkey]

4 minutes ago, AweSIM said:

Hi Mike!

Can you please tell how I can try doing the same? How do I choose which driver to use? I use SGP and ASICAP and both I think use the native driver. But I don't understand where and how can I change the WB. Is there a manual you can point me towards?

Thanks!

Asim

Hi Asim,

Sorry I'm not sure, I did the test with my AllSkEye software in which I've implemented the white balance controls of the native driver. In SGP I'm sure you can just select ASCOM camera and then choose the ASI camera from the ASCOM camera chooser panel (provided you did install the ZWO ASCOM driver).

[AweSIM]

21 minutes ago, AngryDonkey said:

Hi Asim,

Sorry I'm not sure, I did the test with my AllSkEye software in which I've implemented the white balance controls of the native driver. In SGP I'm sure you can just select ASCOM camera and then choose the ASI camera from the ASCOM camera chooser panel (provided you did install the ZWO ASCOM driver).

I'll try this next. I'm pretty sure I haven't installed the ascom driver and only installed the core zwo driver from its website 

[vlaiv]

19 minutes ago, AweSIM said:

I'll try this next. I'm pretty sure I haven't installed the ascom driver and only installed the core zwo driver from its website 

I think this solves the problem. You need to use ASCOM driver to do your captures.

Once you install that driver, you will have option in SGP to choose ASCOM driver and there you will be able to set gain and offset.

[AweSIM]

50 minutes ago, vlaiv said:

I think this solves the problem. You need to use ASCOM driver to do your captures.

Once you install that driver, you will have option in SGP to choose ASCOM driver and there you will be able to set gain and offset.

Hi Vlaiv, I tried installing the ASCOM driver for ZWO but it started giving me issues. Frequently, no image would be downloaded from the ASCOM driver. So I uninstalled the driver and reverted back to the native driver. I checked out the ASICAP app that I also use to test the camera's live view for focus and stuff and found that there INDEED is a hidden setting about WB for this camera. And as you said, its set to R(52) B(95). Now I can change these values and the image updates accordingly. The issue is, what values do I set it to? I've tried setting both to 95 and then both to 52 and some other values as well. The issue is that now the bayer pattern seen in bias is such that R and B are of equal intensities, but there's a difference between G and R/B channels, What values should I set it to to achieve no WB?

[vlaiv]

29 minutes ago, AweSIM said:

Hi Vlaiv, I tried installing the ASCOM driver for ZWO but it started giving me issues. Frequently, no image would be downloaded from the ASCOM driver. So I uninstalled the driver and reverted back to the native driver. I checked out the ASICAP app that I also use to test the camera's live view for focus and stuff and found that there INDEED is a hidden setting about WB for this camera. And as you said, its set to R(52) B(95). Now I can change these values and the image updates accordingly. The issue is, what values do I set it to? I've tried setting both to 95 and then both to 52 and some other values as well. The issue is that now the bayer pattern seen in bias is such that R and B are of equal intensities, but there's a difference between G and R/B channels, What values should I set it to to achieve no WB?

My guess is that it should be 50/50 for no WB, but I can't be 100% sure.

If you had trouble with ASCOM driver download - did you try changing USB speed parameter in ASCOM driver (lowering it)? BTW for ASCOM driver to work - you need to have both driver installed (just in case if you only tried ASCOM and it did not work).

[AweSIM]

Hi Vlaiv, I did as you suggested (set both B/R to 50/50) and sure enough, the FITS has been tamed.. =P

THANK YOU SO MUCH for helping me figure this out!!! You have been a GREAT help! Keep rocking!

Asim

[AweSIM]

Hi @vlaiv,

Apologies for re-awakening a resolved thread, but I have some follow up issues I'd like to hear your thoughts about. My earlier issues of white-balanced BIAS frames has been resolved. However for the past few days, I have been frustrated by FLATS. No matter what I do, my flats are not calibrating my LIGHTS correctly. I've tried both DSS and PI. Both are resulting in horrific images.

I used ZWO ASI294MC Pro cooled down to 0C for all frames. The telescope is an 8" F5 Newtonian with a coma corrector and Optolong L-Pro filter threaded into the camera. The imaging train was not touched at all during any exposure. I shot:

1. LIGHT frames of length 300s.
2. Capped telescope and shot BIAS frames of length 0s.
3. Capped telescope and shot DARK frames of length 300s.
4. Covered telescope with white t-shirt, pointed at an evenly-lit wall and shot FLAT frames of length 4s to reach an ADU of 3,500, and of length 40s to reach an ADU of 30,000.
5. Capped telescope and shot DARK_FLAT frames of length 4s and 40s.

I've tried a number of different workflows, but as soon as I stretch the final stacked image, I see clear vignetting and dust motes on the image. Here are the different approaches I've tried:

---[ DSS ]---

1. Loaded LIGHT (300s), DARK (300s), FLAT (4s, DARK_FLAT (4s) and BIAS (0s).

2. Loaded LIGHT (300s), DARK (300s), FLAT (40s), DARK_FLAT (40s) and BIAS (0s).

3. Loaded LIGHT (300s) and FLAT (4s).

4. Loaded LIGHT (300s) and FLAT (40s).

---[ PI ]---

5.1. Integrated BIAS into MASTER_BIAS.
5.2. Calibrated DARK (300s) with MASTER_BIAS.
5.3. Integrated calibrated DARK (300s) into MASTER_DARK.
5.4. Calibrated DARK_FLAT (4s) with MASTER_BIAS.
5.5. Integrated calibrated DARK_FLAT (4s) into MASTER_DARK_FLAT.
5.6. Calibrated FLAT (4s) with MASTER_BIAS and MASTER_DARK_FLAT.
5.7. Integrated calibrated FLAT (4s) into MASTER_FLAT.
5.8. Calibrated LIGHT (300s) with MASTER_BIAS, MASTER_DARK and MASTER_FLAT.

6.1. Integrated BIAS into MASTER_BIAS.
6.2. Calibrated DARK (300s) with MASTER_BIAS.
6.3. Integrated calibrated DARK (300s) into MASTER_DARK.
6.4. Calibrated DARK_FLAT (40s) with MASTER_BIAS.
6.5. Integrated calibrated DARK_FLAT (40s) into MASTER_DARK_FLAT.
6.6. Calibrated FLAT (40s) with MASTER_BIAS and MASTER_DARK_FLAT.
6.7. Integrated calibrated FLAT (40s) into MASTER_FLAT.
6.8. Calibrated LIGHT (300s) with MASTER_BIAS, MASTER_DARK and MASTER_FLAT.

7.1. Integrated DARK (300s) into MASTER_DARK.
7.2. Integrated DARK_FLAT (4s) into MASTER_DARK_FLAT.
7.3. Integrated FLAT (4s) into MASTER_FLAT.
7.4. Integrated LIGHT (300s) into MASTER_LIGHT.
7.5. Used PixelMath to implement (LIGHT - DARK) / (FLAT - DARK_FLAT).

8.1. Integrated DARK (300s) into MASTER_DARK.
8.2. Integrated DARK_FLAT (4s) into MASTER_DARK_FLAT.
8.3. Integrated FLAT (4s) into MASTER_FLAT.
8.4. Integrated LIGHT (300s) into MASTER_LIGHT.
8.5. Used PixelMath to implement (LIGHT - DARK) - (FLAT - DARK_FLAT).

In each of these cases, I'm left with images that have over-corrected or under-corrected vignetting and dust-motes after stretching the final image. Can you help me figure out what exactly is going wrong with my setup? I can say for sure that till about a week ago, both DSS and PI (first workflows of each category) were working perfectly fine for me and I didn't even need to inspect the frames since there were no flaws in the final images. For reference, here is what I ended up about a week ago: M42 Orion and NGC2244 Rosette. PS. You helped me solve the WB issue after these images. So there is something clearly wrong with my approach all of a sudden. Or something is wrong with the camera. =/

M42_Light_300sec_frame1_0C_20200201_231925.fit Calibration_Bias_0sec_frame1_0C_20200202_012654.fit Calibration_Dark_3.9sec_frame1_0C_20200202_011756.fit Calibration_Dark_40sec_frame1_0C_20200202_011910.fit Calibration_Dark_300sec_frame1_0C_20200202_002537.fit Calibration_Flat_3.9sec_frame1_3C_20200202_010618.fit Calibration_Flat_40sec_frame1_1C_20200202_010725.fit

Edited February 1, 2020 by AweSIM

[vlaiv]

Your flat subs look good - both of them together with flat darks.

In fact - this is what they look like after flat dark subtraction and stretch:

Both look the same. Short one is obviously more noisy but they contain the same dust shadows and vignetting.

This is of course when we inspect them by eye, but there is much better way of inspecting two flats if they match - we do what is called - flat/flat calibration - or we calibrate one flat with the other - result needs to be pure noise and gray image without any obvious features. Here is what it looks like when I divide two flats together:

This is sort of text book example what flat/flat calibration should look like - histogram is beautiful bell shaped curve with value of about 0.1 (this is because we divided 4s flat with 40s flat - a bit less than that because it was 3.9s flat in fact) and the image itself is just pure noise - no features to be seen.

Let me see what happens when I try to do calibration of that one sub to see if there is something else wrong (it's not flats usually when flat calibration is not working - it's darks or light leak or some other thing but rarely flats, unless of course flats are clipped - but they are not in your case).

Ah yes - everything is fine with your flats - they calibrate out ok, except one thing - dust is no longer where it used to be:

If you look at this image - which is calibrated single sub and then binned x6 to get enough SNR (this of course looses the color, but we don't care now - we want to see what is going on with calibration) - it looks fine with exception of perhaps two dust particles - that moved. vignetting is fine - it is neither under nor over correcting, and if you look at above flat - let's do another screen shot of it:

I marked here with two red arrows - tow shadows of dust particles that moved in the mean time, and with blue arrows - all other dust particles and their shadows that simply calibrated out fine and can't be seen in light sub above.

This happens sometimes and there is pretty much nothing you can do about it - except calculate distance of surface that has these dust particles (either coma corrector or filter) and then use blow bulb from time to time to blow away any loose dust particles so this would not happen again.

Sometimes people have this issue with filter wheel if their filter wheel can't reposition precisely each time - any small shift will create above effect - it's like "double" or "embossed" dust doughnut. Here it is on one of my images - sometimes it looks so strange that people ask if that is some sort of planetary nebula they have not spotted before

Hope this helps

 

[AweSIM]

Hi @vlaiv

Thanks a million for clarifying it all to me. When I took the subs, I pointed the scope to the west and later when I took the calibration frames, I swung the scope 180 degs around to face a lit wall and take flats there. So yes, there is a possibility that any dust would have moved inside. I guess I've been fretting over this stupid issue for a week, wasting night after night, when the simple (correct) solution was to clean my optical train. 😃

BTW, I'm amazed and inspired by how much you know your stuff. And not just that but the science and reasoning behind it. Thank you so much for not just telling me what I did wrong but also explaining and showing exactly what I did wrong. Thank you so much once again. 😃

[AweSIM]

34 minutes ago, vlaiv said:

Ah yes - everything is fine with your flats - they calibrate out ok, except one thing - dust is no longer where it used to be

This reminds me of a quote I read somewhere around here: Dust never sleeps. It keeps shifting.

[iapa]

On 28/01/2020 at 00:12, vlaiv said:

Btw, for ASI294 I would use gain 120 - as that puts camera into low read noise domain, and I would raise offset until I get very nice histogram separation from the left side on my bias subs. Not sure what that should be with ASI294, but with my ASI1600 I use offset of 64.

 

I've just been looking at offset on my ASI1600, and tried 64, which you suggested works for you.

Just tried it my self with a single BIAS fram using SGPro and it seems to have lifted the bottom end up nicely for em, so, I'll stick with that for a few months.

 

Thanks for the tip

[Dan13]

On 28/01/2020 at 09:52, vlaiv said:

Ok, here are my findings:

1. There is no light leak so you are ok with that.

What I suspect is going on with bayer pattern in the darks / bias subs is some sort of white balance in drivers. I'm not familiar with drivers of this particular camera, but if you turned on automatic white balance in drivers - you might want to turn that off.

I'm suspecting AWB because this camera is 14bit ADC camera - that means that all values in recorded raw sub should be divisible by 4 (16 bit total, 14 bit used - remaining two bits set to 0 - same as number being multiplied by 4). It is not case with your bias and dark files - there are odd values as well as even. This can happen only if values have been changed once they were read out of the camera - for example AWB process in drivers could explain that. AWB could also explain bayer patter. Since there is some offset to pixel value (pixels in bias are not 0, but rather positive value roughly equal for each pixel) - multiplying that with different factors for each color will create different values. Even values in bayer matrix show this could be so - since strongest is blue then red and green is the weakest. If you look at ASI294mc QE curves - it is other way around - green is strongest, red is second and blue is the weakest - which means that they have to be multiplied with certain factors to "even out" giving strengths of pixels in bias/dark that we see.

In any case - I would recommend turning off auto white balance in drivers. With that turned on, I can't judge if offset of 10 is good value because I can't see actual read out values of pixels but only WB corrected values.

Another thing - AWB should not mess up your calibration and everything should work fine.

2. Your subs indeed work fine. Here is comparison on one sub between proper calibration and raw sub without calibration:

This is the same sub - left has been calibrated by (ligth - avg_dark) / (avg_flat - avg_flat_dark) while right one is without calibration. Both subs were binned x8 to get enough SNR to show features of the image. As you can see - you get pretty decent sub with calibration - no vignetting and it looks clean in comparison to raw sub.

As you see - another confirmation that above is probably AWB, as that is not going to affect calibration, and indeed calibration works fine.

3. I would just recommend you in the future to use Gain 120, to turn off AWB and check if offset has good value - and continue taking nice images

HTH

Hi @vlaiv apologies to jump on an old post here and disturb you but you made some points that may help my issue. I imaged to Veil nebula last week that turned out ok considering integration time spent, with that i moved onto another target. This time using unity gain and offset as previous was at 0 my histogram was clipping in the black levels, i bumped it up to 30 and things seemed ok....

Ive come to process 4 hours of data and im really not happy with the noise levels at all...

!. can bumping the offset up cause extra unwanted noise

2.what do you mean by making sure the "gain value" is ok?

many thanks for any input, i also have a 294mc pro recently coming from a 600d

Dan