Problem with the ASI294MC sensor as indicated by flat frames

[alcol620]

Hoping someone can give me feedback on the cause of spotty pattern on flats. This seems to have appeared a few weeks ago. Using Pixinsight, I have stacked around 30 flats callibrated with dark flats. The Pixinsight rejection-high image resulting from calibrating and stacking the flats is attached illustrates the area of distortion on the image. This also appears on the master flat image, to a lesser degree. Using the flat to calibrate the lights leaves slightly darkened area on the stacked light frame over the samea area as appears in the flats.
I also shot the flats with the same settings but with the camera rotated. The distortion appeared in the same place on the image. Suggesting that the problem is inside the camera. I opened the camera and cleaned it. It made little difference, apart from adding a few dust motes. Is there some sort of problem with the sensor? the flats are 9s duration and ADu around 35000. I have attached an image showing the area rejection-high  and a stacked flat frame. Feedback very welcome.

Edited August 2, 2023 by Cornelius Varley
background colour to text removed

[andrew s]

I had the same issue a while back. It's ice crystals. I fixed it by cooling slowly over 20 mins. Other keep it above 0c.

Regards Andrew 

[alcol620]

Many thanks Andrew, it didn't occur to me that could be the problem, although on a couple of occasions the issue didn't arise. I am cooling to -13deg in about 5 minutes for flats, dark-flats and lights. I assume that only cooling to 0deg will give more image noise. Next time out, if we ever get a clear night, I will cool down much more slowly like you suggest. You have raised my hopes of a solution to the problem that has caused the loss of lots of subs. Thanks again Ale

[Anthonyexmouth]

Yeah, definitely frost. I cured mine, after much hair pulling, by doing an argon purge on the chamber. That was last summer and been fine since. I was getting the issue even when I slow cooled. 

[alcol620]

Thanks Anthony   Argon purge?

[Anthonyexmouth]

16 hours ago, alcol620 said:

Thanks Anthony   Argon purge?

Yeah, I tig weld at home so always got argon on hand. I made a little enclosure, filled with argon and reassembled the camera in the argon environment. 

[alacant]

Hi

294, so isn't this simply a case of putting the dessicant tablets in the microwave oven for a minute or so?

To gain access, remove the 4 screws at the sensor end and use tweezers to extract the tablets.

Cheers 

[alcol620]

Next on my list to try, but first I am checking if the problem goes away if I slowly cool to just above freezing. If the fails the the tablets will get dried. Thanks for you  help.

 

[alcol620]

Hi Folks       first test completed. set target temperature at 1 deg and took 10 minutes to cool down, left to run for 10 minutes. Took 30 flats usual settings and stacked them. Result: no indication of the usual noisy area.
Then cooled the camera down to -13deg left for 10 minutes and took another 30 flats. Result: the usual noisy area appeared.
Job for tomorrow: dry the tablets and see what the result is.
One thing that puzzles me it that if it is dampness inside the camera, why does the noisy area always appear in the same area of the image? I would have thought that the moisture would move around and the crystals, whatever they are, would form on different parts of the sensor?

[Adam J]

On 02/08/2023 at 16:51, alcol620 said:

Hoping someone can give me feedback on the cause of spotty pattern on flats. This seems to have appeared a few weeks ago. Using Pixinsight, I have stacked around 30 flats callibrated with dark flats. The Pixinsight rejection-high image resulting from calibrating and stacking the flats is attached illustrates the area of distortion on the image. This also appears on the master flat image, to a lesser degree. Using the flat to calibrate the lights leaves slightly darkened area on the stacked light frame over the samea area as appears in the flats.
I also shot the flats with the same settings but with the camera rotated. The distortion appeared in the same place on the image. Suggesting that the problem is inside the camera. I opened the camera and cleaned it. It made little difference, apart from adding a few dust motes. Is there some sort of problem with the sensor? the flats are 9s duration and ADu around 35000. I have attached an image showing the area rejection-high  and a stacked flat frame. Feedback very welcome.

As others said Ivan on the sensor caused by moisture in the chamber, it's just going to keep getting worse. Best bet is to open it up and refresh the desiccant. 

Adam

[Adam J]

2 hours ago, alcol620 said:

Hi Folks       first test completed. set target temperature at 1 deg and took 10 minutes to cool down, left to run for 10 minutes. Took 30 flats usual settings and stacked them. Result: no indication of the usual noisy area.
Then cooled the camera down to -13deg left for 10 minutes and took another 30 flats. Result: the usual noisy area appeared.
Job for tomorrow: dry the tablets and see what the result is.
One thing that puzzles me it that if it is dampness inside the camera, why does the noisy area always appear in the same area of the image? I would have thought that the moisture would move around and the crystals, whatever they are, would form on different parts of the sensor?

The cooler is not in direct contact with the sensor. The pins for the sensor are on the rear so heat is conducted away via these, causes some none uniformity in cooling. 

Once the tablets have been refreshed you need to leave them to dry the chamber for 48 hours ish before re-testing. 

Adam

Edited August 3, 2023 by Adam J

[Oddsocks]

15 hours ago, alcol620 said:

“One thing that puzzles me it that if it is dampness inside the camera, why does the noisy area always appear in the same area of the image? I would have thought that the moisture would move around and the crystals, whatever they are, would form on different parts of the sensor?”

——————-

When ice crystals form for the first time on the sensor they are usually impure, made of a mixture of water gas and other contaminants, electrostatically bonded to the airborne water molecules before they coalesced and froze on the surface of the sensor glass.

Some of those contaminants will be sub-micron sized crystalline dust particles, too small to be seen visually except with an electron microscope, and not visible in the images that the camera produces.

When the sensor warms slowly after the cooler is switched off the ice crystals may sublimate directly back into water gas carrying the tiny dust particles with them, or, if the dust carries a strong enough residual static charge it may stick to the sensor glass until it loses that charge and drifts off somewhere else, many days/weeks/months later.

If the ice crystals are large enough they may undergo a phase change and will turn to liquid water droplets on the surface of the sensor glass, before slowly evaporating back into a gas, and in doing that any microscopic contaminants that were mixed into the water droplets are left behind, stuck firmly in place by electrostatic charge, or bonded to traces of other organic and inorganic gaseous materials that were present in the sensor chamber and attracted to the water molecules as they coalesced to form ice crystals.

The next time that the sensor is cooled those microscopic dust particles left stuck to the sensor glass will act as “seeds” that allow water in gas form to clump together and form ice crystals again.

And so, once ice has formed on the sensor one time the tiny traces left behind after it has evaporated make it much more likely that ice  will form in exactly the same place again, over multiple cycles, ad-infinitum, and until the sensor glass is physically decontaminated or the water gas is removed from the chamber.

The initial shape of the ice field that appeared on the sensor glass the first time that water gas froze into ice crystals is defined by the distribution of heat transfer across the back of the sensor where the Peltier element is not in complete contact across the full width/height of the sensor and a temperature gradient exists as the sensor cooled down to sub-zero temperature, plus the initial “cleanliness” of the sensor cover glass where any pre-existing microscopic contaminants, fingerprint residue or manufacturing polishing flaws will provide the “seeding” sites for ice to coalesce.

HTH.

Edited August 4, 2023 by Oddsocks
Missing OP quote

[steppenwolf]

Excellent answer from Oddsocks 👍

[alcol620]

Thanks everyone, especially the fulsome answer from oddsocks. So many helpful and knowledgeable people on here. I am awaiting the next sunny and hot day to take the camera apart in my extremely hot and dry conservatory to deal with the tablets. Reassuring to know that you are still watching over us Steve.

[alcol620]

Hi  folks.  Followed the process of drying the inside of the camera. Took the 4 tablets out 2 minutes in the microwave, cleaned the sensor and put the camera cover back on, waited 48 hours for tablets to do their thing.  Just took some new flats. First ones at +1degC 10 minute cool down time. Took the second set of flats at -13deg C 10 further minutes to cool down. Result, I still have this noisy area on the flats, both at 1deg and -13deg. I have attaached images to illustrate the noisy area on the flats, including the "rejection high" images produced by Pixinsight and the resultant master flat.

Any further ideas folks? Help gratefully received

[Oddsocks]

Couple of ideas.

I’m not convinced that rapid drying of desiccant tablets in a microwave oven is that effective for large solid tablets, there isn’t sufficient time for the absorbed water in a bulky tablet to migrate to the surface and boil off, it’s a different situation with desiccant beads where the bulk is much smaller, plus some microwave ovens go into a no-load protection mode if the oven detects there is nothing in the oven absorbing microwaves.

Try conventional oven drying of the tablets and get them back into the camera as soon as they are cool.

The artefact still looks like ice and in shape it appears to be a fingerprint, perhaps that’s just a coincidence but there have been similar fingerprint shaped artefacts reported with ZWO cameras before and these were cured by cleaning the sensor with one-wipe (single use) double-ended sensor swabs and sensor cleaning fluid (Amazon etc).

When the cameras are hand assembled on the production line they should be wearing breathable protective gloves but you can imagine the gloves may get a bit grubby and salty after a few hours of wear and the chance of leaving a salty, greasy glove-print on the sensor must be fairly high.

Try wet cleaning the sensor using two or three one-wipe (single-use) double-ended swabs from a DSLR sensor cleaning kit and bake the desiccant tablets in an old-tech convection oven for at least an hour.

(Make sure to select the right sized swab kit for your sensor, a swab that is too narrow for the width of the sensor will leave a residue trail across one edge, the swab used must be the right size to cover the full width of the sensor in a single swipe)

If the problem remains after that then I have no idea what the cause could be and maybe the camera manufacturer tech support would have seen this problem before and have more appropriate advice for you than can be offered here?

HTH

 

Edited August 10, 2023 by Oddsocks

[alcol620]

Thanks HTH much appreciated. I took some flats earlier this evening at 5degC to remove the ise of ice forming and an initial look suggested no noisy pattern. Now taking lights at the same temperature to check out the result. I have "cleaned" the sensor a couple of times although not with a wide swab. Not of that cleaning has changed the pattern significantly, but I will get some wider swabs and give it another go. I will also give the tablets a bake in a conventional oven.

[alcol620]

A quick update. Calibrated with new flats and stacked the new lights taken at +5degC and no suggestion of any noisy pattern on the master light. Awaitingthe arrival of the wider swabs and will then re-clean the sensor and bake the tablets as suggested.

[alcol620]

A belated update. Since my last post I dried the existing tablets in a conventional oven, again cleaning the sensor. Replaced the tablets. Result no change. I bought new tablets from FLO, put them in the oven for an hour and put them in the camera. The new tablets were left in the camera for 10 days before the camera was next used. The result was no change, I still had the same noisy pattern in the same area of the subs with the camera cooled below 0deg C. Above this figure no noisy pattern appears. I have wasted so much time and mental effort trying to resolve the problem and have decided that the only option is to take subs a couple of degrees above freezing. I assume this will make subs a bit noisier but at least the noisy area does not appear? Sending the camera back to ZWO and losing it for 2/3 months and having to pay for whatever is found is not an option. Any further comments are very welcome

[LandyJon]

Saw your post a while back but couldn't add to what had been said already.

Sorry to read your still having trouble, at this stage maybe try replace the sensor window.

https://www.firstlightoptics.com/zwo-accessories/zwo-ar-replacement-protective-windows.html

I presume you followed this guide when you recharged the dessicant 

https://astronomy-imaging-camera.com/manuals/How_to_clean_ASI_camera_and_redry_the_desiccants_EN_V1.2.pdf

Not something I've done, but if you've already had it apart replacing the window and cleaning the actual sensor could be whats needed.

Edit oh you said you'd cleaned the sensor already, not sure this will help, you did both sides of the window too I guess ?

Have you tried a flat frame without the sensor window, it would at least confirm if the error is on the sensor or the window.

Edited September 23, 2023 by LandyJon

[symmetal]

Some Zwo cameras are not airtight around the sensor with some models being worse than others. The internal cable assembly from the airtight sensor PCB to the rest of the electronics should be fully sealed with silicone rubber or similar, where it passes from the airtight chamber to the outside. If it isn't fully sealed then the desiccant tablets can get saturated fairly quickly as they absorb the damp external air leaking through the seal, and so stop working leading to ice forming on the sensor as you've seen when cooled below 0C.

The ASI071 was notorious for this problem so in the end I just cooled it to 0C, when the issue doesn't occur. Changing the tablets or drying them out had little effect as even after a few days kept indoors the problem would reoccur. My ASI6200 will sometimes do this when the humidity is high outside and cooled to -10C so I also just cool this camera to 0C as well to avoid ice issues.

The graphs produced by ZWO for dark current are a bit misleading at first glance, as the y axis is exponential and not linear, and the straight line graph makes you think the dark current reduces linearly with temperature which isn't true.

If you replot the graph with a linear y axis you end up with this graph which shows that at temperatures below 0C the reduction in dark current is not so significant. The lower graph is just the top one expanded to show it better at lower temperatures. The dark current roughly halves for every 10C reduction in temperature. Admittedly the ASI294 has higher dark current overall compared to some other cameras, (for example the ASI6200 has only 0.003 e/s at 0C), compared to the ASI294 at 0.02 e/s, so a lower temperature will give a more significant improvement on the ASI294, compared to the ASI6200.

If we compare that to ASI294 camera read noise which is 1.8 electrons at gain 117, when the HCG (high conversion gain) mode is enabled, how long can we expose an image where the dark current and read noise are about the same. It's about 200 seconds for -10C and 100 seconds for 0C

I don't think you'll notice any difference, using 0C compared to say -10C in actual images when all the other noise contributing factors are taken into consideration. With CMOS cameras the noise from the sky background from will swamp any read noise, (or read noise and dark current for that matter) after a few minutes exposure. Or even less time in areas with more light pollution. This makes the sky background noise the dominant noise source, and the read noise and dark current actually become insignificant.

1 hour ago, LandyJon said:

Sorry to read your still having trouble, at this stage maybe try replace the sensor window.

As the ice crystals or dew are in focus they will be on the sensor itself. If they were on the sensor protect window thay would be well out of focus and just give a fuzzy haze over the image centre.

Alan

[alcol620]

Thanks for the feedback you 2 guys. For clarification when I talked about cleaning the sensor, I now realise that I was only cleaning the sensor window. I am not sure that I have a job description that includes removing the sensor window to get to the sensor itself!!

With symmetal's info on noise, I think I will stick with simply taking subs with the camera temperature set above 0deg C.

Thanks again

[LandyJon]

I wouldn't be too daunted by cleaning the sensor, as long as you're careful and follow the instructions in that pdf I linked from ZWO using a proper sensor cleaning kit off amazon or wherever.

Its just a matter of unscrewing 4 screws (or twisting off the whole end cap on older models) and the sensor is there in front of you.

Ideally with these cameras you want to keep the chamber factory sealed, but since you've already opened the dessicant chamber thats no longer the case, so I don't see the harm in popping the lid off to solve your problem properly.

Best of luck either way.

[Oddsocks]

11 hours ago, alcol620 said:

I am not sure that I have a job description that includes removing the sensor window to get to the sensor itself!!

With symmetal's info on noise, I think I will stick with simply taking subs with the camera temperature set above 0deg C.

Thanks again

Cleaning the sensor (cover) glass is nothing to be concerned about, the official ZWO cleaning document that  @LandyJon linked to describes the steps clearly.

The steps required are really not that different to those that owners of DSLR's who regularly change lenses have to do many times a year, the only practical difference is that with a DSLR you only need to remove the lens, open the shutter and lock-up the mirror to reach the sensor, with your camera you have to unscrew the chamber cover to reach the sensor, and, with a DSLR there is a comparatively soft and easily damaged antialiasing and IR filter array directly over the sensor cover glass, in your ZWO camera those additional filters are not installed and the harder quartz-glass sensor window is exposed.

Alan, @symmetal, describes the minimal effect on dark current of running at temperatures above 0c and while it is true that the difference to linear dark current is very small with increasing sensor temperature of these CMOS cameras you might see slightly more "pixel speckle" in the resulting images due to the non-linear response to increasing sensor temperature of warm pixels.

That effect can be mitigated though with dithering and a sigma reject algorithm used when combining the stacked images.

The only concern with leaving the sensor with moisture "seed" traces on the sensor cover glass and running just above 0c is that you risk moisture building up on the backside of the sensor where the cold finger from the TEC cooler abuts the sensor back plate, this area will be a few degrees below freezing while the sensor itself will be at 0c and if the moisture on that interface lingers it can spread over time and corrode the tracks on the PCB.

If you run a cooled camera at 0c to avoid the appearance of ice on the front of the sensor glass you'll never know how damp the sensor chamber is becoming. The sudden appearance of ice crystals on the front of the sensor when running just below 0c is a useful, albeit a nuisance, indicator of the humidity levels inside the sensor chamber.

William.

 

Edited September 25, 2023 by Oddsocks

[Bukko]

Following this thread with interest...

My 294MM has the same issue and cannot run it below 0c.

I have purchased new dessicant tablets and plan to change over in October. I am surprised that someone in the factory would think it OK to touch the chip surface and the QC team should be working hard to stop this from happening. So I now also need to clean the chip as well..

Anyway, my plan to open the camera includes using a glove bag - a large sealable bag with built in gloves. This is in an attempt to prevent unwanted dust getting inside. Also plan to use an argon purge so the dewpoint inside the bag should be lower than -50c, so at least I can start the new dessicant with a dry chamber.

If there is any interest, I will post the work here.

Gordon.