Flats problem or something else

[philherbert]

Hi All,

I have recently converted from an old Canon DSLR to a ZWO ASI1600MC-Pro. The individual subs look great with the new camera, but I am having seemingly insoluble problems processing, so I need the assembled wisdom here! The first image shows a single sub on the left (auto-stretched), and the result of stacking some 25 subs on the right, combined with flats. The flats have corrected quite nicely for the dust doughnuts, but given me inverted vignetting. (I should say that I have tried processing with darks and bias as well, but the basic problem of the reversed vignetting remains. Processing was done in Deep Sky Stacker, although I get the same results using PI and the BPP script).

I initially thought this was down to using the wrong ADU values for the flats, so I've tried various from 5,000 up to 30,000 in 5,000 increments, but nothing improves - the higher the ADU the worse the effect seems to get. I've also tried different techniques for taking the flats - grey skies (easy at the moment) with t-shirt layers in front, or a tablet screen on white. Flats are seemingly ok - here is an individual sub on the left and master flat on the right. OK, they are rather brownish, but apart from that they look ok to my (DSLR-attuned) eye. Different ADU values seem to fail to correct for the dust doughnuts, as well as making the vignetting worse.

I have one further thought / problem to add. The over-corrected centre of that processed image (top, right-hand side) shows some odd details. Here's a zoomed version. There are fairly broad swirly lines showing up in there. Actually the lines are across the whole image, but most visible in the centre where the vignetting shows it up.

My only thoughts are that this could be condensation or similar? Might this account for the flats not correcting properly? As I said I'm using the ASI1600MC-Pro, and have it set to cool to -20C for all lights, darks and flats. I've not encountered a problem like this with flats failing to work so miserably, so I'm after any and all knowledge at this point.

To a fair extent I can take out the vignetting with ABE/DBE, but that then leaves me the dust doughnuts. I really don't want to have to 'edit' my pictures to that extent, and feel that the flats should take care of the vignetting.

Thanks for looking.

Phil

[vlaiv]

Could you post single each of:

Light, dark and flat and dark flat subs?

Just raw fits files without any processing done to them (no bias removal from dark, flat and dark flat).

[philherbert]

Sure. This are examples of the light (top left), dark (top right), flat (lower left) and bias (lower right). All are raw subs, just auto-stretched with the STF tool. 

[vlaiv]

No, not jpegs of stretched files, actual fit files, so they can be inspected for values

[philherbert]

Sorry, my bad. Here you go. Light, dark, flat, bias in that order. Phil

https://www.dropbox.com/s/4erb24i7brxbsq9/2018-01-07_18-44-14_L_180s_G139.fit?dl=0

https://www.dropbox.com/s/5w1q7rmm6siyvdo/2018-01-14_10-38-21_D_180s_G139.fit?dl=0

https://www.dropbox.com/s/49hh0d4qoo8y3u7/2018-01-21_12-23-59_F_1.55s_G139.fit?dl=0

https://www.dropbox.com/s/yjp7prj6gcrbf4v/2017-12-30_13-12-09_B_0.001s_G139.fit?dl=0

 

[ollypenrice]

Yves and I were plagued by inverse vignetting arising from flats with one camera we used. We were capturing in Nebulosity. Eventually I captured some flats in AstroArt and these worked. Harry Page had suggested that inadequate flushing between flats might have been the cause. I suspect he was right.

I have also read that the capture mode used to make the flats can cause this problem if it is not the same mode as was used to take the lights. Are you being consistent here?

Inverse vignetting can also be caused by flats which are not calibrated with darks-for-flats. I use a master bias as a dark-for-flats (AKA flat darks). In my struggles with Yves' camera I tried dedicated flat darks, bias as flat-darks, no flat darks but our problem was not arising from any of these issues.

Olly

 

[alexbb]

When calibrating with PI, disable master calibration files optimization.

[kens]

The ASI1600 seems to give inconsistent bias frames so it is generally recommended to calibrate the flats with flat darks rather than bias.

[vlaiv]

1 hour ago, philherbert said:

Sorry, my bad. Here you go. Light, dark, flat, bias in that order. Phil

https://www.dropbox.com/s/4erb24i7brxbsq9/2018-01-07_18-44-14_L_180s_G139.fit?dl=0

https://www.dropbox.com/s/5w1q7rmm6siyvdo/2018-01-14_10-38-21_D_180s_G139.fit?dl=0

https://www.dropbox.com/s/49hh0d4qoo8y3u7/2018-01-21_12-23-59_F_1.55s_G139.fit?dl=0

https://www.dropbox.com/s/yjp7prj6gcrbf4v/2017-12-30_13-12-09_B_0.001s_G139.fit?dl=0

 

Yes indeed it is over-correcting and I suspect that the reason it is doing so is that flats were taken at 1.55s and you are using bias files to calibrate flats - 0.001s

I've found that bias files are not reliable with ASI1600, and in reality there is no need for them.

Try taking a set of flat-darks with exact same parameters as flats - gain 139, same offset, -20C and 1.55s exposure and calibrate flats with those - I think it should solve the problem.

You can do this without scope, just connect camera, get it cooled and take good number of dark flats - keep camera covered of course (some suggest using alu foil with plastic cap - I just use regular wooden table and turn camera so it is resting on plastic cover - "sensor down" position).

[ollypenrice]

An interesting difference between CMOS and CCD. Good information here.

Olly

[steppenwolf]

1 hour ago, vlaiv said:

Yes indeed it is over-correcting and I suspect that the reason it is doing so is that flats were taken at 1.55s and you are using bias files to calibrate flats - 0.001s

I've found that bias files are not reliable with ASI1600, and in reality there is no need for them.

Try taking a set of flat-darks with exact same parameters as flats - gain 139, same offset, -20C and 1.55s exposure and calibrate flats with those - I think it should solve the problem.

Thank you, this is a very interesting set of observations. We are discussing the ASI1600 here but are Bias frames generally an issue with the new CMOS sensors or just this one?

[vlaiv]

40 minutes ago, steppenwolf said:

Thank you, this is a very interesting set of observations. We are discussing the ASI1600 here but are Bias frames generally an issue with the new CMOS sensors or just this one?

I've found that some CMOS sensors have issue with bias frames (with my limited experience with them), and can be split into two categories:

1. Dynamic offset - There are quite a few older sensors that do some sort of internal calibration of bias level, so each time camera goes thru power up cycle - and that can be when it is connected to USB or even when drivers initialize, one ends up with different bias mean levels. This impacts darks and lights as well. Here solution that I've found to be working is that both lights + darks be taken in same power cycle - shooting lights then without powering off camera or even capture application, one should cover up scope and take set of darks. Same goes for flats / flat darks. This was mostly on non cooled cmos cameras, so taking darks after each session is certainly advised.

2. Something I've noticed only on ASI1600MMC - mean ADU for bias frames is higher then 1 minute dark sub (without any calibration). Mean ADU levels actually drop with longer exposure and then stabilize at some point - can't remember exact value but I think it is between 10 and 30s, after that they seem to start acting normally - longer sub larger mean ADU - as one would expect with dark current present.

ASI1600 can be calibrated out fine as long as bias frames are not part of equation. There is good stability between sessions at certain sub length. So I can do 2 sets of darks on different days, and they will have same mean adu, and if I create two stacks, one for each day and subtract - I get "flat" 0 ADU mean random noise residue. This means that darks can be shot once and used afterwards for multiple sessions.

For flat calibration I use flat-darks. This just means same settings and exposure duration, and I get really good flat calibration out of it. My flats are really short, order of few ms, so almost "bias" in nature, and there is stability between such short darks as well. I usually shoot set of flats each time and add set of flat-darks (because it takes me like 10 - 15 minutes to do x256 of each, due to short exposure), but once I used same set of flat darks across filters - shot one color, then flats and flat-darks, then changed filter (no filter wheel, but rather filter drawer, so filter orientation is not guaranteed), but shot only flats this time (for second filter after lights) and it worked fine as well.

So in general I would advise owners of CMOS sensors to do some simple measurements when they start to use camera to see how it behaves - just compare different exposures for mean ADU, create multiple stack and subtract and such, like basic diagnostic, and based on findings create calibration routine that works.

[philherbert]

Thanks to all for the suggestions. I'm just creating some flat darks (never used those before) matching the flats and will report back... 

[philherbert]

Hmm, I've created a set of dark flats / flat darks with same temperature, bias, offset and exposure duration as the flats, just without the light :-) . I've just run them through with Deep Sky Stacker and get no appreciable difference - the vignetting is still over correcting in the centre of the frame. Here's a link to a dark flat in case it is of any help: https://www.dropbox.com/s/tnwdg51aevtir94/2018-01-31_21-20-25_DF_1.55s_G139.fit?dl=0

I've stacked with lights, flats and dark flats (no difference), and also with lights, darks, flats and dark flats (still no difference). I've even thrown the bias in there as well as an experiment (no appreciable difference). 

[vlaiv]

Hm, this is interesting, using flat-darks did improve calibration but it clearly did not remove problem with over correction. Here are my findings:

First is stats for bias file, and second is for flat-dark. From noise content (stdev) it is obvious that second one has contribution of dark current noise in it besides read noise as noise is larger. But looking at mean value of each one just confirms my previous findings on ASI1600 - mean of bias is larger than mean value of dark (in this case 1.55s dark). Any longer exposure should have higher mean value, so it should be other way around - like stdev.

Here is comparison of one sub calibration:

Left is calibrated with flat-bias, right is calibrated with flat - flat-dark. To my eye right calibration is a bit better / flatter, except the fact that both still show over-correction in central region.

And here is difference between above images to emphasize what has been changed:

So clearly using flat-darks has improved flatness of calibration on edges and in corners.

But here is interesting find (:D just realized this while I was typing this post and fiddling with data):

Central bright spot looks familiar? This is dark stretched very hard. It looks like it has some sort of bright central smudge.

My guess is that there was some sort of light leak while you were taking darks, and that calibration problem is not due to flats at all but due to darks. Try taking another set of darks, making sure that there is no light leak to see if pattern repeats itself.

 

[vlaiv]

One more thing to check out, it just occurred to me what also might be a problem.

Such bright spot in darks might have something to do with camera it self. There might be a problem with the way thermal compound was applied between cold finger and sensor itself. Maybe there was a pocket of air trapped or something like that and that central spot is just not cooled as efficiently as the rest. 

This in it self should not be a problem except if it changes in time - if trapped air is "breathing" and it changes temperature without regulation. This would mean that same darks taken on different occasion will have different amount of central bright spot - something really hard to calibrate out (it is almost like no temperature regulation). It also means that lights will have different levels of dark current in them.

Do another set of darks and see if you can spot the same central bright smudge in any of them. Also inspect each of darks that you already have and see if bright spot is present in each and if it changes in intensity / position.

[philherbert]

Hi Vlaiv, thanks for your analysis! I have checked the other darks, and there are certainly some variations between them. I have also compared them to another set of darks that I took under much better circumstances and there is much less variation between them - I used 'blink' to compare them. Unfortunately, processing again with this better set of darks still leaves the inverse vignetting.

I was thinking - if the darks are over-correcting, can I 'reduce' the effect of them somehow (perhaps using Pixinsight's PixelMath) do you think?

[vlaiv]

1 hour ago, philherbert said:

Hi Vlaiv, thanks for your analysis! I have checked the other darks, and there are certainly some variations between them. I have also compared them to another set of darks that I took under much better circumstances and there is much less variation between them - I used 'blink' to compare them. Unfortunately, processing again with this better set of darks still leaves the inverse vignetting.

I was thinking - if the darks are over-correcting, can I 'reduce' the effect of them somehow (perhaps using Pixinsight's PixelMath) do you think?

From what I've seen it looks like offending zone is less strong than amp glow. That is problem, and I would advise you to get to the bottom of it - find out what is really going on.

In theory one would be able to "isolate" offending signal in darks by using comparison between different dark subs and then doing some fancy math to minimize it or even completely remove it. Problem is that you can't "partially" apply dark to lights and expect good results since amp glow is dominating. There is also possibility that such "hot" zone is present in lights as well. This means that no "special" math applied to darks only will help, and I think that if it is the case, no special algorithm and arithmetic will be able to isolate that in both darks and lights properly.

Only way to deal with situation in my mind is to try to create set of darks without this effect by taking care of the way you shield sensor when taking darks. Take another set of darks to see if artifact appears again but take every precaution not to have light leak of any sort - to rule out light as a source of problem. If you manage to get clean darks - great, use those to calibrate your images and it should be fine.

At this point I need to ask, you mention that you have another set of darks that you took under much better circumstances. Can you give some sort of description of circumstances and why you believe one set is preferred to another?

In any case if you end up with taking another set of darks and artifact is still present but varies in intensity, I would recommend you to contact your dealer, describe the problem and look into options for replacement of camera with another one.

If there is manufacturing defect that is causing this, like improper application of heat conductive material between sensor and cold finger of cooling system - it will be virtually impossible to do good calibration, and in any case item should be considered defective and replaced.

[philherbert]

Yes you make a good point about not being able to isolate the problem area from the amp glow, so that is not an option.

The original darks you have analysed were taken with the scope lens cap on, but in daylight (on a cloudy day). This was not ideal of course. The second set of darks that I mentioned were taken in the evening with the lens cap on and a large dark towel over the telescope as well. I have looked at these darks and there is only the amp glow, no sign of the central signal. However after processing I am still left with the reverse vignetting from some kind of over correction.

Here is one of the better darks: https://www.dropbox.com/s/uj6joog55nfzgtq/2018-01-21_17-49-53_D_180s_G139.fit?dl=0

(By the way, for anyone else with similar issues, the bright central area was not really visible in the original flats until it was debayered and then given a very strong stretch - it didn't show in the grey version of the dark. In reality it only showed in a few of the original dark subs.)

So in summary I have a fairly good set of darks (unless someone can show me otherwise of course!), and using dark flats I get slightly better calibration than using bias frames, but I still get inverse vignetting. The effect is shown below: 

With PixInsight and the AutomaticBackgroundExtraction tool I get a result looking like:

The left-hand image shows the removed signal; the right-hand image the result. As you can see it leaves a 'halo' around the centre of the image. It also leaves the snaking lines in the background which I think might be symptomatic of another problem. Any takers? I'm running out of idea!

Thanks,

Phil

[vlaiv]

Ok, this is much better than before:

I'm going to take a wild guess here, so correct me if I'm wrong:

Flats were taken during daylight, and your scope is newtonian?

If I'm right, here is explanation for the problem (it is of a good kind, camera is ok, it is all due to light leak):

You need to make sure you don't have light leak in OTA. Inspect main mirror cell to see if any light can get inside on sides of main mirror. Main mirror cell is often "open" so air can circulate and cool main mirror more efficiently, but if not made properly it can also let some of the light in, especially if there is source of bright light shining somewhere at the back of the OTA - like doing flats / darks in daylight or having strong light turned on at the end of the session when you do flats / darks.

You can easily check this: Do this in dark, put the scope cover on, put your eye right at focuser and look at the secondary - you should not be able to see it. Now use some sort of torch to shine at back of the ota - make it fully illuminated, but shield your eye / focuser from light - if you can see difference in brightness on secondary (it should reflect any light coming in OTA from behind) - you have a light leak.

Good practice is to take darks with camera separated from scope with cap on, and shielded from any light. With cameras that have mechanical shutter - it is not problem to do it with camera attached to telescope, but with CMOS sensors (and rolling shutters) you want to minimize stray light.

Also if you identify area of light leak - shield it somehow when taking flats.

 

[philherbert]

My scope is a refractor with a focal reducer. I have just checked and there was a very slight amount of slack in the connection between the focal reducer and the main scope, so conceivably this could have allowed a small amount of light in, potentially affecting the flats and possibly the original set of darks.

It may be that my previous approach that worked fine with an old DSLR just isn't rigorous enough for such a sensitive CMOS sensor. I'll take another set of darks tonight (in the dark so that there is no stray light). I'll also see if I can minimise any stray light and take a new set of flats. And then I'll report back...

 

[Oddsocks]

Hi Phil.

I think you may be looking at this problem the wrong way.

I don't think it is a hardware fault or a calibration process but a very basic oversight with the imaging system as a whole.

While a little light leakage around the focuser and camera to focal reducer is not helping, the major issue is I.R. contamination of the flat frame.

I'm guessing you have fitted the ASI 1600-MC directly to the telescope without an IR-UV blocker in the light path?

The standard ASI 1600-MC camera is supplied with a non IR-UV blocking A.R. port window.

When you look at your single posted light frame, debayer it and in Pixinsight use the STF tool with the colour channels unchained you will see that the image is strongest in the green, as expected because there are twice as many green pixels as red or blue, the red and blue channels are almost identically balanced.

If you look at your posted flat frame, debayer it and perform the same unchained STF stretch your will see that the red channel is many times stronger than the green or blue and this is mostly down to the lack of an IR-UV blocker filter anywhere in the image chain. The light source you have been using is strongly emitting in the I.R. region and this is contaminating your flats.

For flats with a mono camera and separate filters this is not normally an issue since the individual filters are I.R.-U.V. blocked but it can be a big problem with OSC cameras where no I.R.-U.V. filter is used, or in the case of a modified DSLR where the I.R. blocker filter(s) have been removed and not replaced with a broad band I.R.-U.V. filter. With a standard unmodified DSLR or a DSLR that has been partially modified by removing only the near I.R. filter while leaving the far I.R. in situ then I.R. contamination is not such an issue.

When you consider the image chain as a whole the vignetted field is fairly predictable, purely down to the optics and the aperture of any restrictions due to undersize couplings and baffles that are in the light path but added to the vignetted field is all the off-axis light that is reflected off the baffles and connecting rings, focuser tubes etc.

The problem with the I.R light wavelengths in particular is that the black anodised surfaces found inside many focuser draw tubes and camera nose pieces, image chain couplers etc, do not appear black to I.R. quite the reverse, black anodised surfaces become white under I.R. and reflect and transmit I.R. very strongly. This is because the black colour of anodised aluminium is created by a dye rather than a pigment and dyes attenuate different wavelengths by differing amounts.

When you created your flat frames the light source you used would have contained much more I.R. than the actual night-sky star images and the extra off-axis I.R. present when creating the flats would have glinted and reflected off the anodised surfaces creating an exaggerated "donut". Sky flats taken of a grey sky during the day time (with or without a tee shirt) would have contained a huge amount of off-axis I.R. as would flats taken indoors under tungsten lighting. Tablet screens emit quite a bit of U.V. as well as a little I.R. and the washing powders used on the white tee shirt contain fluorescent chemicals that emit in the blue when struck by U.V.

Without an I.R-U.V blocker in the image system the only sky flats that will work properly are those taken of a clear, cloudless, pre-dawn or after sunset sky approx 90 degrees away from the below-the-horizon position of the sun where the amount of I.R. present will be close to that of the dark night sky.

For flats with an artificial light source and an image system with no I.R.-U.V. blocker then an Electro Luminescent (cold emitter) panel would work quite well since no I.R. or U.V. is produced, or an L.E.D. panel that is made up of separate broad-band red, green and blue LEDs (many of the display and draughting panels made only with white LEDs produce the white light by emitting in U.V. and a coating of fluorescent powder on the inside of the L.E.D. glows in discrete red green and blue wavelengths to produce the white light but many with large emission gaps in the visible continuum and a fairly large U.V. leakage).

The easiest solution is to fit a screw-in I.R.-U.V. blocker filter somewhere in the image chain as close as possible to the camera sensor. This will not affect the overall sensitivity of the camera since a purpose made "Astronomical" I.R.-U.V. blocker still passes the wanted Ha and SII bands without attenuation while blocking nearly all the unwanted long wavelength I.R. that goes to create the "donut" appearance of the flats.

If this is not possible, then you can reduce the reflected off-axis I.R. hitting the sensor by painting the anodised surfaces inside the focuser draw tube, any couplers and baffles etc with a black pigment paint that absorbs I.R. High temperature matt-black barbecue paint is ideal for this (from B+Q and other DIY outlets, note that matt black paint is the one required, not the "gloss" version).

This web document explains how unwanted I.R. causes problems with flats:

http://diffractionlimited.com/flat-fields-stray-light-amateur-telescopes/

As an experiment, to prove the excess (I.R.) red of the flat was the problem I took your posted images and instead of calibrating the light frame straight away I first debayered all four images, light, dark, bias and flat, then I split the flat RGB frame into it's three colour channels. Using the LinearFit tool I balanced the red and blue channels using the green channel as the reference and then recombined the three colour channels back into a single RGB flat image. Once done I carried out a standard calibration of the debayered light frame using the debayered bias and dark plus the newly balanced flat. The outcome was an almost perfect flat field correction needing only DBE to remove the remaining gradients and SNCR to remove the excess green noise. A little colour saturation boost was applied to check no remaining colour gradients were present. The resulting image is posted below for reference, obviously very noisy as it is  the result of a single frame but this was only to prove the point that the unbalanced, I.R. contaminated, flat frame was the source of the problem.

This not a real "cure" for the problem though, rebalancing the flat in this way is not going to create a perfectly linear flat though it is quite close and it does involve extra steps that means you can't use the batch debayer script in P.I. For example as a manual process you could try separately creating a master flat, de-bayering the master and splitting the colour channels, apply LinearFit to red and blue with green as the reference, recombine and save as the "balanced" master flat. Create a master debayered bias and a master debayered dark. no need for for the LinearFit tool with these as they are both "in-the-dark" frames, then batch debayer your lights and finally use the (manual) Image Calibration tool, add the debayered lights to the file list plus the master debayered bias and dark with the new balanced flat master and run the calibration tool. Finally take the now calibrated lights and use the combine tool to build the final calibrated and combined image. The outcome should be an almost flat and corrected image ready for DBE gradient removal. Not perfect though, for that you need to solve the I.R. contamination at source.

If you find after taking your new set of flats in as dark conditions as possible that you are still presented with the overcorrection "donut" then consider the above as a possible alternative explanation and I.R. reduction mitigation steps taken should solve the problem, either the use of an appropriate I.R.-U.V. blocker filter included in the image path, painting the anodised metal surfaces in the image path with a mat-black pigment or the use of a non-I.R.-U.V. emitting light source for your flats creation, preferably all three!

HTH.

William.

 

 

[vlaiv]

15 minutes ago, Oddsocks said:

While a little light leakage around the focuser and camera to focal reducer is not helping, the major issue is I.R. contamination of the flat frame.

No, IR contamination will not create over-correcting flats. With OSC camera it will however create wrong WB (correctable in post processing) but since Flat is relative pixel brightness it will not be a problem with additional IR light.

Reddish debayered flat is quite ok for ASI1600MCC if full spectrum continuous light source was used for flats. Look at the surface under the red curve - it is the largest compared to blue and green.

[michael8554]

Interesting theory William, particularly as it cured the problem.

However, in the first post the problem exists with different methods of flats, including T-shirt and sky, not a strong IR source.

Michael 

[philherbert]

On 02/02/2018 at 08:46, philherbert said:

My scope is a refractor with a focal reducer. I have just checked and there was a very slight amount of slack in the connection between the focal reducer and the main scope, so conceivably this could have allowed a small amount of light in, potentially affecting the flats and possibly the original set of darks.

It may be that my previous approach that worked fine with an old DSLR just isn't rigorous enough for such a sensitive CMOS sensor. I'll take another set of darks tonight (in the dark so that there is no stray light). I'll also see if I can minimise any stray light and take a new set of flats. And then I'll report back...

 

Well I've done the new darks being careful to block out any sources of stray light. I've also taken a new set of flats (the dull overcast weather is useful for something). However there is minimal difference when pre-processed, and I still end up with the reverse vignetting.

I'm now wondering whether the problem might lie with the original light subs instead of the calibration subs? (Ok, there were some minor issues with the calibration subs, including the need for dark flats, that I believe I have now sorted out, but I still have the original problem.)

My  theory is that the flats are actually working correctly. My very basic test for this is whether they compensate for the dust shadows, and they do. If this theory is correct, and they are working properly, then they are also correcting for the vignetting. Which leads me to think that whatever remains after pre-processing is some stray light that was collected during the light subs. (When I say stray light, I mean not signal from the target, and not noise inherent in the sensor etc, so sky glow / street lights / house lights / internal reflections).

Apart from re-taking the lights, and trying to minimise stray light, I can see two options: (1) Work out a way to test whether there is any stray light in the subs or (2) Create a synthetic flat.

For (1) I haven't yet thought of a way to tell. There is some noticeable difference between the first sub and the last sub, so I was wondering about subtracting one from the other, once aligned, to see what remains as perhaps a difference in the stray light at the time?

For (2) this article http://trappedphotons.com/blog/?p=756 is very interesting. Has anyone tried anything like this?

Phil