ASI1600 NB flats nightmare - 2 years on and still not fixed!

[Tommohawk]

Hi again! I looked back at my earlier post re flats with the ASI1600 and I see that was 2 years back - and I still haven't sorted it. A number of questions and I'd appreciate any advice please cos TBH I'm on the verge of throwing in the towel with imaging. It's a long thread, with quite a few images but please bear with me!

So at considerable expense I replaced my v1 ZWO narrowband filters with the v2 type. Apparently these can be used without the masks and therefore give the 31mm aperture as advertised (whereas the masks required with the v1 type reduced the aperture to 26mm, even though they are advertised as 31mm..... don't get me started  etc.) But even with the new filters I still get some odd results - maybe its more to do with the camera. Anyhow, the flats look wildly variable and not in any logical way.

Firstly, a few things I've slowly worked out. I discovered that my 300ses NB lights seem odd even before any calibration, in that the borders/corners seem brighter than the centre - not what you  would expect with typical vignetting. However when I look at the dark, its also lighter at the borders. So this is "amp" type glow, or dark current whichever you prefer.  After a lot of thought I figured if I stack the lights with the dark only - no flats - I should then be able to see if there's any true vignetting, and I would then need my flats to be comparable to that. This would at least give me some idea of what the flats should look like.

So below I've posted an Ha light, followed by a dark, followed by the stack with a dark only, all very stretched just to show the effect. This seems to give some sensible results:

Light:

Dark:

Stack with dark only, no flats:

 

But this is where it gets odd. For flats I reasoned, as others have, that the gain doesn't matter, because you're only correcting an optical issue. So although the lights are at high gain (300) I did the flats at Lo gain (139) to minimise noise, (cooler at -15 same as the lights and dark.) I tried both flat panel and plain wall techniques, and both results are odd - way too much variation. Here are the light flats, same stretch as the lights. To complicate things it looks like the filter wheel hasn't aligned properly on one, but leave that aside. (Or maybe the panel light isn't flat?) I don't recall the exposure time, but about 20 seconds.

Ha light flat, plain wall method:

Ha Light Flat panel method:

 

 

These flats clearly aren't right - both done in SharpCap with histo at 50% BTW. Oddly I get much better results if use High gain - here are the same flats done at high gain, also 50% histo:

Ha light flat, wall method, high gain (300) (no cooler this time as was only an expt, and only stacked x8):

Now this looks more like I would expect. But what the heck is going on??? How come I'm getting such different results for what should be the same flat?? And which  is correct - or maybe neither of the above?

The story gets even weirder when I looks at dark flats. The low (139) gain dark flat has a horrible grating pattern, where the high gain one looks much better. The high gain dark flat below is actually not from quite the same optical train, but it makes the point:

Dark flat, Low 139 gain:

Dark flat, High 300 gain:

So I could maybe just use the high gain versions and have done with it. But here are three critical questions:

Q1. Why is this happening?

Q2. What are the correct settings for the NB flats. Makes a bit of a mockery of calibration if I just use the ones that "look about right"

Q3. Even if I determine that the high gain flats work well for high gain NB imaging, where does this leave me with  LRGB? I only use low gain for LRGB and to use high gain flats would seem bonkers. Maybe it's no surprise that my LUM imaging has always been awful if the flats are dodgy. Any ideas??

There are further complications here with LRGB because with a flat panel the exposure is so short there is image banding due to the scan rates. I'll do some more experiments and post again but I think this is enough for now!

Thanks in advance - and fingers crossed!

 

[carastro]

Sorry to hear all this Tom, I have no experience of CMOS cameras and gain, so can't advise, but there is clearly something not right.  Hopefully someone will chip in with an answer very soon.

Carole 

[eshy76]

Hi Tom,

Could I ask if you are using an OAG to guide? The intruding element in the top centre of a couple of the flats look suspiciously like a misaligned OAG prism...I had this issue with my ASI1600 imaging train and fixed it using some shims to line up the prism with the long edge of the camera sensor to avoid darkening the corners/edges.

Sorry I have no insight on the gain questions...however I was suffering from inconsistent flats issues a while back - then I noticed my LED panel flickering more than usual when the flats were being taken. I ended up getting a posher Artesky one where you can control the brightness, which did away with having to dim my panel with sheets of paper, etc...that also fixed issues I was having with artifacts appearing in my flats...

...I hope that helped narrow things down a little?

[teoria_del_big_bang]

Frustrating to say the least.

Due to me not have enough experience I can sympathise but not help much at all but to add the banding on the RGB flats due to the scan is exactly what I had using the light panel without something else in front to diffuse or dim the light further.

Without some sheets of white printer paper the exposure times were about 0.04 seconds for G & B and a little longer for R.

I tried dimming with a dew band controller but these can make it worse as essentially they switch the light on and off very fast anyway to reduce the power.

I found the best way was to use a 12V battery, so I knew I had a steady 12V not  switched mode supply with 2 sheets of white printer paper to extend the exposure times to a minimum of about 0.2 secs.

I did intend to try a 9V battery and see if I could remove the paper but not got round to it yet.

Steve

[Tommohawk]

1 minute ago, eshy76 said:

..I hope that helped narrow things down a little?

Hi and thanks for the response. No OAG, so that at least is one less variable. Re the flat panel, yes I agree I'm just using my monitor which may have flaw/extraneous light etc....

BUT note that I get the same apparent overcorrection in the flat done with the flat wall method if the gain is set at 139. So it seems to be more a problem of the gain, rather than the method used to produce the flat.

It may be that its more to do with the exposure time rather than the gain setting itself. Obviously when using higher gain the exposure time reduces, so it could be that. This camera is the ASI 1600mm cool, not pro,  and I know there were some issues with download times which were apparently remedied by the use of a buffer. But I'm not sure if that's relevant.

[vlaiv]

12 hours ago, Tommohawk said:

both done in SharpCap

What drivers and software are you using to capture these subs?

You should be using ASCOM drivers for DSO imaging. SharpCap offers native drivers for ZWO cameras - these enable very fast frame rates and are suitable for planetary imaging but might not be the best option for long exposure.

[vlaiv]

Yes, another thing - could you post above files as single fits files for examination (no stacking, no calibration, just single unaltered sub from camera in fits format)?

That might give us clue of what is happening.

[eshy76]

1 minute ago, Tommohawk said:

Hi and thanks for the response. No OAG, so that at least is one less variable. Re the flat panel, yes I agree I'm just using my monitor which may have flaw/extraneous light etc....

BUT note that I get the same apparent overcorrection in the flat done with the flat wall method if the gain is set at 139. So it seems to be more a problem of the gain, rather than the method used to produce the flat.

It may be that its more to do with the exposure time rather than the gain setting itself. Obviously when using higher gain the exposure time reduces, so it could be that. This camera is the ASI 1600mm cool, not pro,  and I know there were some issues with download times which were apparently remedied by the use of a buffer. But I'm not sure if that's relevant.

Yes that's fair regarding the wall flat point, so definitely investigate the gain/download angle more before plumping for an expensive flat panel!

On the pro, there is a usb speed setting in the ascom driver, if there is something like that for the cool, maybe you can try lowering that setting?

[Tommohawk]

13 minutes ago, teoria_del_big_bang said:

Frustrating to say the least.

Due to me not have enough experience I can sympathise but not help much at all but to add the banding on the RGB flats due to the scan is exactly what I had using the light panel without something else in front to diffuse or dim the light further.

Without some sheets of white printer paper the exposure times were about 0.04 seconds for G & B and a little longer for R.

I tried dimming with a dew band controller but these can make it worse as essentially they switch the light on and off very fast anyway to reduce the power.

I found the best way was to use a 12V battery, so I knew I had a steady 12V not  switched mode supply with 2 sheets of white printer paper to extend the exposure times to a minimum of about 0.2 secs.

I did intend to try a 9V battery and see if I could remove the paper but not got round to it yet.

Steve

Thanks for that. TBH I'm going to avoid the flat panel completely for now as it introduces yet another variable, although as I mentioned above it cant be the main culprit because the plain wall gives the same problem when done at low gain.

You  may be onto something with  the paper diffuser / T shirt type idea because possibly extending the exposure may help solve the problem or at least give a few more options. That said, the high gain shorter exposure flats seems least problematic so Im not sure extending the exposure will help!

[Tommohawk]

5 minutes ago, vlaiv said:

What drivers and software are you using to capture these subs?

You should be using ASCOM drivers for DSO imaging. SharpCap offers native drivers for ZWO cameras - these enable very fast frame rates and are suitable for planetary imaging but might not be the best option for long exposure.

Hi Vlaiv - nice to hear from you again! You mentioned before the issue of the driver and maybe there's is something in that. I think when you raised this before I mentioned it to Robin of SharpCap and he felt the driver shouldn't be an issue. I'm not sure how to alter the driver option in Sharpcap but its got to be worth a go. 

6 minutes ago, vlaiv said:

Yes, another thing - could you post above files as single fits files for examination (no stacking, no calibration, just single unaltered sub from camera in fits format)?

That might give us clue of what is happening.

The funny thing is I can see nothing wrong with any of the lights, of any duration - its just the flats, (both dark and light). I wondered at one point if Sharpcap uses a different and potentially problematic method for creating flats. The obvious test here is to capture the flats as though they were lights, rather than using the flat capture routine. I'll try this when I have a moment.

I could post all the above as FITS but they are pretty massive files and TBH on visual inspection the reduced size JPEGS above appear identical to the FITS. I'll attach a light - but TBH what I'm now wondering is what the individual flats look like before averaging by Sharpcap. I don't have individual flats subs cos Sharpcap just  stacks them and doesn't keep separate subs. I'll redo the flats as individual subs and repost.

Single unfiddled HA light, 300 sec, gain 300:

Capture_00002 20_39_26_Ha.fits 

[spillage]

I have something similar with mine but from memory it appeared on all subs and was moisture on the sensor.

Have you tried giving APT a go for collecting flats? I think I used 15-20k.

I have never used sharpcap and I just add the flats into dss. After this is done once dss creates a master flat that can be added in the future saving some time.

[carastro]

Tom it might be worth having a chat with Roger (ApophisOAS - have messaged him with a link to this thread).  He has a ZWO CMOS camera (He's also one of the Cairds people, though not sure if you have met him yet).  He uses APT for capture and after some teething problems, he seems to have flats sussed.

Also I think Rodd has the same camera as you. 

Carole 

Edited October 28, 2019 by carastro

[vlaiv]

29 minutes ago, Tommohawk said:

I could post all the above as FITS but they are pretty massive files and TBH on visual inspection the reduced size JPEGS above appear identical to the FITS. I'll attach a light - but TBH what I'm now wondering is what the individual flats look like before averaging by Sharpcap. I don't have individual flats subs cos Sharpcap just  stacks them and doesn't keep separate subs. I'll redo the flats as individual subs and repost.

Ok, here is couple of things you should try out.

First, yes go for ASCOM drivers. Those can also be selected in SharpCap as far as I remember. In my (outdated) version of SharpCap, these are available in menu:

SharpCap then issues a warning that you are using ASCOM or WDM driver for camera that SharpCap can control directly - but just click that you are ok with that.

Alternative would be to use different capture software. Nina seems to be popular free imaging software that you can try out:

Second would be examining your subs. I wanted you to post your subs for examination - but that is just a few measurements and overview of fits header information that can help. You don't need to post complete files you can just post following:

1. Statistics of each sub - one flat, other flat, related flat darks, ...

2. Histogram of each sub

3. Fits header data

For sub that you uploaded, here is that information:

From this I can tell you straight away that you have offset issue. You should not have value of 16 in your sub even for bias subs, let alone subs that contain signal!

Here is histogram for example:

This one looks rather good, but I can bet that darks and flat darks will be clipping to the left due to offset issue.

Here is fits header:

So I see that you have your gain set at 300 that temp was -15, that you used 5 minute exposure and so on ... When comparing this information between subs one can see if there is mismatch or whatever.

Most important thing - don't do "auto" things. When you let software do something automatically for you - you loose a bit of control. In my view it is best to know exactly what software is doing if you are going to do that. Collect your darks, your flats and your flat darks like you do lights and then you can make your own masters the way you want. It also gives you chance to examine individual subs and see if there is something wrong with them.

Only when you know that your settings work "manually" and you know what software is doing automatically (by adjusting parameters / reading of what it does in certain procedure) - then use automatic approach to save time.

[ApophisAstros]

Just one question , i see you mention Gain, and wondered what the Unity Gain was for you camera and whether you have used it , obviously needs to be set same for whole session Dark,Flats,Subs.

I mostly keep to to Unity Gain and don't seem to have problems others have , also heard that monitor screens are not best as they flick on and off.

I use APT flats tool and may help if you wanted something different.

Have you ever had acceptable flats

HTH

Roger

[Tommohawk]

On 28/10/2019 at 10:56, spillage said:

I have something similar with mine but from memory it appeared on all subs and was moisture on the sensor.

Have you tried giving APT a go for collecting flats? I think I used 15-20k.

I have never used sharpcap and I just add the flats into dss. After this is done once dss creates a master flat that can be added in the future saving some time.

Hi thanks - I haven't tried APT. It may be that it would solve a problem but I have so many bits of software I'm desperately trying to not learn yet another one! In general Ive found SharpCap works well and If I can figure out whats going wrong I'd rather do that. The subs are basically fine its just the flats that are odd

On 28/10/2019 at 11:06, carastro said:

Tom it might be worth having a chat with Roger (ApophisOAS - have messaged him with a link to this thread).  He has a ZWO CMOS camera (He's also one of the Cairds people, though not sure if you have met him yet).  He uses APT for capture and after some teething problems, he seems to have flats sussed.

Also I think Rodd has the same camera as you. 

Carole 

OK thanks Carole - as  it happens I think I've made a breakthrough!! see below.

On 28/10/2019 at 11:26, vlaiv said:

Ok, here is couple of things you should try out.

First, yes go for ASCOM drivers. Those can also be selected in SharpCap as far as I remember. In my (outdated) version of SharpCap, these are available in menu:

 

SharpCap then issues a warning that you are using ASCOM or WDM driver for camera that SharpCap can control directly - but just click that you are ok with that.

Alternative would be to use different capture software. Nina seems to be popular free imaging software that you can try out:

Second would be examining your subs. I wanted you to post your subs for examination - but that is just a few measurements and overview of fits header information that can help. You don't need to post complete files you can just post following:

1. Statistics of each sub - one flat, other flat, related flat darks, ...

2. Histogram of each sub

3. Fits header data

For sub that you uploaded, here is that information:

 

From this I can tell you straight away that you have offset issue. You should not have value of 16 in your sub even for bias subs, let alone subs that contain signal!

Here is histogram for example:

 

This one looks rather good, but I can bet that darks and flat darks will be clipping to the left due to offset issue.

Here is fits header:

 

So I see that you have your gain set at 300 that temp was -15, that you used 5 minute exposure and so on ... When comparing this information between subs one can see if there is mismatch or whatever.

Most important thing - don't do "auto" things. When you let software do something automatically for you - you loose a bit of control. In my view it is best to know exactly what software is doing if you are going to do that. Collect your darks, your flats and your flat darks like you do lights and then you can make your own masters the way you want. It also gives you chance to examine individual subs and see if there is something wrong with them.

Only when you know that your settings work "manually" and you know what software is doing automatically (by adjusting parameters / reading of what it does in certain procedure) - then use automatic approach to save time.

Vlaiv -thanks for all that! You may be right about the drivers, but because the lights essentially always look good I'm reluctant to change. As it is I have new filters, new scope, new reducer, and new APP software so the less things I change the better.! 

Re the offset - I thought the idea was that the offset should be present in the lights? Maybe I have this wrong.

Re auto things - yes I know what you mean, and generally I'm like minded. But … your knowledge is a good way ahead of mine and I really do have to depend a bit on some of the auto stuff and default settings etc! Anyhow, see my latest findings below!

On 28/10/2019 at 12:06, apophisOAS said:

Just one question , i see you mention Gain, and wondered what the Unity Gain was for you camera and whether you have used it , obviously needs to be set same for whole session Dark,Flats,Subs.

I mostly keep to to Unity Gain and don't seem to have problems others have , also heard that monitor screens are not best as they flick on and off.

I use APT flats tool and may help if you wanted something different.

Have you ever had acceptable flats

HTH

Roger

Thanks Roger - I use unity gain (139) for LRGB but always use high gain (300) for NB.

BTW the same gain for darks is a must, but I don't think its necessary to use the same gain for flats. You are only trying to replicate the vignetting/bunnies etc ie the optical issues, so (in theory!) the gain shouldn't matter. I was actually going for a lower gain than the lights just to minimise noise in the flats.

Anyhow, big fanfare... I think I've made some progress!! I think I'm right in saying that the flats are really nothing more than "blank" lights. Sharpcap has a routine for taking multiple flats and averaging them - it works quite well. BUT it seems the capture is very different to if I take it as a light. This surely cant be right.. can it?

The first FITS below is taken as a single flat using the Sharpcap flats routine. HA filter, gain 139, 3.7s exp, about 50% on the histo.

The second has identical settings, but was taken as an ordinary light. It obviously looks very different.  Any thoughts as to whats going on?? It almost looks like Sharpcap is applying a big stretch or something??

I think the averaging routine on the first one  some of the metadata, but you may see this differently? Theres no cooler and these are obviously single subs so very noisy but makes the point

2019-10-29-1840_9_.fits

Ha G139 3.7s light_00001 18_43_46_Ha.fits

[Tommohawk]

Whooaahh - hold the fanfare and bunting, recork the bubbly, cancel the marching bands.

So now I've figured out what was going on with the flat routine and why it appears different to capturing the flats as though they were lights..... I think.

In the Sharpcap flat capture routine, there is an option to "apply the flat to light taken" or similar. so what was happening is that I was taking the flats using the flats tool , and then that flat was being applied to the lights which were taken subsequently - so the vignetting appeared almost gone. which is of course what one would expect! 

I think that when Sharpcarp restarts, it no longer applies that flat - though I need to check this with Robin. So sometimes, if I took the flats as lights BEFORE using the flats tool, I got the raw uncalibrated results, where if I did it the other way around, ie did a flat capture using the tool, I got "calibrated lights" which look better.

So the upshot of all this is that the flats I've taken should be correct, although I need to find out if any lights I've taken already had the flat applied. Not sure how easy that will be to verify, although when I do lights for real, I would never run a flat routine first so providing Sharpcap starts fresh each time with no preloaded flat I should be OK. 

Of course, if my lights do all have a flat applied without me realising, this might explain why the lights don't appear to exhibit much vignetting. OK so I'm rambling now... time for a malted bedtime drink.

[vlaiv]

1 hour ago, Tommohawk said:

Re the offset - I thought the idea was that the offset should be present in the lights? Maybe I have this wrong.

Offset is applied to all subs, so bias has it, dark has it, flats and lights have it.

Point with your offset is not that it is missing, but rather - it is set too low and should be higher. It is set so low that even in 300s sub that has it "all" - you have minimal values. That should not happen.

I know that changing software is sometimes not easiest thing to do, but like I said - don't do stuff automatically (in this case you even were not aware it is being done by software). Best to just capture all the files in capture application and do your calibration elsewhere - in software that will do specifically that - and let you choose how to do it.

[Tommohawk]

I'm having a look at NINA

[david_taurus83]

How long are your flats?

[Tommohawk]

Those ones are 3.7 seconds. 

[Jokehoba]

I too have had issues with flats using this camera. I've found that shorter flat exposure gives more consistent results. I prefer to get under 1s. 

[Tommohawk]

15 hours ago, Jokehoba said:

I too have had issues with flats using this camera. I've found that shorter flat exposure gives more consistent results. I prefer to get under 1s. 

When I went through all this before I think I found something similar, but if too short then are are "banding" artifacts. To my mind flats should be simple and it shouldn't matter what gain/exp one uses so long as the histo is somewhere in the middle with no black or white clipping, and so long as the dark flats settings match the light flats. 

It's maddening that I've spent probably more time shooting flats than lights!

[ollypenrice]

I wouldn't touch a fancy flats-capturing software option with a ten foot pole because I always want to know what's going on. I always make master flats exactly as if they were lights except that I combine them using a simple average rather than a sigma routine. And I do this before applying the master flat to the stack of lights. You'll need a master 'dark for flats' as well, so first you need to find your flats exposure time and gain and then use them again to make darks for flats. I would be absolutely paranoid about light leakage as well. Light has an uncanny ability to get past things intended to keep it out.

Olly

Edited November 5, 2019 by ollypenrice
clarification

[andrew s]

On 30/10/2019 at 14:05, Tommohawk said:

When I went through all this before I think I found something similar, but if too short then are are "banding" artifacts. To my mind flats should be simple and it shouldn't matter what gain/exp one uses so long as the histo is somewhere in the middle with no black or white clipping, and so long as the dark flats settings match the light flats. 

It's maddening that I've spent probably more time shooting flats than lights!

With a CMOS camera I would advise using the same gain for flats and lights. Changing the gain may impact the individual A to D and amplifiers differently which would invalidate the flats. I also insert an exposure delay in taking Flats and Darks ( and Bias with CCDs) to ensure the camera has stabilised between readouts. 

It is best to be over cautious incase there are hidden effects. 

Regards Andrew 

[Tommohawk]

On 05/11/2019 at 15:16, ollypenrice said:

I wouldn't touch a fancy flats-capturing software option with a ten foot pole because I always want to know what's going on. I always make master flats exactly as if they were lights except that I combine them using a simple average rather than a sigma routine. And I do this before applying the master flat to the stack of lights. You'll need a master 'dark for flats' as well, so first you need to find your flats exposure time and gain and then use them again to make darks for flats. I would be absolutely paranoid about light leakage as well. Light has an uncanny ability to get past things intended to keep it out.

Olly

hi Olly and thanks for the response. 

Vlaiv makes the same point about not trusting automated stuff and I know what you mean, but I need to simplify wherever possible because otherwise I just forget the routines. Remember I do this infrequently whereas it can become routine for you more easily. TBH I think my routine here will be determined by how AstroPixel Processor - which  I'm becoming increasingly enamoured of - deals with masterflat creation and library filing. But to be fair to SharpCap, the flat routine does produce reliable flats, and the only thing that's caught me out is the default application of the flat to subsequent lights, so that when I did test flats using a sequence of "lights" with a flat target for comparison, the flat correction is no longer present. If that makes sense. 

Re dark flats, yes I've got that thanks, and same duration and gain. 

Now here's a funny thing. I did 3 lots of flats using gain at 100 139 and 300, each with the exposure adjusted for a 50% histo, and did matching flat darks for each. The light flats varied as per the images above.  I then did 3 integrations on a stack of Has, with each of the different flats and matching dark flats, and the resulting stacked images were identical, even with big stretches. It may be that the apparent difference in the light flats is adjusted by the associated dark flat. There's probably a clever way to test this but not enough hours in the day and all that!

Re light leakage when doing darks or dark flats, the cap is a seriously solid aluminium number, which I then cover in foil and a lightproof bag and shoot in a dark room with the lights out. Hopefully 100% light proof!

On 05/11/2019 at 15:35, andrew s said:

With a CMOS camera I would advise using the same gain for flats and lights. Changing the gain may impact the individual A to D and amplifiers differently which would invalidate the flats. I also insert an exposure delay in taking Flats and Darks ( and Bias with CCDs) to ensure the camera has stabilised between readouts. 

It is best to be over cautious incase there are hidden effects. 

Regards Andrew 

I certainly agree that keeping the same gain for light and flats is playing safe, though if the flats process is only correcting the optical issues, not dark current/shot noise etc it really shouldn't matter, so long as the dark flats match the light flats. But better to be over-cautious as you say, and as it turns out the "nicest" looking flats, for Ha at least, are those with  matching gain of 300 - plus of course its quicker to shoot them!

Its also likely that when doing LRGB flats I'll be using lower gain to match the lower gain of the lights so that should work out OK.

Re the delay - yes I've wondered about that, esp as mine is the unbuffered "Cool" variant of the ASI16600, not the "Pro" version. More importantly perhaps, maybe I should also leave a delay when doing lights?

One other point of interest. I've played around rotating my scope to see just how "flat" my flat panel and flat illuminated wall are and the answer is... not very! Rotation of the scope/filters/camera assembly show big variations in different orientations. The application of a T shirt reduces this markedly. I'm in the process of fabricating some frosted opalescent lens caps which ought to give a nice result - fingers crossed! I'll post again when I've got that done. 

Edited November 6, 2019 by Tommohawk