ASI1600mm Cool - Flats nightmare, Pt 3

[Tommohawk]

Hi again - hopefully someone will come to my rescue with this. I posted before re this, but started a new thread because the other one got a bit muddled.

Summary is I have multiple issues with flats. Over the last week I have been experimenting with different techniques for flat capture - T shirt vs flat panel etc - because I discovered that my flats just weren't very flat due to uneven illumination. I now have that sorted, and  maybe I'll post about that separately because the result works well.

However I still have issues with integration and odd artefacts in  the flats themselves.

For narrowband, the issues had to do with variation in the flats pattern according to which gain I used. I figured that although I used Gain 300 for my lights, I would get less noise with flats at Gain 139 - but these flats look distinctly odd. The solution is to simply use Gain 300, same as the lights. With the cooler on and a big stack of flats the results are fine - sorted.

For LRGB I use Unity Gain 139, and the lights look fine. (It may be that Gain 0 would be better but that's another issue and doesn't solve the problem described below)

BUT the flats have an odd grid / chequerboard pattern, which I have seen reported on another thread on CN (although that relates to problems with the lights) Of course I'm using a big stretch to check this, but I don't think this is "pixel-peeping" - the resulting integration also shows a similar artefact, although of course its reduced because of the image signal.

I've tried reloading the new ZWO driver, reloading and using the ASCOM driver, altering the offset from 25 to 50 (on the basis that the narrowband ones shot at Gain 300 use an offset of 50 and they look fine). I've tried subtracting the bias in the flat as offered on the SharpCap flat routine, I've tried capturing the flats as lights to avoid any issues in the SC flat capture routine, and I've also tried capturing in Firecapture in case SC was the issue. Also with and without cooler.

I wondered about altering the light intensity to either increase or decrease the exposure time. A longer exposure would seem more natural ie more like the lights. However, the NB exposures with  300 gain use similar exposure times as the LRGB at 139 gain, ie about 1 second, so it cant be purely to do  with the exposure time.

There are tiny variations but in all cases I get a pattern in the flat much like below. This example materflat is 32 stacked cooled shot with SharpCap using ASCOM driver. Note that in Sharpcap at least the offset option no longer exists when using the ASCOM driver - seemingly the latest ZWO SDK has the offset locked to 50. (But not sure how this prevents access using the ASCOM - maybe the ASCOM driver has the offset locked to 50?) Maybe ZWO have fixed the offset to 50 in the ASCOM driver??

Masterflat: MF-ISO_gain_139.0-exp_1.125s-32subs-ASI_Camera__1_-4656x3520-NR-noBl-avg.fits

Integration: St-avg-3960.0s-MWC_1_3.0_none-x_1.0_LZ3-NS-full-qua-add-sc_BWMV_nor-AAD-RE-noMBB.fits

 

Really grateful for any insights!

Edit: BTW I also tried leaving 10 sec gaps between flat downloads - flats still the same.

 

 

Edited November 18, 2019 by Tommohawk

[spillage]

what software are you using to get you flats and what ADU?. I find using APT and setting ADU from anything from 15 to 25k.

[Tommohawk]

Thanks for your input. I'm using sharp cap although as I mentioned I have the same problem in other software like firecapture.

To be honest I've never really understood how to use ADU settings. I set the gain as I need it, in this case unity ie 139, and set the histo for 50%, and it is what it is! 

The histo in Sharpcap does show ADU information I think... there is a cursor you can move across the graph. The bottom of the histo shows a "value" of 30000 the peak at 33000 and the other side of the curve is that 36000.

The true ADU is 2000 at the peak.

[Tommohawk]

Spending yet another futile evening trying to sort flats and a further thought/complication has emerged.

There seems to be a difference of opinion on whether to use Bias or Dark flats, (or maybe both.?)  I've read that bias is not the best way, as it can be unstable and dark flats is better.... but just to  be clear, are the dark flats applied to the lights, or the (light) flats, or both? 

[spillage]

In APT you set the ADU level needed and it runs through different exposure times. I use a light box and I need to add or remove sheets of paper to adjust the light strength to make sure all the filters are able to reach the ADU set.

If you use a filter wheel it runs this for every filter then throws out a the exposure times and you then just add the amount of frames to be taken. It normally takes a few minutes.

I have never used sharpcap but I would suggest you try different settings and see how they effect your finished image.

I do not take bias frames and used dark flats which I  use in DeepSkyStacker along with flats and darks.

[Tommohawk]

Hi Mark - OK that's all useful info. I need to understand better how to use ADU in SharpCap. 

That said, most folk seem to say that putting the histo somewhere in the middle ie 50% should do the trick, so long as B and W aren't clipping - with 50% histo its nowhere near clipping.

One tentative conclusion I have reached is that the flats are only to correct vignetting, bunnies etc and nothing else - any pattern noise or other photoelectric (PE) issues should be corrected by the darks. Therefore, the flats should not show any grid pattern type issues at all - apart from bunnies, they should have smooth grading in the grey pattern. However the flats I get at gain 139 have a distinct grid type appearance - wheras those at gain 300 don't.

BUT... It might be that flats will inevitably have some PE effects in the same way that lights do, and the purpose of the dark flat is to correct this - that would seem very logical; hence my query in the previous post.

So - what are the dark flats for? Correcting the flats or the lights ...or both???  

Maybe the darks correct PE effects in the lights, and hence are of the same duration/temp, and the dark flats correct PE effects in the flats, and hence are of the same duration/temp - that sounds convincing … slightly! 

[vlaiv]

16 hours ago, Tommohawk said:

Spending yet another futile evening trying to sort flats and a further thought/complication has emerged.

There seems to be a difference of opinion on whether to use Bias or Dark flats, (or maybe both.?)  I've read that bias is not the best way, as it can be unstable and dark flats is better.... but just to  be clear, are the dark flats applied to the lights, or the (light) flats, or both? 

Dark flats are used to create master flat.

You take set of flats - stack those (average stack method). You take set of dark flats - stack those (again average stack method).

Subtract the two and that will give you master flat (flat stack - flat dark stack).

Again, what seems to be the issue with your flats? I've downloaded two attached files but can't really see what is wrong with them?

[Tommohawk]

Hi Vlaiv

Re the issue of flats, dark flats, bias etc I think I've slowly got my head round this - in my defence there is a some muddly and downright duff information out there!

Please correct me if I'm wrong, but it appears that darks correct for the thermal or photoelectric type effects in the lights. The same is true for dark flats correcting similar issues in the (light) flats. There is no need to subtract bias, because both the darks and the dark flats have the bias and therefore this is subtracted from the respective lights and (light) flats. Correct?

Now one thing that would be problematic is if the bias were variable between the lights and the darks (or light flats and dark flats). BTW it seems that ZWO have locked the offset in the latest SDK for some ASI cameras but probably not all, so when I run Sharpcap in ASCOM mode there is no Offset option. If I use Sharpcap drivers I can alter the offset as I wish. See thread here, But I think I also read somewhere that this has changed since!  

Leaving that aside, what I would like to see is that the calibrated flat ie the flat with the dark flats applied, shows no pattern type defect at all - it should only show vignetting/bunny corrections. But I don't think APP saves the calibrated flat. It saves the masterflat and the masterdark flat, but not the calibrated flat. There is probably a way of forcing this - maybe if I loaded the masterflat as a light, and the masterdark as a dark, and integrated, I could create that result. Does that make sense? Or maybe I'm going a bit bonkers. 

Re those uploaded files and patterns, I'm beginning to question now if what I'm seeing is some kind of rendering artefact. I uploaded a TIFF version just so we could both see the same thing on the screen - and I cant see the problem! Similarly I see no defect with a JPeg version. So I viewed the FITS file in FITs viewer (rather than in AStro Pixel processor) and I get different artefacts depending on the zoom. I've posted below a screen shot of the appearance in APP. This appearance is with a massive stretch, but it does seem consistent across multiple integrations, and is not to do with any screen defect because its independent of screen position, nor dependent on zoom ie it resizes exactly as the whole image resizes. 

Hopefully can you see the artefacts I'm describing?? What the heck is going on?? Do you see the pattern when you open in in your FITS viewer or similar?

APP:

 

FITS viewer "fit in preview", more like a grid pattern:

FITS viewer 16.67%, more like the banding effect as in APP:

 

 

 

Edited November 21, 2019 by Tommohawk

[Merlin66]

7 minutes ago, Tommohawk said:

There is no need to subtract bias, because both the darks and the dark flats have the bias and therefore this is subtracted from the respective lights and (light) flats. Correct?

Yes.

I use AstroArt and the only time bias is mentioned is when you need to compensate your darks to another temperature....

 

[vlaiv]

1 hour ago, Tommohawk said:

Please correct me if I'm wrong, but it appears that darks correct for the thermal or photoelectric type effects in the lights. The same is true for dark flats correcting similar issues in the (light) flats. There is no need to subtract bias, because both the darks and the dark flats have the bias and therefore this is subtracted from the respective lights and (light) flats. Correct?

Correct.

1 hour ago, Tommohawk said:

Now one thing that would be problematic is if the bias were variable between the lights and the darks (or light flats and dark flats). BTW it seems that ZWO have locked the offset in the latest SDK for some ASI cameras but probably not all, so when I run Sharpcap in ASCOM mode there is no Offset option. If I use Sharpcap drivers I can alter the offset as I wish. See thread here, But I think I also read somewhere that this has changed since!

As far as I know - there was a version of drivers with locked offset at 50, but now there seems to be version that has advanced settings where you can again adjust offset.

This happens with PRO version of cameras. My ASI1600 has offset setting in drivers enabled all the time. I might be wrong at above though.

1 hour ago, Tommohawk said:

Leaving that aside, what I would like to see is that the calibrated flat ie the flat with the dark flats applied, shows no pattern type defect at all - it should only show vignetting/bunny corrections. But I don't think APP saves the calibrated flat. It saves the masterflat and the masterdark flat, but not the calibrated flat. There is probably a way of forcing this - maybe if I loaded the masterflat as a light, and the masterdark as a dark, and integrated, I could create that result. Does that make sense? Or maybe I'm going a bit bonkers. 

Masterflat should be one with master flat dark applied - not entirely sure, but that is the usual name for final flat used for calibration - masterflat.

You can check that by loading single flat and measuring mean ADU value on it and doing the same with master flat - if they are the same, then flat dark was not removed, if they differ (mean value of master is a bit lower) then master flat dark was removed.

1 hour ago, Tommohawk said:

Re those uploaded files and patterns, I'm beginning to question now if what I'm seeing is some kind of rendering artefact. I uploaded a TIFF version just so we could both see the same thing on the screen - and I cant see the problem! Similarly I see no defect with a JPeg version. So I viewed the FITS file in FITs viewer (rather than in AStro Pixel processor) and I get different artefacts depending on the zoom. I've posted below a screen shot of the appearance in APP. This appearance is with a massive stretch, but it does seem consistent across multiple integrations, and is not to do with any screen defect because its independent of screen position, nor dependent on zoom ie it resizes exactly as the whole image resizes. 

Hopefully can you see the artefacts I'm describing?? What the heck is going on?? Do you see the pattern when you open in in your FITS viewer or similar?

If you are talking about files that you have uploaded at beginning of this thread - no they do not suffer from such artifacts, and any artifact that you are seeing is due to software used to view them.

Some software will use bilinear interpolation when rescaling images for view at smaller size (zoomed out) - try looking at 100%, 1:1 pixel ratio (one screen pixel - one image pixel) - there should not be artifacts present. Bilinear interpolation can create such bands - those are in fact not signal bands, but bands of different SNR (slightly lower noise in some regions due to bilinear interpolation) - and will only show if you stretch so that noise is clearly visible.

Here is what your light looks like to me:

That is left edge of image stretched to insanity

Btw, I can also recreate effect which you are seeing by resampling to quarter size (about fit to screen) with bilinear resampling:

Here is what stretched histogram version looks like (gimp level / curves):

Again no sign of artifacts that you mention. It's full of stars

 

[Tommohawk]

22 hours ago, vlaiv said:

Masterflat should be one with master flat dark applied - not entirely sure, but that is the usual name for final flat used for calibration - masterflat.

You can check that by loading single flat and measuring mean ADU value on it and doing the same with master flat - if they are the same, then flat dark was not removed, if they differ (mean value of master is a bit lower) then master flat dark was removed.

Sounds like a plan - but not sure how to do this. Can I check mean ADU in FITS liberator?

BTW the masterflat still looks pretty lumpy when stretched very hard - difficult to believe its had the MDF subtracted. Also APP creates both a MasterFlat from a stack of flats, and MasterDarkFlat from a stack of dark flats, each with its own integration and outlier rejection options - I suppose this has to be done to get the calibrated MasterFlat. It would then be pretty confusing if the resulting calibrated masterflat were also called just "masterflat"  (MF-MDF=MF !!)

Hang on though!!! - I just did a search and oddly came across exactly this issue in the APP forum. Mabula - APP author said:

"APP will only create a MasterFlat correctly, when the flats can be calibrated before integrating to a MasterFlat., so if the flats can't be calibrated, APP will issue a warning !

So the masterbias and/or masterdarkflat is applied on the flats and these calibrated flats are then integrated, to create a masterflat.

So once the masterflat is created, the masterdarkflat has become redundant for calibration of the light frames. Any MasterDarkFlat loaded will have zero influence on the Light frame calibration. The MasterDarkFlatss will only be used for flat frame calibration before integration of the MasterFlat.

Please let me know if this is clear 

Kind regards.

Mabula"

This fits with the APP file labelling system, in that the calibrated light is referred to as "light n1 MD-1 MF-1..." etc. with no mention of MDF. It implies that if working on a library basis, or if reprocessing in APP, only the "Masterflat" needs to be stored/applied.

Personally I think this is very misleading - to  my mind this should be called "Calibrated MF" or similar. After all, a stack of bias is a masterbias, and a stack of darks is a masterdark.

Re the pattern artefacts - well it looks like most of the problems have to do with how the FITS files render in APP and FITsliberator. When I transfer the resulting TIFF files into PS it all looks fine. But its going to make it interesting doing preliminary post-processing in APP if there's load of artefacts! I've just done another integration in APP and that looks fine - so maybe theres just something a bit weird with the data.

I'll try doing a full integration again and see what happens!

Many thanks for all the input - I've learned a lot. 

Edited November 21, 2019 by Tommohawk

[vlaiv]

21 minutes ago, Tommohawk said:

Sounds like a plan - but not sure how to do this. Can I check mean ADU in FITS liberator?

Yes - if it does what it "says" it does - checks mean ADU level in linear stage (prior to any stretch).

22 minutes ago, Tommohawk said:

BTW the masterflat still looks pretty lumpy when stretched very hard

One issue that I have with attached master flat is that it is still 16bit format? That is "no-no" in my book - it should be 32bit float. 16bits is simply not enough to preserve all precision needed.

Don't know why that is, but all calibration masters (and intermediary masters like - master flat dark) should be in 32bit precision. That way you won't add any noise due to bit precision (and that will happen pretty quickly with 16 bit format if you go above 16 calibration subs - and most people use and should use more than that).

As for flat master itself - don't know, it looks rather "normal". It might be a bit noisier than I'm used to, but looks fine otherwise?

Here is comparison between your master flat and my master flat (my is 32bit 256 subs stacked with 256 flat darks - at unity gain):

Yours (1:1 zoom and crop somewhere near center):

Mine (same zoom and roughly somewhere near center):

Mine looks less noisy / smoother, but it shows certain pattern that is present on my ASI1600 - it is much better seen when one zooms in more:

It is sort of checker board pattern of pixel sensitivities - nothing to worry about as differences between pixels are at max about 2%, and it calibrates lights fine. This is manufacturing "artifact" - but you need a lot of exposures to "expose" that variation - otherwise it stays within noise.

By the way - if you shoot your flats at high gain - use a bit more than 32 subs for them - it will reduce noise, or preferably shoot at normal / unity gain.

[Tommohawk]

On 21/11/2019 at 22:09, vlaiv said:

One issue that I have with attached master flat is that it is still 16bit format? That is "no-no" in my book - it should be 32bit float. 16bits is simply not enough to preserve all precision needed.

Ahaaaa - good point. I've only just noticed that in APP creating 32 bit flats isn't enabled by default - that seems a bit odd. I know Mabula is working on the idea of saving processing profiles, so hopefully in due course I can have this enabled by default. In checking this I've also noticed that SharpCap only creates masterflats as 16 bit,  so I'll definitely be avoiding that. 

Thanks again and hopefully someone else might find this thread helpful.