Flats with autofocuser and multiple filters

[Barns]

I've recently bought a ZWO EAF autofocus unit. Not having had clear skies since (of course) I find myself pondering how this will affect my workflow with regards to flats.

Currently, I have a stock set of flats taken through each ZWO filter at a fixed focal point (that's the same every night as it's manually set by hand). So I don't re-take flats every night; I reuse this set I took once, and I guess I'll re-take this stock set if I start seeing unwanted dust rings appearing. I actually took the flats all at the same focus point, despite discovering the OIII filter is not quite parfocal with the Ha and SII - but it doesn't seem to unduly affect the use of the flats.

However with the new autofocus unit, the whole idea is that it'll autofocus by itself (using SGP) right throughout an imaging run, refocusing for each filter, and also with temperature changes. How do you account for all these different focus positions when it comes to flats? Do you attempt to somehow cover every eventuality with numerous different sets of flats for each filter at each temperature (and hence focus position) ? Or do you just use one set of flats at a "standard" temperature for each filter, and assume that the tweaks for temperature are minor enough that they won't affect the flat utility as focus changes?

In general, do people take new flats every night or do you use a standard set you reuse for a while? I'm hoping I can keep using a standard sets as I always find generating flats a right pain, mostly because I haven't cobbled together a decent flat box yet! (I do have an EL panel that works ok but I haven't got suitable housing, t-shirts, bits of paper or whatever to make it easily usable with no faff - more DIY required).

[Jkulin]

Unless you have an observatory and even then you should take flats after each session as dust bunnies can happen during any period of imaging.

I do them after every night of imaging for the filters that I have used on that session

[tooth_dr]

I don’t agree that you need to retake them every session if permanently setup ie camera isn’t moved.  Flats are a wee bit of a faff with some cameras, like those with mechanical shutters for example, or if you have a large aperture scope.

i have taken flats for each filter in the past but have found flats from just one filter works for them all.

Temperature - I always match that with the lights.

Minor focus discrepancies should make zero difference. 

[daz]

I'm with John on this. Flats for each filter used after each session and I'm in an observatory. I generally just use the last focus position of the last filter to take all the flats, and this seems to work OK.

I guess you could take an average for each of the images for each of the filters and take the flats at that, but that is indeed faff - capital F!

And no, the slight tweaks in focus position won't matter

[Adreneline]

I am also with John.

I find there are significant differences between the flats I take for Ha, OIII and SII and not just dust bunnies - which seem to move around or even disappear completely sometimes - but never for good.

Of my three filters the OIII flat is significantly different to the Ha and SII and I have no idea why.

I also take Dark-Flats at the same time - lens cap on and I use a special filter made of 'black-body' card to ensure no photons (within the visible spectrum) get to the sensor - very hi-tech!

 

 

For all I use the Ha focus position.

HTH

Adrian

Edited October 9, 2019 by Adreneline
Clarification

[david_taurus83]

Not sure how SGP works but Kstars records the last position of the focuser on each filter and restores it to each position should you choose to do flats after the lights.

If your filters and sensor cover are completely free from dust and the flats only show vignetting then you could probably get away without doing flats each session. Even if you have a few dust bunnies I find they tend to stay in the same place. But the risk is you could shoot 3/4 hours of great data and 1 spec of dust could have moved and you could end up with a wasted session. There have also been reports of the ZWO wheels not returning to the exact previous position of each filter, though I've not experienced this myself.

 

Unless the weather is playing up mid session, or I'm not too happy with the quality of the subs or im just too tired, I shoot new flats each session. The fact is we spend so much on all the kit to get so little time to use it because of the weather, I personally think taking flats is a price worth paying each night.

[Barns]

Seems like the consensus is that I basically need to get my act together and cobble my flat box into decent shape! I do agree the flats between the filters are definitely different - each of the Ha, OIII and SII flats look quite different when stretched, with weird banding and all sorts. Seems to come out in the wash during processing though. I'm happy to hear that experience shows I needn't worry about the minor refocusing due to temperature changes thouh - that's a relief.

I find it tricky taking flats at the end of a session as often I have about 5mins between waking up in the morning and then having to get the kids up, to school etc, in which time I need to hastily disassemble the scope and bring it indoors. Not enough time for flats at that point so they'd get done later in the day or on another day entirely. And with clear skies as preciously rare as they are, I'm not about to "waste" time at the start of a session doing flats (especially when they take quite a chunk of time to get a decent set through narrowband with their longer exposure times).

[DaveS]

Need to do some flats as I've just swapped filters, cleaning them in the process. About the only time I redo flats as I find any dust bunnies pretty stable.

TBH, unless I'm focusing in twilight I have to stretch the hell out of my stacked images to see any bunnies at all, and a gradient removal will take care of the slight vignetting in my system. Whether it will be clear enough to do twilight flats tonight is a moot point.

[Adreneline]

1 hour ago, Barns said:

Not enough time for flats at that point so they'd get done later in the day or on another day entirely.

I am sure that is the situation for many who have to strip down each session and bring equipment back inside. I use an iPad running a lightbox app propped up against the wall pinned by the end of scope/lens with a with tee-shirt stretched over the end.

I try to take my flats and dark-flats at the same sensor temperature (-20) as the lights but I'm not convinced that is really necessary as the exposure times are typically one to two seconds for my setup; can't see that amp noise/glow is going to have any effect at all but I stand to be corrected.

Good luck!

 

[spillage]

14 hours ago, tooth_dr said:

I don’t agree that you need to retake them every session if permanently setup ie camera isn’t moved.  Flats are a wee bit of a faff with some cameras, like those with mechanical shutters for example, or if you have a large aperture scope.

i have taken flats for each filter in the past but have found flats from just one filter works for them all.

Temperature - I always match that with the lights.

Minor focus discrepancies should make zero difference. 

+1 for this. I do not think flats are dependent on focus and I only bother using calibration frames from the L filter. 

[ollypenrice]

Interesting range of opinions! My contribution (potentially heretical) for what it's worth:

1) There is no way in the world that you will notice differences in flattening between lights with small adjustments in focus. We are not doing photometry here, just getting rid of dust bunnies and vignetting. You will find it near impossible to get flats so accurate that the greatest deviation from accuracy comes from small focus changes. Just forget it.

2) My flats last anywhere between a week and six months in my observatories. Every night??? You must be joking!

3) Dust bunnies (visible ones) don't usually come from dust on the filters in my case. They don't, except very exceptionally (once every three years?) vary from filter to filter. So I use a luminance flat for everything, but see my next point.

4) Any slight flattening irregularities in an RGB layer will be re-illuminated by the luminance layer, by definition, so if that is well flattened you have the definitive flatness you need. Even NB imagers sometimes use their Ha layer as luminance so if the Ha is well flattened so are the other two layers.

I take my imaging seriously but that includes weeding out unproductive activities. Flats are very important indeed but what I describe above covers my workflow most of the time. On the rare occasions it doesn't work I shoot the extra flats necessary - but that is a very rare event. I take a practical rather than a religious approach to flats based on hundreds of hours of exposure every year.

Olly

[Davey-T]

7 minutes ago, ollypenrice said:

I take my imaging seriously but that includes weeding out unproductive activities. Flats are very important indeed but what I describe above covers my workflow most of the time. On the rare occasions it doesn't work I shoot the extra flats necessary - but that is a very rare event. I take a practical rather than a religious approach to flats based on hundreds of hours of exposure every year.

On the subject of flats Olly how many do you take ? I've only ever bothered to take 16 as there are plenty of other things to stuff my images, @vlaivon the other hand advocates lots of them, also seems to be different opinions about it for CMOS but no idea as I don't have one.

Dave

[ollypenrice]

29 minutes ago, Davey-T said:

On the subject of flats Olly how many do you take ? I've only ever bothered to take 16 as there are plenty of other things to stuff my images, @vlaivon the other hand advocates lots of them, also seems to be different opinions about it for CMOS but no idea as I don't have one.

Dave

I generally do about 30. Once you have the exposure time right they are not much bother to shoot. In all honesty I've never done a comparison but Per, who was into the technical side of imaging in a way that I'm not, did just 15 of everything, darks, bias flats etc.

Olly

[vlaiv]

36 minutes ago, Davey-T said:

On the subject of flats Olly how many do you take ? I've only ever bothered to take 16 as there are plenty of other things to stuff my images, @vlaivon the other hand advocates lots of them, also seems to be different opinions about it for CMOS but no idea as I don't have one.

Dave

Ok, I'll admit - I might be overboard with the number of flats I take

Here is the reasoning on how much flat subs one needs - it involves a bit of math, and results do differ based on type of sensor one is using (and settings):

We will use "extreme" case to do calculation - that way we can be sure that it will work in majority of cases. We will "correct" extreme vignetting of about 50% - so half of light is reaching sensor.

For all intents and purposes, we want difference between real and corrected value to be less than about 0.4% - that means less than 1 out of 256 intensity levels that modern displays offer.

Imagine we have 0.5 intensity of vignetted pixel and we want to make it 1 again (attenuation of 50%). We need to divide it with ~0.5 so that it has less error than said 0.4%. In math terms it is like this:

0.5 / something = 1.004, or something = 0.5 / 1.004 = ~ 0.498008

Max error in our case is 0.5 - 0.498008 = 0.001992 so needed SNR should be around 250. That will correct about 66% of pixels to required value. If we want really good correction rate - we need to do something like 3 sigma (three times that SNR) - that will give 99.7% pixels that will be corrected better than 0.4% error.

In another words - our flat needs about 750 SNR in 50% signal area. Let's see how much SNR there is per flat sub in 50% vignetted area.

CCD case:

let's use common e/ADU value of 0.5 for CCD (you can do math for your sensor), and histogram peak at 80%. CCDs operate on 16 bit, so max ADU will be around 32000ADU (half of 16 bit). Converted into electrons - that will be around 16000e. 80% of that will be 12800 and 50% illumination will be half of that - 6400e.

This means that 50% vignetted signal will contain roughly around 6400e. SNR in that region is square root of this value and that is 80.

Single flat carries SNR of 80. In order to get SNR of ~750 we need to improve SNR by x9.375, so we need to stack (square of that value) about 88 subs.

let's use common e/ADU value of 0.5 for CCD (you can do math for your sensor), and histogram peak at 80%. CCDs operate on 16 bit, so max ADU will be around 64000ADU. Converted into electrons - that will be around 32000e. 80% of that will be 25600 and 50% illumination will be half of that - 12800e.

This means that 50% vignetted signal will contain roughly around 12800e. SNR in that region is square root of this value and that is ~113.137.

Single flat carries SNR of 113.137. In order to get SNR of ~750 we need to improve SNR by ~x6.63, so we need to stack (square of that value) about 44 subs.

CMOS case (particularly the way I use it, so ASI1600 12bit at unity gain):

Max signal in this case is about 4000e (4096 to be precise but let's round it up). 80% histogram peak, means that max illuminated part of flat will be at 3200e, while 50% vignetted part will have 1600e.

Single flat in this case has SNR of 40. To reach SNR of 750 we need to "boost" it by x18.75 - by stacking ~350 subs.

From above calculations we can see two important things - CMOS because lower bit count if used at unity gain require higher number of flats to get same SNR. We can also see that if we want to have minimal error in flat correction (below one in 256 so practically indistinguishable on 8bit displays) while having severe vignetting and we want to achieve that level of performance in 99.7% pixels - we need to take a lot of flats.

In reality, light attenuation will not be severe as 50% but closer to 10-20%, we can also make other criteria less strict - like error in single pixel being 1% (most will have difficulty telling if pixel should have intensity 230 instead of 231 or 232), and we can settle for less pixels having "perfect" correction - this will bring needed number of flats down of course - one of the reasons why people use 16 flats without much of impact.

 

Edited October 8, 2019 by vlaiv
I messed up calculation of CCD par by factor of 2

[Davey-T]

3 minutes ago, ollypenrice said:

did just 15 of everything, darks, bias flats etc.

Interesting, I read somewhere that you should do 16 as it was something to do with square roots or some such so 16 , 32 etc, so many "expert" opinions it's no wonder folk get confused 

Dave

[carastro]

Quote

15 of everything, darks, bias flats etc.

Ditto.

Carole 

[vlaiv]

2 minutes ago, Davey-T said:

Interesting, I read somewhere that you should do 16 as it was something to do with square roots or some such so 16 , 32 etc, so many "expert" opinions it's no wonder folk get confused 

Dave

I advocate 16 for practical reasons - finite precision.

Having 16 subs and doing average of that means you divide everything with 16 - which in binary representation means "shifting" by 4 binary places (same as dividing with 100 for example means moving decimal point 2 places) - it does not introduce infinite number of digits. Imagine you have 3 subs and divide with 3. In some cases you will get infinite number of digits as result - 1/3 = 0.3333333...... Once you write that down with finite number of places, you introduce error. If you write it down like 0.333, you are in fact introducing 0.0003333333.... error.

Maybe I'm just overly caucuses not to introduce additional error even if it is way to small to make impact

 

 

[Jkulin]

I take 25 of each filter for flats at 30K ADU

My reasoning is that although I have a permanent pier, putting a cover on each night and with an open truss design, does encourage more dust as they are more exposed.

If I was shooting most of the time with my Esprits then things may be different.

But to be truthful I now shoot my flats before the session starts in twilight, so it really is no big deal and takes up very little time.

Since obtaining some really nice flats panels, I have found a massive improvement in my images: -

A4 Flat Panel, perfect for smaller OTA's

A3 Flat Panel, perfect for upto 12"OTA's

They run on a 12v feed from my Pegasus UPB.

I ordered some coloured perspex cut to my sizes and then sandwich lighting gels between to get the perfect ADU vs. speed, I acknowledge that with the G2-8300 that to avoid the shutter curtain showing you need exposures over 2 secs.

I have different panels for NB and RGB.

[ollypenrice]

12 hours ago, Davey-T said:

Interesting, I read somewhere that you should do 16 as it was something to do with square roots or some such so 16 , 32 etc, so many "expert" opinions it's no wonder folk get confused 

Dave

Another of these theories advocated an odd number when using Median to combine them - but I use average anyway!

The idea that anybody could possibly tell the difference in the final image is - shall we say - fanciful. 

Olly

[carastro]

 If the rig is still set up from a previous imaging session and the camera not moved, I re-use my flats having done a separate set for each filter. 

Also I agree with Olly that the minute difference in focus between the filters is not worth worrying about, I never change focus between filter flats.

Carole 

Edited October 9, 2019 by carastro

[ApophisAstros]

27 minutes ago, carastro said:

 If the rig is still set up from a previous imaging session and the camera not moved, I re-use my flats having done a separate set for each filter. 

Also I agree with Olly that the minute difference in focus between the filters is not worth worrying about, I never change focus between filter flats.

Carole 

If you have a flatbox on the scope then it's impractical to focus for each filter for flats unless you have an auto focuser already with each offset,

Roger

[ollypenrice]

The real issue is that you need good flats and if there is anybody out there who always gets good flats, which just work and never over-correct, I'm all ears. I follow the same routine each time and mostly it works but always, certainly not. I wish I knew why this is, but I don't.

Olly

[alan potts]

I wish my mother was still alive, dust never had time to settle with her around, always cleaning, bless her.

Alan

[vlaiv]

34 minutes ago, ollypenrice said:

The real issue is that you need good flats and if there is anybody out there who always gets good flats, which just work and never over-correct, I'm all ears. I follow the same routine each time and mostly it works but always, certainly not. I wish I knew why this is, but I don't.

Olly

Fancy having a go at cracking that mystery?

My bet would be on improper calibration, or slightly "bending the rules" of calibration, because "it works that way as well".

Let's see what could be a cause of over correcting, and to do that first - let's agree on what over correction is - I understand that term to be used when vignetting or dust shadow becomes brighter than it should be.

If we look at flat correction as - true value = recorded value / flat correction, and over correction means that true value is greater than it should be - we have two choices:

1. recorded value is larger than it should be

2. flat correction is lower than it should be

Point 2 is unlikely to happen (might happen if you use incorrect flat darks - one of longer exposure, or ones shot at higher temperature than flats), but there is rather simple explanation for point 1 - you need only light signal in your recorded value. Any other signal still there will make it larger than it should be.

Therefore, things like calibrating with bias only (not removing dark current) will produce over correction.

[ollypenrice]

13 hours ago, vlaiv said:

Fancy having a go at cracking that mystery?

My bet would be on improper calibration, or slightly "bending the rules" of calibration, because "it works that way as well".

Let's see what could be a cause of over correcting, and to do that first - let's agree on what over correction is - I understand that term to be used when vignetting or dust shadow becomes brighter than it should be.

If we look at flat correction as - true value = recorded value / flat correction, and over correction means that true value is greater than it should be - we have two choices:

1. recorded value is larger than it should be

2. flat correction is lower than it should be

Point 2 is unlikely to happen (might happen if you use incorrect flat darks - one of longer exposure, or ones shot at higher temperature than flats), but there is rather simple explanation for point 1 - you need only light signal in your recorded value. Any other signal still there will make it larger than it should be.

Therefore, things like calibrating with bias only (not removing dark current) will produce over correction.

I tried all these tests, Vlaiv, and they made no difference whatever. With one setup the flats always over corrected. I tried different exposures, different brightnesses of source, using bias as dark, using 'correct' flat darks... The over correction continued in precisely the same way. Comparing differently acquired flats as Ps layers with blend mode difference showed them to be nearly identical. I was capturing everything in Nebulosity. Eventually I shot some flats in Astro Art and they worked for many months. Then one morning they started over-correcting again! They just did. Nothing that I'm aware of had been changed.

More recently a set of flats from the Moravian did the same, captured in SGP. Because of the shutter, exposures are quite long so I calibrated with 'correct' flat darks, not bias. I shot another set using the same values and they worked. I can't explain it.

The only thing I conclude from this is that my Atiks, running in Atik's Artemis Capture, have never ever generated problem flats. Why do I like Artemis Capture???  Only cameras running in other software have given trouble - but this could be chance or human error. If it's error I don't know where it came from because I ended up using a written checklist and following it to the letter. I also followed it with guests watching over my shoulder.

Olly

Edited October 11, 2019 by ollypenrice
Clarification.