Flats

[OK Apricot]

Hi ladies and gents,

While not specifically reading our watching videos about flats, I've heard there may be an ideal exposure length, especially for cameras such as the ASI294MC-Pro. Is this true?

The ASIAIR Plus will calculate the appropriate exposure length for flats depending on the light source, for me some of them being 0.8s, others closer to 10s. Is this going to produce usable flats or should I pay more attention to the histogram? 

Since I started applying flats to my stacks, I feel like my background of space turns out a lighter shade of black. It is most likely my processing but it could be anything. Please could I have a little theory behind this? It looks to be clear for a few hours tonight so useful testing time. 

Edit - I use a white screen on an ipad and a white t-shirt. 

Cheers! 

Edited December 29, 2022 by OK Apricot

[Elp]

Your overstretched uncalibrated stack, and overstretched calibrated stack will help.

If you're planning on using Optolong filters brace yourself.

Also, prepare for a long and bumpy ride.

Edited December 29, 2022 by Elp

[vlaiv]

There is not much theory behind this.

You need to make sure two things are satisfied:

1. You get good SNR for your flats

2. Your flats are in linear region of sensor (no saturation / clipping or other non linearity of any kind).

If you make your exposure too short and you don't compensate with number of subs you stack for flats - you run a risk of adding too much noise from flats into your images.

On the other hand - if you expose for too long - you might over expose your flats. Even when you don't over expose your flats - you can create issues if you hit non linear region of sensor.

In general - you don't want to be near max value of what your sensor can record.

Histogram peak should stay at 2/3 to 3/4 and not more. This is generally just rule of the thumb.

Consider following - you are shooting Ha flats and you have gain pushed very far. Your effective full well capacity is say 1000e (although camera has something like 20K normally - due to high gain you reduced it significantly). At large values, Poisson distribution is very similar to Gaussian one, so we can use that to approximate things.

Say we place histogram at 900e (or 90% peak) - then our sigma will be 30e. x3 sigma will be 990e.

There is a chance that 0.13% of all pixels in the bright part of the flat - be saturated (or at least over 990e). That might not sound like much, but if you have 3000x2000 - that is 60 million pixels, and let's say that 1/9th is in central bright region - that is still around 9000 pixels per image that are saturated.

Each sub will have this and if you stack them - they will "skew" things in final stack as they represent incorrect / clipped values.

To minimize that - on higher gain settings (or when shooting with smaller full well capacity in general) - go with 2/3 or even 1/2 of histogram as peak. Use 3/4 rule only when using lower gain setting or larger full well.

If you want to test your flats without applying them to actual image - use following thing:

- shoot one set of flats with very short exposure

- shoot one set of flats with longer exposure - following above rule.

Calibrate one with another (including flat darks) and you should get perfectly flat image - uniform gray with noise.

[ollypenrice]

 

This is a great test. I use it to demonstrate the purpose of flats when helping beginners.

12 minutes ago, vlaiv said:

Calibrate one with another (including flat darks) and you should get perfectly flat image - uniform gray with noise.

[Elp]

From what I've read about the 294 due to how the HCG is implemented causes variability as to what gain setting you can use. If you use pc software you can potentially alter a few more settings related to the gain/HCG factor multiplier, with an Asiair all you can change is the gain value.

I've had the uncooled cam using an lextreme any flats show red/green swirls mostly prominent around the edges but also infringe onto the target, this becomes exasperated when applying a dynamic background extraction to the linear calibrated stack. I just read this this morning from a post somewhere (I'll have to find it again) as since this relates to the above which I shorthanded:

"If you are using a camera based on the IMX294 or IMX492 sensor, test the gain that you intend to use before committing to image with it. The way to test is to set up the camera at the gain you want and significantly overexpose the image (long exposure, lots of bright light, etc). If you can get the histogram to be a single spike at the far right hand side then you are OK. If you cannot get the histogram to go to that single spike at the right in spite of massive overexposure then *do not use that gain value* - try a higher one if you want to take advantage of the lower read noise in HCG mode or a lower one if you want a bigger full well depth."

I've also seen videos by Adam Block showing similar test images and it relates to the dark flats overcompensating on the flats. I don't use PI so I can't personally test this as you can adjust the master dark flat before applying it.

I also know this sensor doesn't like fast frames as the biases tend to look very different from each other (taking a lot more and averaging may help), yet some people say they take less than second long flats, I don't see how that's possible with this camera.

I've also tried diffusing my led tracer panel even though it's on its dimmest setting running off the asiair when taking the flats (mine don't like usb power banks as they flicker or don't draw enough current so they turn off) with four 3mm thick transparent perspex sheets, two opal, one light grey, one dark grey closest to the objective so it doesn't reflect too much ambient light (I need to make a blacked out housing really), when you look up at the panel, it barely looks on, certainly dimmer than a sky flat sky.

Edited December 29, 2022 by Elp

[vlaiv]

2 minutes ago, Elp said:

From what I've read about the 294 due to how the HCG is implemented causes variability as to what gain setting you can use.

I now think I understand what is the root cause of the issues with 294.

Poor implementation in sensor itself (or firmware).

If I recall correctly - these are sensors have very small pixels that are "joined" into large pixels in groups of 2x2.

In that case - saturation must be handled carefully. I've described case where some pixels hit saturation and others don't - it is because of random nature of the signal. This can also happen inside group of 2x2 pixels.

Here is example - say we have signal that is 100e on average but clipping is at 110.

In group of 2x2 pixels some will record 96e some will record 105e - and it might happen that one pixel records 112e - but it can't - as clipping is at 110e.

When we join pixels - we can assume that clipping is now at 440 and we will have:

96 + 105 + 102 + 110 as actual pixel value so it will be 413 - less than 440 / clipping value - so we can think it is all ok - but it is not, as actual pixel value should be:

96 + 105 + 102 + 112 (this one got clipped to 110) - or 415

We get non linearity at high values without clipping.

This probably happens only on some gain settings close to switch when FWC is small or something and is probably design / implementation flaw.

 

[ONIKKINEN]

If you're planning to take flats with your 533, you really dont have to worry about nonlinearity or sufficiently long flats exposures as the camera is very linear from 0 to max value. If your panel gives you 0.8s flats, then great use those. If it gives you 10s flats then maybe try to brighten it a bit, you will spend unnecessarily long taking those and there is no real reason to. With very narrow bandpass filters there may not be a choice, but generally no need for so long exposures with flats.

With my 571 OSC camera i take 50 flats that are either 0.2s with a UV/IR filter or 1.1s with a narrow(ish) band triband filter. Neither have ever had any issues with calibration, even if the histogram is all over the place with reds being around 1/4th - 1/3rd of the way, greens bit behind middle and blues at slightly over half to 3/4ths of the histogram. As long as none of the channels are clipped to black or white, the flat will definitely work.

Not sure what you mean with a lighter shade of black, but if you mean that the edges are brighter than the center (overcorrection), then you have not applied darks or darkflats properly (or at all). With your 533 you can use bias frames as darkflats if youre lazy.

[Elp]

From what I've read, I've read quite a bit on it until I find a solution, it's generally recommended with the zwo 294mc (and the mm) not to use a gain any less than 200, HCG is supposed to kick in way before then, last time I used 250 which was okay but that was with a luminence filter.

With an lenhance or lextreme filter the issue is very troublesome.

One workaround I read this morning which I haven't tried, due to the RGB shifting so much in the flats with the optolongs was to just extract the blue channel from your master flat and use that as your master flat as the red and green are the ones which vary adding to the overcorrection, being blue channel though it will be quite blotchy.

Edited December 29, 2022 by Elp

[vlaiv]

2 minutes ago, ONIKKINEN said:

Not sure what you mean with a lighter shade of black

That is just software effect and has nothing to do with data - it happens because of this:

Imagine you have background that is not flat calibrated - and you have dark edges and brighter center - and you set your white point and black point according to those.

Now you apply flat correction without readjusting black and white point - background will turn to uniform gray as dark areas will no longer be dark (small in value) - but rather corrected.

Or in numbers - say you have target at 1000e, bright background at 100e and dark background at 10e - so your white point is 1000 and black point is around 10e - and you divide with flat which has 1 for bright background and 0.1 for dark background.

Target and bright background stay at 1000e and 100e / 1 = 100e, while dark background changes from 10e to 10e / 0.1 = 100e. Now all the background is uniform 100e and 100e - but black point is still at 10e and background looks greyish because of this.

[OK Apricot]

I'm not young to even pretend to have understood the above but I appreciate the time taken, thanks chaps. I'm going to try and make some sense of this. 

In the mean time, I will post up some flats on this thread that I will take tonight (if it's clear). A single flat and a master flat right? I just want to be sure I'm not making my images worse by applying bad calibration frames. 

Thanks once again 🙂

[Elp]

One of the benefits of the latest update is you can stack on the asiair with calibration frames, but you can't image whilst its doing it. So bit of a convenience factor if youre patient for the little unit to chug through the stacking.

To best analyse your images you can look at each individual stacks (lights, darks, etc etc) and histogram stretch them to see any defects and certain things should match up via a visual comparison (vignetting, dust, amp glow), but you won't see the actual effect until you fully integrate them into one stacked image. It's all pixel maths when they're all integrated, visually you can't really see issues as it's on a per pixel level when they're individual files, but with experience you know any warning signs to look out for.

Just go out and try it, the auto flat exposure usually is quite decent on the asiair, but for precaution I usually always take another manual set of calibration usually below the auto exposure. I wouldn't be very apprehensive about it, AP wouldn't be 4/5 as fun if it wasn't a challenge.

[ollypenrice]

Before using a flat I always look at it. We have a pretty good idea of what they should look like and, occasionally and for whatever reason, they don't look like that. Be very suspicious of such flats because they are almost certainly wrong.

Basically they should show some vignetting, a lot or a little depending on scope/sensor, and you can expect a few dust bunnies. That means a roughly circular brightening towards the middle with darker corners. Dust bunnies look like darker doughnut shapes.

Remember that an unstretched flat will look very flat indeed. That's normal. Even in an extreme case of vignetting, the 'dark' corners will have 75% of the illumination of the 'bright' centre. This is not a glaring difference. When we stretch a flat, the difference does become glaring.  That's why we need flats.  Unlike normal photographers, we stretch our data and so do the same things to our images that we do to our flats when we stretch them - we make the uneven illumination glaring.

Make it a habit to look at your flats and get a feel for what looks credible.

Olly

[Elp]

5 hours ago, Elp said:

post somewhere

https://forums.sharpcap.co.uk/viewtopic.php?t=5561

 

[Fegato]

Without going into any technical details, the advice I was pointed at was that the sensor did not behave consistently at very short exposures. So bias frames don't work effectively, and flats should be shot to about 2 seconds minimum (to be safe) and calibrated with flat darks. When using the camera, this is what i do, and it seems to work OK. I use an Aurora flat panel and have some darkening filters which allow me to get the flats above the desired exposure.

[OK Apricot]

OK so tonight's session has just finished. I took 30 flats and dark flats instead of 20 that I usually do. I completely forgot to take a screenshot so I will post up what they look like tomorrow. 

Do you need me to post just a single flat frame? Or do you need a master flat? Dark flat? Both stacked? 

[Elp]

Do your usual stacking routine with a very brief outline of what you did, histogram stretch the stack until it's overblown slightly bright whites dark blacks high contrast difference (same as levels editing in PS or GIMP) and post a screenshot for now. The raw files will take a while to upload. If you use Siril, histogram stretch preview is a simple drop down box toggle from the default to "histogram" in the lower centre half of the UI. Any opened image at the time will immediately overstretch and you'll see any issues with your session as a display only preview in the main window of the UI.

I usually use it to check every session by dragging and dropping each target stacks into the program once DSS is done stacking.

Edited December 30, 2022 by Elp

[OK Apricot]

2 hours ago, Elp said:

Do your usual stacking routine with a very brief outline of what you did, histogram stretch the stack until it's overblown slightly bright whites dark blacks high contrast difference (same as levels editing in PS or GIMP) and post a screenshot for now. The raw files will take a while to upload. If you use Siril, histogram stretch preview is a simple drop down box toggle from the default to "histogram" in the lower centre half of the UI. Any opened image at the time will immediately overstretch and you'll see any issues with your session as a display only preview in the main window of the UI.

I usually use it to check every session by dragging and dropping each target stacks into the program once DSS is done stacking.

With just the flat/dark flat frames or  everything (lights etc) as well? 

[Elp]

Apply everything other than bias.

[OK Apricot]

OK, so a short description of the stacking process - Uploaded 58 lights, master dark (20 of, couple months old), 30 flats (3.2s) and 30 dark flats. Kappa-sigma clipping of lights, median kappa-sigma for F/DF. These are just default settings, I've not played with these. Stacked 57 out of 58 light frames and went into photoshop.

I wasn't quite sure how to respond to you saying to slightly overblow the image, so all I've done here is use levels and stretch until it's looking overblown. Whether or not it's helpful is another matter 😂. I've uploaded both the linear and overblown files. Hopefully they're of some use? I feel like the masterflat is telling me all is well and my flats are doing what they're supposed to be, but I just wanted to check with you more experienced folk. If anything else is needed, please let me know!

Here is a single flat with histogram:

 

Here is the masterflat:

 

 

Overblown.tif Linear.tif

[Elp]

Your colour stacked image doesn't look too bad but your flat looks uneven in colour, if you overstretch it you can see the issue:

 

Now, in RGB imaging background uneven sky illumination (gradient) is usually the main issue you need to address in order to process your images further. I normally address this within Siril via the background extraction function. Conducting this on your image results in this:

This is clearly incorrect, it is overcorrecting due to the flat/dark flat initial overcorrection. You can sometimes get away with it by not overstretching the image too much but if you were imaging something faint or something like a reflection nebula, you'd be limited on how much you can stretch it due to this overcorrection in the red and blue. This is the issue the 294MC sensor causes. The effect is much much worse when using an Optolong  L-enhance or L-extreme (maybe other filters also).

At this present moment I haven't found a solution as of yet.

[OK Apricot]

 

On 01/01/2023 at 15:52, Elp said:

Your colour stacked image doesn't look too bad but your flat looks uneven in colour, if you overstretch it you can see the issue:

Is this because of using a duo band filter? The master flat looked pretty uniform so I'm a bit confused here.

Gradients I can for the most part deal with in post, but it would be nice to lessen the burden with some quality calibration frames. 

What should I do differently to get a more uniform result? 

[Elp]

If it's not affecting your post processing id stick with what you're doing already as no reason to change if it's working.

For me and my camera it's an issue so I'll continue to try other things.