Why do my tablet flats over-correct?

[Richard_]

Hi all,

I've run into a bit of a strange instance when calibrating my OSC subs using my master flat generated from a white tablet screen, especially with my L-eXtreme filter. One corner seems to overcorrect using these flats, however, when I use skyflats I don't see this overcorrection and the end result looks pretty good. Whilst the answer is clearly "take skyflats", I'm not always fortunate enough to take flats first thing in the morning (weather wise, it is the UK!) so it would be great to understand what's going on with my tablet and how to avoid this issue. Details are below, but my questions are:

1. How can I prevent over-correction with tablet flats? Am I over-exposing them?
2. If I change my light source, will I see this same issue if using an EL panel (e.g. like this one )or a bespoke flat field generator (like those provided by Geoptik, Pegasus Astro etc.)
3. Is there anything in the PI WBPP 2.0 script which can help identify overcorrection and adjust the amount of correction applied?

 

Acquisition

For the tablet method, I capture my flats using the auto-exposure tool in my ASI Air Pro (it calculates the optimum exposure time based on the light source presented). The light source is my tablet display at 100% brightness, no blue-light/night time filter, and displaying this blank, white page provided by Covington on the screen. The display is large enough to cover the entirey of my Redcat 51. I attach 2 layers of t-shirt to the front of the telescope and the camera used is an ASI533. It's a small sensor, so I wouldn't expect to see a significant amount of vignetting compared to a full frame camera. For the skyflats, I used the same approach but used 4 layers of t-shirt isntead of 2 to ensure I don't end up with too short of an exposure time.

The exposure time has been adjusted to ensure the average value of the individual frame is ~30,000 ADU for a 14-bit image. For the tablet flats, this results in a 7 second exposure using 2 layers of t-shirt. For sky flats (overcast day), this results in 180ms flats using 4 layers of t-shirt to ensure the exposure time wasn't too low. The optical train had not been disturbed in this time, and both approximate exposures were determined by the ASI Air Pro auto-expose feature (I rounded them up when shooting). There were 30 x flats for each case. The tablet flats were taken at midnight, the skyflats were taken in the morning. Other than this, there was no difference in imaging conditions.

Processing

All stacking is performed in PixInsight using the WBPP 2.0 script. All settings under flat are left at default settings. The script was executed twice: the first was using tablet flats, the second was using skyflats. No other changes were made in the stacking process.

Flat Images and MasterLights

Below are four screenshot images. The top row shows debayered master flats (purely for illlustration purposes), the bottom row shows the final stack. The left side shows the tablet flats, the right side shows the skyflats. The statistics are from the master flat, pre-debayered, and show there there is a small sifference in mean values. However, the images show much brighter corners for the tablet flats, especially in the lower left where I see over-correction. You can see this clearly in the master light images. Looking at the master light which was calibrated using tablet flats, you can see a red cast over the lower left corner which I suspect is overcorrection from the flat. However, you do not see this on the master light which was calibrated using skyflats, the background looks fairly uniform.

 

[michael8554]

I'm used to seeing Flats that have an even overall colour caste, not Flats that are red in the middle and green at the edges ?

Michael

[vlaiv]

What scope are you using?

One thing to try would be to add long dew shield kind of shroud made out of very dark material (flocked on the inside) and take your flats like that.

Flat panels work well with baffled and flocked scopes. Some scopes can't cope with that much light hitting inside of the scope at different angles. Newtonians are prime candidate for this sort of behavior.

[Richard_]

1 hour ago, michael8554 said:

I'm used to seeing Flats that have an even overall colour caste, not Flats that are red in the middle and green at the edges ?

Michael

I'm using a OSC camera (ASI533) with the dual narrowband L-eXtreme filter. I understand that flats don't require debayering for normal calibration and stacking, but the images above were debayered just to illustrate the brighter corners (for some reason, I struggled to show the brighter lower corner as a good comparison of the master flats without debayering). So perhaps the colour cast is a consequence of this filter type and the fact that the master flats have been debayered? I could try taking flats using just my UV/IR cut filter to determine if I see the same thing in the corner?

 

1 hour ago, vlaiv said:

What scope are you using?

One thing to try would be to add long dew shield kind of shroud made out of very dark material (flocked on the inside) and take your flats like that.

Flat panels work well with baffled and flocked scopes. Some scopes can't cope with that much light hitting inside of the scope at different angles. Newtonians are prime candidate for this sort of behavior.

Per my OP, the telescope is a Redcat 51. The dew shield is always extended during imaging and acquisition of flats. The dew shield has the original black flocking on the inside and has not been replaced. I couldn't comment on how effective the black flocking is as I have nothing to compare it to.

If the issue is too much light reflecting off the flocking, then surely the skyflats would be most affected since the lower exposure time means it's a brighter light source than what is provided by the tablet? 

[vlaiv]

6 minutes ago, Richard_ said:

If the issue is too much light reflecting off the flocking, then surely the skyflats would be most affected since the lower exposure time means it's a brighter light source than what is provided by the tablet? 

Sky is very far away and tablet is very close - that is the difference. In ideal world - only rays of light that are few degrees off optical axis should reach sensor (depending on focal length and size of your FOV) - but flat panel close to opening of telescope emits light in all directions and all different angles.

In fact, now that I think about it - maybe issue with screen as flat panel is not because it is close - but with viewing angles.

Take your tablet and see if illumination and color changes with angle. That might be causing the issue.

In any case, you can get flat panel for very small amount of money nowadays. Don't purchase ready made flat panels for astronomy - those are ridiculously expensive. You can purchase simple LED panel light. I recently installed lighting in my new house and had opportunity to see what cheap led lighting looks like - and it struck me - ideal for flat panels for telescopes.

They come with AC/DC transformer that you can easily remove and hook them up to 12V.

[Craig a]

On 27/09/2021 at 09:44, vlaiv said:

What scope are you using?

One thing to try would be to add long dew shield kind of shroud made out of very dark material (flocked on the inside) and take your flats like that.

Flat panels work well with baffled and flocked scopes. Some scopes can't cope with that much light hitting inside of the scope at different angles. Newtonians are prime candidate for this sort of behavior.

Interesting that is because I was getting some horrid circle gradients using an led tracing pad on my 10 inch f4 Newtonian with the pad laying straight ontop of the scopes aperture even with a t shirt it must of still been too bright even though the scope was flocked and has baffles the light still must of been bouncing around in there 

[newbie alert]

Firstly , do you need to callibrate out dust bunnies or vignette?

14 bit camera's have a adu range of? 30000 seems excessive as over correcting shows...

[Richard_]

Thanks for the suggestion, Vlaiv, I'll look into that.

On 10/10/2021 at 21:21, newbie alert said:

Firstly , do you need to callibrate out dust bunnies or vignette?

14 bit camera's have a adu range of? 30000 seems excessive as over correcting shows...

It's mostly for vignetting caused by the filter. The sensor size is far smaller than the image circle so there is no significant vignetting like you see with full frame cameras. Q

Since my scope is a small refractor it's easy to blow dust off the objective lens. My image train does not get disassembled so the likelihood of introducing dust is fairly low. 

Although the camera is 14-bit, the ASI displays values representative of 16-bit imagos. Therefore, the maximum value is 65,536 ADU. My ASI Air automatically sets the exposure time to get into the 30,000 - 35,000 ADU range but of course I can choose whatever exposure time I want. 

[newbie alert]

16 hours ago, Richard_ said:

 

Although the camera is 14-bit, the ASI displays values representative of 16-bit imagos. Therefore, the maximum value is 65,536 ADU. My ASI Air automatically sets the exposure time to get into the 30,000 - 35,000 ADU range but of course I can choose whatever exposure time I want. 

A friend also had an Air, and with his flats they were also over correcting...I don't know why zwo say  50% of the cameras range but if it's over correcting then it's obviously too much  .. I'd suggest a third, so a adu of about 22000  is about right..  so try it and see

But a pure white evenly illuminated light source  shouldn't introduce colour gradients 

[ollypenrice]

Your badly flattened version, in the problem corner, shows channel brightnesses in descending order, of red, blue, green, in the background sky. they are miles out of balance with green being far too low. This suggests that your panel has a corner which is brighter in green than elsewhere, no?

Olly

[tomato]

+1 for using a flat light panel rather than a tablet screen. My flat calibration procedure was much more consistent when I moved from an IPad screen to a light panel.

[wimvb]

How do you calibrate your flats? Overcorrecting flats can be caused by poor flat calibration. With a cmos camera, you need to be very precise with calibration.

Keep the exposures such that the sensor is in its linear response region (for me that is the case when I aim for 25000 ADU).

Use a colour neutral light source if possible. If not, make sure that the three colour channels in an osc are all in their linear response region.

Keep the camera at a constant temperature, gain and offset.

Shoot flat darks at the exact same settings as your flats. Flat darks that don't match flats, can leave an offset in the master flat that will result in over- or undercorrection during light frame calibration.

 

[wimvb]

On 10/10/2021 at 22:21, newbie alert said:

Firstly , do you need to callibrate out dust bunnies or vignette?

14 bit camera's have a adu range of? 30000 seems excessive as over correcting shows...

The 14 bit output of a sensor is stored in the high bit range of the 16 bit image files. So, pitch black is 0 in 14 and 16 bit. The next light level is 1 in the 14 bit ADC output, and 4 (100 binary) in the 16 bit image file. Then 2 in 14 bit and 8 in 16 bit, etc. The brightest level is 16000 (rounded) in the 14 bit ADC, which becomes 64000 (rounded) in the 16 bit image file. So, 30000 in the output image file, is actually 7500 in the ADC output of the sensor. But depending on the camera or settings, that is not necessarily in the linear response range of the sensor. Especially since the 30000 value is an average of the entire image. The central region of the image (and local pixel response) may very well be a bit higher.

[Richard_]

Ok, so I've just ordered a light panel similar to the one Vlaiv posted so let's see how I get on when it arrives! For almost £6, it'll be worth a shot. 

Some good discussion and problem solving here, thanks for the suggestions all! 

On 17/10/2021 at 08:28, wimvb said:

How do you calibrate your flats? Overcorrecting flats can be caused by poor flat calibration. With a cmos camera, you need to be very precise with calibration.

Keep the exposures such that the sensor is in its linear response region (for me that is the case when I aim for 25000 ADU).

Use a colour neutral light source if possible. If not, make sure that the three colour channels in an osc are all in their linear response region.

Keep the camera at a constant temperature, gain and offset.

Shoot flat darks at the exact same settings as your flats. Flat darks that don't match flats, can leave an offset in the master flat that will result in over- or undercorrection during light frame calibration.

 

The images in my original post show the ADU readouts for my master flats, which are on average ~22,500 and ~20,700 for tablet and sky flats respectively. Granted, the AAP can over correct as it aims for just over 30,000 ADU, but I purposely reduced the exposure time to prevent this from happening. These are well below the ~30,000ADU range most people aim for and given the negligible difference in values between tablet and sky flats, I'm inclined to think that the problem is due to non-parallel light emitted from the light source, exactly as Vlaiv mentioned in his post. 

The tablet light source colour was as neutral as I could make it. Redlight/night light filters were disabled and I was using maximum display brightness whilst displaying a white image. 

Camera settings were exactly the same as lights (even the cooling temperature, which I'm told doesn't really make a difference). Dark flats were taken under the same conditions too using the exact same exposure time. With respect to the below quote, what exactly do you mean?

If not, make sure that the three colour channels in an osc are all in their linear response region.

 

 

[wimvb]

4 hours ago, Richard_ said:

With respect to the below quote, what exactly do you mean?

If one colour is stronger than the others, their average or median may be low, but the strongest colour may become saturated. Saturation in camera sensors is not abrupt but gradual. In an osc, the master flat isn’t debayered, and the average and median are calculated for all pixels together.

[Richard_]

Ah got it, you mean to check that the histogram of each channel is centred and not clipping the right side at saturation, right? 

 

If I recall, a common issue with the ASI533 was that the green channel was shifted to the right when people compared their debayered flats. People were talking about changing the white balance in the camera driver settings but this sounds a bit complicated for me and given that I use an ASI Air I don't think I can change those settings for the camera. I didn't find any problem with my regular flats, just my l-extreme flats, so I decided I didn't need to tinker with my white balance. 

When my light panel arrives I'll be sure to run some tests using different filters and without filters to see if I can recreate my issue or whether it's resolved. 

[wimvb]

19 minutes ago, Richard_ said:

Ah got it, you mean to check that the histogram of each channel is centred and not clipping the right side at saturation, right? 

Basically, yes. But saturation effects, or rather, non linear response is subtle. A sensor’s response can become non linear long before a pixel is ”full” or saturated. In the linear response region, an increase of, say, 10% in the input signal results in a 10% increase of the output. In the non linear response region, the increase in output is less than that, but still more than 0%. In the true saturation region, any increase in input signal will not result in a change in output; the pixel is truly full. When taking flats, you need to be well outside of the non linear region, not just the saturation region. Usually being at half of a sensor’s maximum output (30 000 ADU) is ok, but lower can be safer.

[Richard_]

Thanks for clarifying, makes sense to me. Do you know if there's information out there that shows which ranges are linear and non-linear for a given camera/sensor or is your previous comment more of a rule of good thumb and/or experience? 

[wimvb]

55 minutes ago, Richard_ said:

more of a rule of good thumb and/or experience? 

Rule of thumb and experience from an earlier life in the semiconductor industry. So far, I haven't seen linearity diagrams for sony cmos sensors.

[Richard_]

Alright, the LED panel arrived today. The panel I ordered comes with a removable LED driver but the pictures didn't clearly show what voltage and current was outputted. The output of the driver is DC54-72V with a current of 280mA. So this is the voltage/current requirements for the light. See attached pictures. 

Call me an idiot, but when I remove the LED driver, I have no idea how I can connect this directly to a 12V DC output such as my ASI Air Pro to power the light. I obviously tried it but nothing happened. Surely I need some form of device to boost the voltage? Or perhaps I ordered the wrong light and that I should order a light which requires 12V DC outputted from the driver? 

[ONIKKINEN]

On 19/10/2021 at 20:25, wimvb said:

Rule of thumb and experience from an earlier life in the semiconductor industry. So far, I haven't seen linearity diagrams for sony cmos sensors.

Doesn't sharpcap sensor analysis measure this or is this some other number?

Not the same camera, or sensor but from the same Sony Exmor lineage of sensors so could be similar. Mine shows sensor linearity to 99,3%. Does this mean the last 0,7% experiences a nonlinear response, or that its overall 0,7%? Never really understood what that number stood for.

 

I think i have some issues with light panel flats aswell, but unsure if its related to anything discussed here.

On 27/09/2021 at 11:44, vlaiv said:

Flat panels work well with baffled and flocked scopes. Some scopes can't cope with that much light hitting inside of the scope at different angles. Newtonians are prime candidate for this sort of behavior.

I think my issues lie here, i am flocking my tube to see if my flats improve.

[vlaiv]

1 hour ago, Richard_ said:

Call me an idiot, but when I remove the LED driver, I have no idea how I can connect this directly to a 12V DC output such as my ASI Air Pro to power the light. I obviously tried it but nothing happened. Surely I need some form of device to boost the voltage? Or perhaps I ordered the wrong light and that I should order a light which requires 12V DC outputted from the driver? 

In that case look for DC/DC converter.

Something like this:

https://www.hycopower.com/product/12V-to-72V.html

for example or similar - just make sure it can output at least 0.5A