The Flats Challenge

[1CM69]

Hi all,

I seem to be getting in a right pickle in an attempt to capture correctly exposed Flats.

I didn't think that I had an issue until recently moving to APP for my processing & away from DSS. Seems that APP is very strict when it comes to calibration rules.

Anyhow, there seems to be a whole lot of info out there for people using PixInsight in regards to ADU median etc... but I am trying to get Flats for my modded Canon EOS750d.

I am using a homebuilt lightpanel with a 3mm opal acrylic diffuser sheet.

The lightpanel has 3 brightness levels, I put it on the lowest.

In BYEOS I set my camera to AV & keep the ISO as per my Lights & Darks, generally ISO800.

BYEOS gives me a Flat frame keeping the histogram near central & the exposure length is 1/2000th sec. Initially I didn't pay any further attention to this fast capture until a passing remark on the APP forum which led me to believe that my captures were too fast.

With further research I found that I should be aiming for exposures of somewhere between 0.5 - 5 secs.

After hours & hours of messing around with differing setups I found that I could get a central histogram in BYEOS using the AV method, if I set my ISO to 100, place 2 T-shirts between my scope and the lightpanel & exposed for 0.8sec.

Great I thought, I've got it, APP lays out that ISO of Flats & Lights do not have to match but then there is this whole rigmarole in getting extra calibration files  to match these captures of the different ISO.

So, back to the drawing board & more trial an error trying to capture Flats of a reasonable length at the same as my standard go to ISO of 800.

More internet searching.....

I found this page: http://www.myastroscience.com/proper-flats-with-dslr although it is again talking mainly about PixInsight, there's a lot of usable info.

So my DSLR is 14bit & I should be looking for a mean of 8192 as info here: http://cpn.canon-europe.com/content/education/infobank/capturing_the_image/8_or_16_bit.do and the only software that I own that measures in ADU is SGPro.

Following the suggestion in the 1st link I have posted, I setup my camera in SGPro to take a 20sec frame with the 2xt-shirts & lightpanel on low setting, were attached to my OTA. This gave me the overexposed MEAN ADU value, (bearing in mind that SGPro converts 14bit to 16bit values), of 61210.

Now I know that I should be aiming for somewhere close to 61210/2 or 30605 ADU MEAN in SGPro.

In SGPro I used the 'Flats Calibration Wizard', put a check next to ISO800, changed the 'Target Mean' to 30605 & hit start.

After numerous attempts the wizard would not automatically calibrate but I was able to manually achieve the result by altering the 'MAX Exposure Time' section, eventually finding that 0.85sec got me to within 500ADU of the target.

This captured frame looked very good in SGPro's histogram too, pretty much central although having 2 bumps with a central depression.

I now turned my attention back to BYEOS & set the exposure manually with my camera on 'M' not 'AV' & shutter speed to 0.8sec as 0.85 is not available as an option and fired off a shot with lightpanel + t-shirts as they were in SGPro.

Now this new frame in BYEOS shows the histogram way off to the right, so much so that the green channel has completely disappeared with the red channel up against the right side & blue just a little to the left of the red.

However, when I open this frame in SGPro it is almost a perfect match to the 0.85sec frame I actually took in SGPro.

So after all of this messing around, what am I supposed to trust, BYEOS or SGPro and is 0.8sec a satisfactory length for a Flat field frame?

Regards..,

Kirk

[1CM69]

I have just opened the Flats taken in BYEOS & SGPro and loaded them in to Nebulosity to check the Pixel Stats and there is very little difference between them but SGPro also creates a .fits version of the same RAW file and when this is loaded in to Nebulosity, it is quite different to the regular .cr2 files.

Here is a link to my data files: https://www.dropbox.com/sh/kyqn41qyuram0s6/AADjjx8x3S2zyLPpWZMe74ASa?dl=0

Next I opened the .fits file that SGPro created and loaded this in to AstroImageJ & used the Analyse > Interactive 3-D Surface Plot, this is how the Flat looks, is this normal?

regards..,

 

[ollypenrice]

That's an interesting plot. It's a safe bet that the parallel lines of bright peaks crossing the short side of the chip are not of optical origin. They will be electronic. They are very localized and they follow the pixel matrix. It will be some kind of fixed pattern noise. The question is, should they be in the flats? If they're in the lights then perhaps they should be, but if calibration has corrected them in the lights then they'll introduce that noise pattern.

Like many people I use a master bias as a flat dark. (That's to say a dark for flats.) It's possible that this might remove the noise pattern.

Olly

[1CM69]

1 hour ago, ollypenrice said:

That's an interesting plot. It's a safe bet that the parallel lines of bright peaks crossing the short side of the chip are not of optical origin. They will be electronic. They are very localized and they follow the pixel matrix. It will be some kind of fixed pattern noise. The question is, should they be in the flats? If they're in the lights then perhaps they should be, but if calibration has corrected them in the lights then they'll introduce that noise pattern.

Like many people I use a master bias as a flat dark. (That's to say a dark for flats.) It's possible that this might remove the noise pattern.

Olly

That would be my understanding, limited as it is, that the Bias for Flats also has the same pattern & is subtracted from the Flats. 

Just a bit too uniform to be optical I think. 

When I load the fits in liberator & check the black clip I see the image corners highlighted as expected & checking the white clip shows a central area, again, as expected. 

Thought id share the plot as it looked interesting. 

 

Kirk

[sharkmelley]

On ‎16‎/‎03‎/‎2018 at 16:41, 1CM69 said:

 

Next I opened the .fits file that SGPro created and loaded this in to AstroImageJ & used the Analyse > Interactive 3-D Surface Plot, this is how the Flat looks, is this normal?

 

I cannot relate either the CR2 file or FIT file to the above plot in any way at all.  Careful examination of the files shows they are totally normal without any of the lumpiness/bumpiness displayed in that plot. 

Mark

[1CM69]

14 minutes ago, sharkmelley said:

I cannot relate either the CR2 file or FIT file to the above plot in any way at all.  Careful examination of the files shows they are totally normal without any of the lumpiness/bumpiness displayed in that plot. 

Mark

OK, thanks for trying

[wimvb]

The problems you're experiencing (not the banding issue) stem from using in camera white balance.

When I load both your raw files into PixInsight and measure statistics, I get maximum pixel values around 2.1 (SGPro) and 2.3 (ByEOS). These values stay the same when I debayer the images in PI. I also get these highest values in the green channel. This would be expected if you use a dimmed white light source, since the sensor is more sensitive to green. But when I set input debayer options to use camera white balance, the debayered image comes out pink, with too high pixel values (clipping the red and green channels).

Debayered with camera settings (also note the image orientation, applied from camera settings):

Debayered in PixInsight (no image rotation):

In image calibration, darks, bias and flats are never debayered. These should work fine.

I also made a contour plot of the ByEOS image. Apart from a few small dust bunnies, the flats are clean. No sign of banding. On properly exposed flats, the read pattern of the sensor should be burried in the signal and should never be visible. Subtracting the bias signal from flats is only necessary because flats will be scaled in the calibration process. This scaling doesn't work if there is a pedestal (offset, bias) present, which is why it needs to be removed.

(click on the images to see full size versions)

[1CM69]

4 hours ago, wimvb said:

The problems you're experiencing (not the banding issue) stem from using in camera white balance.

When I load both your raw files into PixInsight and measure statistics, I get maximum pixel values around 2.1 (SGPro) and 2.3 (ByEOS). These values stay the same when I debayer the images in PI. I also get these highest values in the green channel. This would be expected if you use a dimmed white light source, since the sensor is more sensitive to green. But when I set input debayer options to use camera white balance, the debayered image comes out pink, with too high pixel values (clipping the red and green channels).

Debayered with camera settings (also note the image orientation, applied from camera settings):

Debayered in PixInsight (no image rotation):

In image calibration, darks, bias and flats are never debayered. These should work fine.

I also made a contour plot of the ByEOS image. Apart from a few small dust bunnies, the flats are clean. No sign of banding. On properly exposed flats, the read pattern of the sensor should be burried in the signal and should never be visible. Subtracting the bias signal from flats is only necessary because flats will be scaled in the calibration process. This scaling doesn't work if there is a pedestal (offset, bias) present, which is why it needs to be removed.

(click on the images to see full size versions)

Hi Wim,

thanks for the in-depth post.

A couple of questions.

Are you saying that there is nothing wrong at all with my Flats?

Should I be turning off WB in my camera for taking Flats?

The image rotation that you have pointed out is strange as my camera was in regular horizontal orientation.

Regards..,

Kirk

[wimvb]

5 hours ago, 1CM69 said:

Are you saying that there is nothing wrong at all with my Flats?

Not as far as I can see. Maybe even underexposed.

 

5 hours ago, 1CM69 said:

Should I be turning off WB in my camera for taking Flats?

Mono astro cameras are much easier when it comes to calibration frames, ad there's no on camera processing. Dslrs were not designed for AP, so there's some processing done by the image processor. The image information (exif) said manual white balance, which normal raw conversion programs use for debayering.

5 hours ago, 1CM69 said:

The image rotation that you have pointed out is strange as my camera was in regular horizontal orientation.

Exif had "rotation by software" set, but orientation was horizontal (landscape). Both PixInsight and RawTherapee rotated the image. I have no idea why.

As I wrote before, calibration frames aren't debayered, so this shouldn't be a problem.

[1CM69]

49 minutes ago, wimvb said:

Not as far as I can see. Maybe even underexposed.

 

Mono astro cameras are much easier when it comes to calibration frames, ad there's no on camera processing. Dslrs were not designed for AP, so there's some processing done by the image processor. The image information (exif) said manual white balance, which normal raw conversion programs use for debayering.

Exif had "rotation by software" set, but orientation was horizontal (landscape). Both PixInsight and RawTherapee rotated the image. I have no idea why.

As I wrote before, calibration frames aren't debayered, so this shouldn't be a problem.

Excellent, thanks.