Flats calibration

[david_taurus83]

Question on flats. Should a master flat correctly calibrate and remove dust artifacts and vignetting from the same flat frames themselves?

The flats would have the bias removed prior to integration. Camera is Canon 6D and ISO will be 1600, same as my lights.

I want to experiment indoors in a dark room to get good usable flats for my lights. If I can get flats to work on the individual flat frames them they should also work with the lights, right?

[vlaiv]

Yes, but it is best not to use individual flats used to create master flat.

Best flat flat test is done when parameters are changed.

Use one exposure length / light source intensity to create set of flats for master flat, then change parameters and take single or few additional flats.

You can then try to flat calibrate additional flats (single or even stack of few) with master flat and it should leave you with uniform image (it will have some offset and some shot noise but it should not have visible intensity changes).

Alternatively - you can exclude one flat from your basic set, use all others to create master and then flat calibrate selected flat with master created out of all others.

[david_taurus83]

Thanks vlaiv. So I should shoot a set of flats, subtract bias, create master flat. Then, perhaps dim the light source even more and try a 10s exposure. Check if I can calibrate that with the master flat.

 

[vlaiv]

Just now, david_taurus83 said:

Thanks vlaiv. So I should shoot a set of flats, subtract bias, create master flat. Then, perhaps dim the light source even more and try a 10s exposure. Check if I can calibrate that with the master flat.

 

Yep, that should be good way to do it. Don't forget to also remove bias (or rather dark as it is 10s exposure) for second sub. Emulate whole process of calibration although your light is actually just created with flat panel.

[david_taurus83]

So some experimenting today.

 

Below single 20s image and calibrated image.

 

Its correcting the dust spots but looks to be over correcting the right side.

 

See below stack of 15 images or so and the master flat.

 

Normally I shoot from my back garden so I would have just assumed this gradient was a result of the local streetlights. But I have found a "dark" site not far from where I live (literally 5 minute drive!) and there is no local LP whatsoever from streetlights etc so this gradient is definitely the flats.

 

Perhaps the vignetting is simply too much?

 

 

All images were taken with my 6D on a Redcat 51 and captured via Ekos/Kstars. What bugs me a little is that Ekos doesnt like Canons CR2 files so converts everything to FITS. In doing so, it also alters the size of the image frames from the standard CR2 size of 5496x3670 to 5472x3648. So I also had to shoot a new set of Bias images to match.

[vlaiv]

6 minutes ago, david_taurus83 said:

Below single 20s image and calibrated image.

Can you do very simple experiment?

I'm guessing that you used bias for calibration of both master flat and single flat of 20s exposure?

Can you measure average ADU of that bias and also average ADU of 20s dark exposure. If these two numbers are significantly different - maybe try using flat darks instead of bias.

[david_taurus83]

Basically identical. 0.001s bias and 20s dark frame. Mean value of 2047.

I've also realized that it was an error on my behalf that the frame size was incorrect. I reset everything and upon restarting, Ekos asks the user to input the sensor resolution. I must have read an incorrect size off the web when I first set up the profile.

 

 

[vlaiv]

1 hour ago, david_taurus83 said:

Basically identical. 0.001s bias and 20s dark frame. Mean value of 2047.

That is rather interesting. Stddev (avgdev) value shows significant distinction.

Stddev is mix of bias (read noise + read signal) and dark current noise. It shows 8.6ADU in first case and 10.7ADU in second case.

That would make dark current expressed in ADU around 6.366.

Can you read off FITS header information? It should show black level automatically removed from raw file when converted into fits - at least FitsWork creates that Fits header - here is example of one raw file converted to FITS:

[ollypenrice]

10 hours ago, david_taurus83 said:

So some experimenting today.

 

Below single 20s image and calibrated image.

 

Its correcting the dust spots but looks to be over correcting the right side.

 

See below stack of 15 images or so and the master flat.

 

Normally I shoot from my back garden so I would have just assumed this gradient was a result of the local streetlights. But I have found a "dark" site not far from where I live (literally 5 minute drive!) and there is no local LP whatsoever from streetlights etc so this gradient is definitely the flats.

 

Perhaps the vignetting is simply too much?

 

 

All images were taken with my 6D on a Redcat 51 and captured via Ekos/Kstars. What bugs me a little is that Ekos doesnt like Canons CR2 files so converts everything to FITS. In doing so, it also alters the size of the image frames from the standard CR2 size of 5496x3670 to 5472x3648. So I also had to shoot a new set of Bias images to match.

It's best to measure the vignetting rather than eyeball a very posterized stretch. Just take the linear master flat and open it in software which lets you measure individual pixel values. I use my stacking/calibrating software, AstroArt, for this but lots of pre-processing packages let you do it. I get considerable vignetting in my Tak 106/2inch mounted filter/full frame CCD setup. The corners will give around 19,000 ADU when the centre is at around 23,500. That's a big difference but it calibrates out fine.

Olly

[david_taurus83]

9 hours ago, vlaiv said:

That is rather interesting. Stddev (avgdev) value shows significant distinction.

Stddev is mix of bias (read noise + read signal) and dark current noise. It shows 8.6ADU in first case and 10.7ADU in second case.

That would make dark current expressed in ADU around 6.366.

Can you read off FITS header information? It should show black level automatically removed from raw file when converted into fits - at least FitsWork creates that Fits header - here is example of one raw file converted to FITS:

No, FITS header doesn't show anything like that.

[david_taurus83]

1 hour ago, ollypenrice said:

It's best to measure the vignetting rather than eyeball a very posterized stretch. Just take the linear master flat and open it in software which lets you measure individual pixel values. I use my stacking/calibrating software, AstroArt, for this but lots of pre-processing packages let you do it. I get considerable vignetting in my Tak 106/2inch mounted filter/full frame CCD setup. The corners will give around 19,000 ADU when the centre is at around 23,500. That's a big difference but it calibrates out fine.

Olly

Hi Olly. The corners of the master flat are around 4600 ADU and the centre is 6300 ADU but camera is 14 bit so only goes up to 16k.

[Seelive]

11 hours ago, vlaiv said:

That is rather interesting. Stddev (avgdev) value shows significant distinction.

Stddev is mix of bias (read noise + read signal) and dark current noise. It shows 8.6ADU in first case and 10.7ADU in second case.

That would make dark current expressed in ADU around 6.366.

Is the large deviation in the Stdev of the 2 images primarily a result of the 'hot pixels' (which are evident in dark image)?  There is a significant positive bias about the mean in both the bias (-124.6, +390.4) and dark (-138.5, +11995.5) images.  Presumably if the Stdev was measured in the same area of both images avoiding the hot pixels, the difference in the Stdev would be much less?

[vlaiv]

39 minutes ago, Seelive said:

Is the large deviation in the Stdev of the 2 images primarily a result of the 'hot pixels' (which are evident in dark image)?  There is a significant positive bias about the mean in both the bias (-124.6, +390.4) and dark (-138.5, +11995.5) images.  Presumably if the Stdev was measured in the same area of both images avoiding the hot pixels, the difference in the Stdev would be much less?

I don't think so. Although visually there is plenty of hot pixels - they are actually rather "sparse" - like less than 1 in 1000. For say 20Mp sensor - that would give roughly less than 20,000 hot pixels - visually that would clutter the image but it would only minimally impact standard deviation.

[Seelive]

1 hour ago, vlaiv said:

I don't think so. Although visually there is plenty of hot pixels - they are actually rather "sparse" - like less than 1 in 1000. For say 20Mp sensor - that would give roughly less than 20,000 hot pixels - visually that would clutter the image but it would only minimally impact standard deviation.

OK, so I thought I would try an experiment; I created a 5496 x 3670 pixel (the same image size as the Canon 6D) 'Bias image' using IRIS with a mean level of 2048 ADU and Gaussian noise of 8.3 ADU (RMS).  Using the STAT command the image standard deviation ('Sigma' in IRIS terms) is (obviously) 8.3.  Adding just 6 individual pixels with an amplitude of 14000 ADU results in a new image standard deviation value of 10.6 ADU (which is the equivalent of adding 6.6 ADU (RMS) of Gaussian noise).

Or am I missing something?

[vlaiv]

7 minutes ago, Seelive said:

Or am I missing something?

Nope, you are absolutely right. I was wrong to assume that it won't make much of a difference.

I quickly created blank image with just a few hot pixels and made a measurement:

It indeed shows that standard deviation is about 7.6 in this case.

My logic was that standard deviation is calculated by dividing with number of samples - which in this case is 5496 x 3670 or 20170320 (about 20 million)  - but forgot to factor in that what is being divided is sum of squares rather than regular sum. 14000 is rather high number and 14000 squared is even larger - 196,000,000

[vlaiv]

18 hours ago, david_taurus83 said:

Below single 20s image and calibrated image.

I still wonder why is calibration failing.

How do you perform calibration in terms of math? Do you use 32bit numbers, do you create master with any sort of optimization - or simple average without rejection?

[david_taurus83]

31 minutes ago, vlaiv said:

I still wonder why is calibration failing.

How do you perform calibration in terms of math? Do you use 32bit numbers, do you create master with any sort of optimization - or simple average without rejection?

I'd have to look at the settings in Pixinsight but I'm sure average is the default setting. Just stack of 50 bias files to create master bias. Then calibrate flats with this and then lights with master bias and master flat. No optimisation used at the lights stage. I've not had this issue when I use NINA but that works with native CR2 files. I think I'll have to give it a go with NINA on a test subject, lights and flats and then follow up with Stellarmate with its FITS files. Compare the 2.

[Seelive]

8 minutes ago, david_taurus83 said:

I'd have to look at the settings in Pixinsight but I'm sure average is the default setting ...

I tend use median stacking rather than average for calibration frames (the default in DSS?) to provide some rejection of outlier pixels.

[david_taurus83]

I've had somewhat of a result with this. Its more of a solution to the problem than finding out the cause and prevention! I had tried taking flat darks, full length darks, CR2 files from NINA, FITS files from Ekos, all the same. I found a method described by Adam Block on his Youtube page where he shows that the flat frame can be manipulated slightly to try and fix over correcting. In his method he applies a value of 0.2 and 0.3 to all the pixels in his master flat (in Pixinsight 1.0 is completely white) and shows how it can affect the calibration. I tried this but I had to use a much smaller value of 0.02. Link to his video below. The best improvement I had though is by unticking a box in the calibration tool in Pixinsight "Separate CFA flat scaling factors". This combined with the Adam Block method gives a much better result. Left image is using standard settings in PI. Right image is with scaling box unticked and master flat with 0.02 value added. I could probably experiment more with the values and try and get a better result but this is a big improvement.

 

 

 

 

 

[newbie alert]

20 sec flat sub??

Surely it's going to massively overexpose that it wont show dust bunnies 

With my DSLR I used the av setting 

[david_taurus83]

20 minutes ago, newbie alert said:

20 sec flat sub??

Surely it's going to massively overexpose that it wont show dust bunnies 

With my DSLR I used the av setting 

I aim for 2 to 3 second flats and 8K ADU. They calibrate out the dust bunnies but over correcting the edge.

The 20s subs above were just lights taken against the flat panel in an experiment to try and calibrate.

[newbie alert]

2 hours ago, david_taurus83 said:

I aim for 2 to 3 second flats and 8K ADU. They calibrate out the dust bunnies but over correcting the edge.

The 20s subs above were just lights taken against the flat panel in an experiment to try and calibrate.

What range does  your camera have in terms of ADU 

[david_taurus83]

Just now, newbie alert said:

What range does  your camera have in terms of ADU 

14 bit so 16k

[newbie alert]

On 21/07/2021 at 20:27, david_taurus83 said:

14 bit so 16k

So  the flats are 50% of the camera range?

I'd try a 3rd

[david_taurus83]

4 hours ago, newbie alert said:

So  the flats are 50% of the camera range?

I'd try a 3rd

You got me thinking again about this. 50% always worked before so why is it different now? Facepalm moment! I had always used my Artesky flats panel in the past but for the above images I had tried laptop screen, mobile phone and a tracing pad, all because they are portable and the Artesky requires 12v supply. So I shot a new set of flats today with the Artesky. The results are very interesting. The colour has changed dramatically. The bottom 2 images are after using the Artesky master flat, the left one with the CFA Scaling box unticked and the right one with the default Pixinsight settings. I cant believe how much of a difference to the colour the choice of flats make though.

 

And below is a mild iteration of DBE on image 8 (Artesky flats, default PI calibration)

 

So it looks like OSC, or my DSLR anyways, is very fussy as to the light source for flats. Looks like I need to lug a 12v supply with me in future and the Artesky panel..