Flat Fields and OSC cameras

[roundycat]

it's clear here, not surprisingly as I'm just down the road. M109 tonight as I have just this afternoon mounted my LX.

Just for the record and at risk of becoming the bore of the year with this SD has nothing to do with gaussian distribution. It's just that it is often used as a measure of how far the individual values in a data set vary from the mean when the individual values fit a gaussian distribution.

dennis

[narrowbandpaul]

roundycat,

I personally dont share your view on flat fields.

see http://www.narrowbandimaging.com/images/ptc_talk_wsp_2009_crisp_final_comments_web.pdf

taken from Richard Crisps site. This was presented at the winter star party in the Florida Keys recently.

In particular note page 21 and page 22.

Page 21 states that the S/N of a image with contrast, like an astro image, is equal to the modulation constant multiplied by the S/N of a flat field taken at the same signal level.

Thus a high S/N is required for a good final S/N. Since S/N is proportional to sqrt(S) increasing the signal level will increase the SNR of the flat.

Page 22, defines a quantity known as Flat Field quality Factor, Qff. This is equal to the number of flat fiels averaged multiplied by the average signal of each flat...ie the total signal in the set of flats.

So both these points state that a higher signal level in the flats will provide better performance....

But the reason that a flat must be of high enough signal level....

Flats are used to remove Fixed Pattern Noise.

This noise grows linearly with signal, and limits your signal to noise.

FPN = P*S

where S is the signal....and P is the PRNU photresponse nonuniformity.

From Janesick...SNR max = 1/P

and the onset of FPN, ie where the ideal case of shot noise yields to this dreaded FPN is 1/P^2

P for a CCD is typically about 1%...meaning that the FPN takes over at about 10,000DN. Your flats must be higher than this otherwise they will not show any FPN that you want removed.

I would say 50% is an absolute minimum for flat fields. The more signal in your flats, the better the FPN is removed....

The more signal, the SNR you will have in your flats and so the better SNR you have in your final image.

High signal flats are essential for good data reduction.

I learnt this from Richard Crisp and from reading James Janesick, the father of the CCD.

Thats my view on flats.

[Psychobilly]

Cheers Ally and Paul for your input... So for "DSLR" flats would i be better adding extra stops of exposure compensation to increase the "signal" level... making sure form the histogram that I am not highlight clipping at all ?

Billy...

[roundycat]

Paul, just a quick note about your comments above, but first to lay down the ghost of standard deviation from one of 2403's comments. Of course I use SD all the time, I sometimes think I worry about it more than if the image is in focus or not! I was probably mis-reading the original comment about SD and thinking it referred to the whole frame as for the average ADU value.

Your note above about not sharing my view on flats I found confusing. I read it several times and also twice skimmed the excellent article that you provided a link to.

Apart from the statement I made about 30-50% of full well I struggle to find any difference between us. I have seen 30-50% widely quoted and it allows for two things, one, to get above the read noise and two, to avoid any non-linearity in the CCD. I usually aim at 25-30,000 average ADU for my flats with my ST10, a NABG camera. The last lot I did were all about 28,000 and work a treat. Using the quality factor argument any one of my flats would do the job if 25,000DN is the criteria. I usually take about 20 which has always proved to be enough.

Dennis

[narrowbandpaul]

billy, you are welcome. Anyone showing an interest in the CCD technical side is alright in my book.

roundycat,

i too worry about SD. I dont think its fair to assign a standard deviation to a spread of numbers not specifically governed by a gaussian distribution.

You are right the article is excellent! and let it be said Richard Crisp is a legend

Basically yes, the quality factor must be high, so taking 20 odd frames at a half signal level would be good.

20*25000 = 500,000DN for a Qff in DN. That ought to be a good number.

But one important point, which wont apply to your Qff of a half million....

and that is if you use too low a signal in the flats and/or not enough are combined together, then apply the flat, you can add noise to the resultant image...which is not what you want.

this point was also made in that WSP presentation....

cheers

paul

[roundycat]

Paul, the problem with statistics is that it is the maths of the indefinite. You can find the SD of any data set but the convention of one, two and three sigma limits taking up 68%, 95% and 99.7% only applies to a gaussian distribution. Playing around with a crop of a nebula background with hardly any modulation I found that the 625 pixels that I was working with fitted a gaussian curve quite well. Hence the statistical solutions are fairly safe with our type of data. But remember what Disraeli was alleged to have said about lies, damned lies and statistics!

Dennis

[ngc2403]

one of my favourite quotes

I was not meaning to question your knowledge of SD only that i is the measure of noise for a PTC, which i feel is the best way of measuring a camera's quality.

I know that the minimum limit is 25000DN on the Quality factor but the upper limit if infinite and to really get rid of all the FPN (fixed pattern noise) you might want to try more than 2.5million for a +90% removal of the FPN

not the true numbers (i forget them) but i know that it will not be far off.

Billy your statement is correct almost, i would make sure that the histogram's main body is not higher than 50000 just incase it is nonlinear above that, a few hot pixels that are saturated will not be a problem though.

If you want to see the noise that you are trying to remove, average lots (+100) of flats together and then up the contrast and stretch them. you will see a nonflat image and that is the FPN. it can't be averaged out hence why you need to do flats

What do the rest of you think of the cropped flat? seems odd to me? like you have a narrowband filter over the chip????

[centroid]

Ok, to 'round this one off', as I started the 'thread', I can can report that my problem is resolved.

Despite what the 'books' say, for whatever reason, with my H9C images, subtracting a 'flat dark' from a 'flat', messes up the colour data.

The 'books' say that a 'flat dark' contains the necessary 'bias' data, and therefore, subtracting a 'bias' frame isn't necessary. Well, may be that is so in some cases, I don't know, but it sure isn't the case with the H9C images.

If I subtract a 'bias' from the 'flat', then when the calibrated 'flat' is applied to the image, the end result is fine.

My 'flats' with an average of 26000 ADU across the frame, do the job very nicely, and my recently 'posted' M65 image had 'flat' applied to it.

20 x 0.35s Flats (26000 ADU) median combined, minus 20 median combined bias frames, did the job perfectly.

Dave

[steppenwolf]

Hi Dave, just found this thread. I have used FLATS successfully on both my H9C and M25C but what a pain taking them is! I believe that your 26,000 target is about right for the H9C.

The key to use of a flat frame with an OSC is that you must destroy the Bayer matrix data in the flat. This can be done by applying a Gaussian blur or using a 'Boxcar' filter if your processing software has one (MaximDL does have).

I would suggest that you don't dark frame subtract the flat, just bias subtract it.

[centroid]

Hi Steve

Yep, the colour info does need to be blurred or removed, which I forgot to mention, and as you say, no 'darks', just 'bias' subtract the 'flats', and away you go

I went down the subtracting 'dark flats' from the 'flats' route, because that's what the "book" suggested, but it didn't work.

As for taking them, it took me only a few minutes when I finished imaging last night's M65.

I made a 'white board' from a 2' x 2' piece of hardboard, painted on the rough side, with matt white emulsion.

The light source comprises, 3 x 9" strip lights, in box, with an opaque whie perspex difuser on the front. I added a dimmer swith on the outside, so that I can control the intensity of the light.

When I've finished a 'session' the scope goes into its 'park' position, facing the obsy wall. The 'white board' hangs on the wall, and the light box attaches to the front of the wedge, angled up towards the white board.

With the 6.3 FR on the 12" LX, 0.35 seconds exposure gives me around 26000 ADU average across the frame.

So, taking a couple of dozen 0.35s 'flats', and the same number of 0.001 second 'bias' frames, takes very little time at all.

Dave

[SteveL]

dark flats are only really useful when the length of your normal flats gets into the several seconds region, otherwise the difference between dark flats and bias frames is negligable.

Also as for OSC flats needing to have the boxcar filter applied, its been a long running discussion on the DSS list about this. I cant speak for Maxim, but as DSS stacks each channel seperately (along with the relevant channel for the flats/darks/bias), then the master flat is normalised on a per channel basis. So long as each channel is between 1/3 to 2/3 of max value, you should have any issues.

I argued for ages about having a boxcar filter option in DSS, only find that the issue I was having with flats was actually being caused by the f/2 light cone angle on the hyperstar causing colour shift. The moment I started taking decent quality image data with anything else, the issues disappeared, and I never really needed the boxcar filter.

(Your mileage may vary, I am just stating the issues or non-issues I have had)

[SteveL]

...When I've finished a 'session' the scope goes into its 'park' position, facing the obsy wall. The 'white board' hangs on the wall, and the light box attaches to the front of the wedge, angled up towards the white board.

I`d still be interested in seeing one of those RAW FITs flats. Illuminating a white surface at an angle from a close distance could give you a gradient in your flats.

[centroid]

I`d still be interested in seeing one of those RAW FITs flats. Illuminating a white surface at an angle from a close distance could give you a gradient in your flats.

PM me your e-mail Steve, and I'll send you one of those that I took last night.

Dave

[centroid]

In the mean time, here's a FIT, to Tiff, to JPEG version of one, resized down.

In going from FIT to TIFF, it showed a 'grid' pattern (not seen in the original FIT), so I applied a mild guassian blur to it in p/shop, to remove it.

As you can see, there's one 'dust mote', but no gradient from my angled light source.

Dave

[SteveL]

Just recived the flat from Dave (thank you!). I have debayered it both in Maxim and in FitsWorks, and the only way that doesnt leave a debayer grid pattern in the final image is RGGB (fitsworks) and x-y offset enabled in Maxim, so thats what I am using here. If this is wrong, please let me know.

Anyway, before debayering, the raw flat is showing some oddities already. Three very distinct peaks in the histogram.

When debayered, its incredibly red

And creating a pixel line diagram in fitsworks shows what those three peaks are

What filters (if any) were you using for your imaging session and flat taking?

This is the second Sony 285 based images I`ve seen this week that are incredibly red, so I am not 100% sure that I am using the correct debayer method to convert the raw image into an RGB image, but as the debayer option is the only one that doesnt leave a grid in the final image, and at least with this flats field, the three peaks that appear in the original raw appear as the same three peaks in the RGB version.

[roundycat]

is that colour response merely mimicking the spectrum of the light source for the flats? I use tungsten light with a mono camera and my red flat exposure is typically 2 seconds whilst that for blue is 8-10 seconds.

Dennis

[SteveL]

Thats a good point.... Dave, what light source are you using to illuminate your dome flats?

[centroid]

Ok, just got back to this one. Busy day, cut the grass (and there's a lot of it in my garden ), washed the car, tidied up the garage, and about half a dozen odd jobs that the missus has been nagging me about for a while .

Right, my light source is three 9" tungsten strip lights, dimmer controlled, and behind a 6mm opaque white sheet of perspex.

As you know, this is shone at a matt white board, with no other ambient light in the obsy.

Following the recommendation for H9C flats, they are around 26000 ADU average.

As I said, the M65 image subs, had these flats successfully applied to them, after they were bias subtracted. The result, as you can see for yourself, is an nice yellow based galaxy, which I reckon is about right.

Ok' must 'fly', dinner is on the table

Dave

[ngc2403]

glad to hear that the problem is fixed,

perhaps you could take shorter exposures and use a stretch on the colour channels to level them?

strip lights are not great for giving broad emission,

much better to use simple filament bulbs, they give an ok Blackbody spectrum, filaments tend to be full of bright emission lines.

A blurring of the image will not give the correct results for a flat field. You need to produce an image possibly in each colour to properly emove the FPN on the camera, however they are fine for vingetting and dust nodes.

[roundycat]

just another thought Dave, as you are using tungsten filament lights, not flourescent, have you had to turn the voltage down to get the right kind of exposure duration. If you have the light will be even richer in red compared to blue.

Dennis

[centroid]

The white board is surprisingly bright when illuminated Dennis, so I wouldn't say that its particularly low level.

However, its only guess work, as set it at what I thought to be, neither too bright or too dim. As it is, 0.35s exposure gives me the desired 26000 ADU.

The 'bottom line' is, the 'flats' were very effective on the image, they removed the 'dust bunnie', and did nothing to upset the final colour.

Once I changed from subtracting 'flat darks' from the 'flats', to subtracting a 'bias', it all came right.

Dave

[ngc2403]

0.35s exposure gives me the desired 26000 ADU.

Dave

the flats you took are not at 26000DN, the average of the RGB is at 26000DN and looking at the stuff the SteveL posted the red channel is too high for a Flat, i.e. you are causing the red channel have extra noise when you use it unblurred.

Am not sure that there is an easy fix for this, only that you might have too just use the blurred flats you get.

[centroid]

I do indeed blur the flats to destroy/degrade the colour info, before applying them to the image subs.

Dave

[centroid]

Following on from a similar discussion I'd been having on the Astroart forum, here's a response from Terry Platt, on the need to apply a 'boxcar' filter, to OSC 'flats':

The problem with single-shot colour flats is that an unprocessed flat will be

composed of pixels which vary greatly in brightness, owing to the Bayer filter

matrix. In a normal mono camera, a flat field is used to remove such

pixel-to-pixel variations, along with vignetting and dust motes, but this would

destroy the colour data in the raw frame from a one-shot camera and so it must

not be allowed to happen. The solution is to remove all of the small-scale

variation in the flat field by applying a suitable low pass filter before it is

used to calibrate the raw colour images. The 2x2 'boxcar' is ideal, as it

matches exactly the nature of the matrix, but other low pass filters will work

to varying degrees. Unfortunately, none of the alternatives will be quite as

good as the boxcar and they will reduce the effectiveness of the flat field for

removing dust motes etc. by distorting the definition of these larger features.

As usual, Terry is very responsive to questions, and speaks from a wealth of experience.

Dave