Flat Fields and OSC cameras

[centroid]

I have never bothered with flat fields, as I don't have vignetting problems, and keep the camera, and back-end optics pretty clean.

Up to now, I've been lucky with dust-bunnies, in that if I do get the odd one, its small and usually falls on the sky background, and so is easily 'clone stamped' out.

However, out of interest, I thhought I'd have play with some 'flats'.

I made a 2'x2' 'white board', from hardboard, painted with matt white emulsion on the rough side, and made a light source, using three 9" strip lights, in a box, with an opaque white perspex difuser. I included a 'dimmer switch' so that the intensity of the light could be controlled.

The 'white board' was hung on the obsy wall, and the light arranged so that it evenly illuminated the 'white board'.

From there on, it was easy to capture nice even 'flats', at the desired 26,000 ADU level (recommended for the H9C), by adjusting the exposure time to suit.

With the light source at a reasonable level, an exposure time of 0.35s, produced a 'flat' of a little over 26,000 ADU. With just the one small 'dust mote', that I saw on the image sub, showing very clearly.

Next, I put the lens cap on the scope, and took a 'flat-dark', at the same 0.35s exposure. I actually took 10 flats and 10 flat-darks.

It is generally recommended that, 'flat field' should be 'bias' subtracted.

This can be done with either a bias frame, of a 'flat dark', which contains the same bias data.

The image subs, flats and flat darks, were brought into AA4, where the flats were flat dark subtracted, and then subtracted from the image subs.

The resulting un-debayered image looked fine, with no sign of the dust mote. So far, so good.

However, when the image subs were then deybayered, I found that the colour was completely 'cocked up'.

What should have been a nice blue and mauve 'ish' Eskimo, was now a bright orange one .

I went back to start, this time taking a single image sub, and subtracting from it, a single flat. Debayering this, produced the same bright orange Eskimo.

Back to the start again, taking a fresh image sub, and a single flat and flat dark. Again, the debayered result, was a bright oarnge Eskimo.

Repeating the whole sequence, using alternative processing software, produced exactly the same result.

So, for some obscure reason, it would seem that 'flat-fielding' an OSC image, does strange things with the colour information.

The next question, is of course, have anyone of you guys and gals, tried 'flat fielding' an OSC image ?

Dave

[Psychobilly]

Flatfielding my DSLR's works fine... and they are OSC's so this is "Wierd" ...

Good luck trying to sort it out Dave

I need to get something sorted for the 8" as one thing that last nights lentil souper showed up was how bad the vignetting is on the 8" at f6.3 with the DSLR's I like the ideal of the light panel...

Billy...

[centroid]

Flatfielding my DSLR's works fine... and they are OSC's so this is "Wierd" ...

.

Do you capture you images as RAW non-debayered subs, or as 'ready made' colour images?

Dave

[Psychobilly]

Hi Dave,

Everything is "Raw" the Lighs, Darks, Bias, Flats and I just chuck them all into DSS and let it sort it all out...

Even if there is "colour" in the flats - like when you take sky flats - this should be ignored in the processing by the software...

Billy...

[centroid]

Is that RAW, in the sense of unprocessed, but in colour.

Or, are you 'de-bayering' to extract the colour info in DSS?

With a OSC astro camera, the RAW image is displayed as in mono, and the colour data has to be extracted by a debayering algorithm.

Dave

[Psychobilly]

I suppose DSS does the debayering...as you can select various options for which method it is going to use... I normally go with the one it recommends based on its "analysis" of the lights..

Billy...

[centroid]

It will be interesting to see if anyone on the forum, has 'flat-fielded' a Sony 285 CCD based OSC camera image.

Steve (steppenwolf) has both an H9C and a 25C, so hopefully Steve will happen upon this 'thread'.

Dave

[ngc2403]

Did you get the Flats info from terry at Starlight express?

Flat fielding theory states that you should expose to get close to full well ~40000ADU is a good number.

What is the mean ADU on an over exposed image? Using the SXV H16 just now i have found that the full well is ~50000ADU this is a gain problem.

Anyway the bias image that you need is the shortest possible dark image you can take, and i would suggest stacking more than 20 of them 100 is a good number. Once you get them averaged together, stretch the image to see if there is a pattern? also compare the mean value on that to the mean on the flat-darks at 0.35sec you had before might be them that caused the problems

hope this all helps, and if your wondering why i know it, then its because i am doing this stuff for a project at the moment so it's all fresh in my mind

ally

[SteveL]

Flats should be taken so that the ADU count is at or just below the point where the CCD response starts to become non-linear, otherwise your flats are not really flat. This is not always (very rarely) at full well capacity.

[ngc2403]

true they don't remove the fixed pattern noise that they are meant to, but they also have to be out of the shot noise region of the camera other wise they can add noise to the image not remove it.

i think that the flats are under exposed and are the cause of the bad colours. you see they devide the image so if they are noisly they make the image really bad and if they are lower in one channel than in others it gets a big boost in intensity, hence orange colour

[Psychobilly]

This is very interesting ... I have been taking flats at 50% with DSLR's seem to remember reading it somewhere... this is teh text from the DSS help files

-----------

Flat Frames

The Flat Frames are used to correct the vignetting and uneven field illumination created by dust or smudges in your optical train.

To create good flat frames it is very important to not remove your camera from your telescope before taking them (including not changing the focus).

You can use a lot of different methods (including using a flatbox) but I found that the simplest way is to put a white T shirt in front of your telescope and smooth out the folds. Then shoot something luminous (a flash, a bright white light, the sky at dawn...) and let the camera decide of the exposure time (Av mode),

The flat frames should be created with the ISO speed of the light frames. The temperature is not important.

----------

Using Av mode (Aperture Priority) the camera will expose to a 50% grey level...

Hmmm Guess its different for "real" CCD's

Billy...

[ngc2403]

thats the method i use for a long time and i worked for the DSLR because it is low noise and i think might correct for pixel fixed pattern noise on the camera??? anyway the best results are got by averaging together lots of images to reduce the random noise in each image and bring out the fixed noise to be used in a flat field.

the larger the signal level the lower the random noise is

A point about the DSLR noise after doing a PTC ( method for noise measurement in Imaging cameras) i have found that my 350D has less fixed pattern noise and a lower read noise than a SXV H16? the DLSR's bias and flats reduce the noise better than the SXV H16s! however the H16 wins in the end because it has far lower dark noise due to the cooling

[Psychobilly]

Cheers for that

I normally take at least 40 flats using my light box (when I rememebr) ... I rotate the lightbox 90 degrees per set of 10 "flats" .. I take anywhere up to 100 bias frames and typically use 50+ off them ..

Its the lights I have problem with... no decent weather to capture anythign worthwhile...

Billy...

Billy...

[centroid]

I can't remember where I read that the recommended ADU level for H9C 'flats', was 26200.

I've 'posted' the issue o the Astroart forum, and Terry Platt monitors this forum, so I might get a difinitive answer.

As I understand it, the ADU level of the 'flat' is measured as a average (Standard Deviation) of the frame, and not not peak level of any particular pixel.

Although the measured average of my 'flats' is around 26000 ADU, I can see 50000+ on indivual pixels.

Most of the 'imaging bibles' say that a 'flat dark' is taken at the same exposure lenght as the 'flat', and that it contains the bias data. Therefore, it isn't necessary to take a separate bias frame.

As I said, earlier, my results are the same, whether a 'flat' of 'dark flat' subtracted 'flat' is applied.

Dave

[SteveL]

i`d be more inclined to take bias frames and not flat darks, especially when your flat exposure times are so short, more so with a sony chip that is very low noise.

If you "peak" values are showing 50000+ then those areas are way past the linear response of the CCD.

[SteveL]

Are you able to upload a light, a flat, a bias and a flat dark somewhere for us to look at?

(And I flat field both my QHY8 and M25C)

[centroid]

Not easily Steve, as the H9C Fits files are quite large, and if I jpeg'd them, the ADU data wouldn't be available.

Dave

P.S. I coud e-mail them to you?

[SteveL]

it would be easier to zip them, upload them to your geocities website somewhere, and PM me a link to them...

[RobH]

Hello Dave.

In Maxim, where you set calibration, there is a tickbox 'apply boxcar filter'.

You need this ticked for OSC flats.

I had the same trouble as you when I tried flats with my Atik 16HRC until I discovered this.

Cheers

Rob

[narrowbandpaul]

to know when the linearity of your sensor drops off you need to make a Linearity Transfer Curve.

This is easily done.

Take a bunch of flats from very dark to full well.

Using a small box, say 60x60 pixels, find the average value of these pixels. Then plot Average signal vs Exposure length.

Using excel makes this easy, you can add linear best fit lines.

It will then be obvious by eye when non linearity kicks in. Keep the sensor illumination just below the nonlinearity bit.

I have done Photon Transfer Analysis for the SXV H9, it has come up with some good results.

I will post this later.

[ngc2403]

Can't see the Quoting thing for this so

Quote from Centroid '' As I understand it, the ADU level of the 'flat' is measured as a average (Standard Deviation) of the frame, and not not peak level of any particular pixel.

Although the measured average of my 'flats' is around 26000 ADU, I can see 50000+ on indivual pixels.''

Some confusion here about ADU and DN

DN is the counts show by a camera, they can be 0 to 65536

ADU is the measure of the noise on the frame? not something to base a flat frame on.

i meant take the flats so the Average DN was 40000DN

the standard devation at this value changes for all cameras dependent on many things, hence it is better to go with 40000DN because like SteveL said if some of the pixels are past linearity then the frame is not a flat and will screw with the image.

as a test take a frame to saturation and then look at the ADU and average DN on that. you might find it interesting as the DN(signal) is fixed the ADU(noise) drops or sky rockets?

Could you crop the same 60X60 box from the middle of the master dark, bias and flat for us to look at?

we don't need the full frame for this

[centroid]

Ok, here's a 'crop' of a single 0.35s flat, and the same for a flat dark

I make the max pixel value of the flat, to be 63396 (Av = 26593) , and for the flat dark, 2053 (Av = 928)

Dave

flat_crop.tif

flat dark_crop.tif

[roundycat]

finally I cannot resist joining in. There seems to be a lot of confusion in peoples minds as to what a flat should do and how it does it. Taking a flat anywhere near full well is wrong. The generally accepted level is around 30-50% of max ADU, that is Analogue to Digital Units or the signal level after quantisation. It has nothing to do with noise as has been wrongly stated above. What DN is supposed to be I have no idea. The ADU can go to full 16 bit values but is modified in the form full well (electrons) multiplied by gain. Full well is measured in electrons, not ADU. My ST10 has an electrons/adu figure of 1.35 so max adu is about 51,000.

Also mentioned above is that ADU is the noise on the frame? Wrong.

The standard deviation value of a flat is largely irrelevant. Just as the SD in a normal picture is irrelevant. This SD value has nothing to do with the camera, it is simply the rms of the variance about the mean pixel value. If you have bright pixels for any reason and dimmer pixel in the corners the SD will be all over the place.

Dave, your mention of the ADU level of the flat being measured as an average (Standard Deviation) is confusing. The two things are quite different, see above. You should measure the flat value as an average of the whole frame although some flat fielding algorithms may take an average over the centre 80% or so of the frame in order not to be skewed too far by dark pixel values in the corners. Maxim's area readout averages the whole lot and is what I use to set flat field exposures.

During flat fielding an average is taken of the flat and each pixel value in the flat is divided by the average value. This gives a co-efficient for each pixel in the light frame. The light frame pixels are divided by this co-efficient to correct the light frame values.

A quieter result for a master flat can be achieved by combining using a sigma reject style algorithm not averaging. Averaging used to be the holy grail of minimum noise but is no longer (look at the math). I routinely shoot 20 flats and subtract bias from each one before combining. My bias frames are in sets of 30 and combined using SD Mask. You may find it useful to run a hot pixel filter on each bias subtracted flat to get rid of the remaining outliers.

Perhaps we could go into this at our next meet as it is bound to be cloudy!!

Dennis

[centroid]

Hi Dennis

What you are saying, ties in with my own thinking, and that is the recommended 26200 ADU for an H9C flat field, is measured as an average of the whole frame, and that the max pixel value is irrelevant.

I do agree that average and Standard Deviation (SD) are different, as SD follows a Guassian Distribution, whereas average does not.

Flat fielding, is definitely and interesting subject for discussion, both for here on the forum, and at the 'local meet'.

Anyway, I've just looked outside, and it looks like its clear, so I'm now going out to the obsy, to see if its worth setting up for some imaging .

Dave

[ngc2403]

?

DN digital number is the industry standard unit for camera counts or ADU as people here seem to be using.

All of my flat fielding theory is taken straight from two books written by the maker of the Hubble wide field planetary camera I and the man how first took CCD's around observatories to show them that they were better than film,

http://www.eurospangroup.com/display.asp?&isb=9780819436986& book one 'Fundamentals of scientific charge-coupled devices'

http://www.alibris.co.uk/search/books/qwork/10586732/used/Photon%20Transfer:%20Dn%20%5BLambda%5D book two 'DN to Lambda'

in here are detail descriptions of how CCD and CMOS chip function and how to remove and characterize all sources of noise.

Flats are a large chapter in the book and inside is the way to properly flat field, which in truth is only possible after having done a PTC or Photon Transfer Curve.

this is done by measuring the Standard Deviation on Flats! largely relevant to finding the Signal level at which the best flats can be taken

taking Flats at a low signal level reduces there quality and causes the need for extra flats to be taken. the average DN of your flats multiplied by the number taken gives the Quality factor. if this is not above 25000DN then you are making your images worse by flat fielding.

sigma reject is not a valid way to scientifically flat field a camera because it will rejects all of the pixels with increased dark noise and all the hot pixels and dead pixels too which can be fixed through a using flats, darks and bias frames.

back too the problem of the orange Eskimo, i don't like the pattern in your flat have you stretched it? it's a black and white checked pattern???

dark looks ok though

do you have a filter on the front of the camera?