What kind of noise... and how to fix it?

[glennbech]

I had some spare time last night and tried to rework my image of IC1805 (Heart nebula). I realized that there isn't much more to do without beeing able to tacle the noise that I have given an example of here. The noise is all over the image background. The only way I found to deal with it was to copy the red channel out, do a full hoise reduction on it (in Ps), and blend it back into the original image as a luminosity layer.

I suspected that this is some kind of "banding" noise, but Noel Carboni's actions didn't do a too god job on removing it. Does it help to rotate the image so that the noise pattern is in fact horizontal before running the action?

Other Ideas?

(click to enlarge)

[narrowbandpaul]

does this pattern get worse with higher signals?

it could a form of fixd pattern noise...it scales with signal.

Have you applied a flat field to the image...if not, then do...around 15 flats averaged together should suffice. And make sure the average signal is high enough or the FPN wont show up...as signal of around 5000 should suffice. The pattern will go after flatting, if it is FPN

If you have already applied a flat, then either its not FPN or your flat is poor.

This is my only idea as to this...

Paul

[Tim]

This is the problem I have been having too. Very frustrating. It gets emphasized by stacking thats for sure, and it seems possible to make it worse using certain dark features in DSS.

The red channel is the one that picks it up worst.

For me, flats did not get rid of it. Its a real PITA, as it prevents proper stretching of the image without causing horrible features.

Anyone sitting on the answer?

TJ

[narrowbandpaul]

doe sound like a FPN problem....

this kind og noise scales linearly with signal..so stacking does not give a boost to SNR.

to effectively remove it, you must take a flat field....but the flat field needs to be taken so that the image is in the FPN regime of the photon transfer curve.

I have measured the PRNU pixel response non umiformity for my 300D, and got an answer of 7.7%, meaning the onset of FPN is at a signal level of 1/PRNU^2...approx 170DN...very low...meaning that the SNR is capped by the FPN very early on in the exposure. Not good.

In fact, it can be shown that the max SNR achievable for a given PRNU factor is SNR max= 1/PRNU...or 13.

So the flats will no doubt be taken in the FPN regime (a value of around 10,000DN) ought to be enough for the sensor to be in the FPN. Just make sure that you take enough (20 should do)

A successful flat will remove almost all FPN, yielding shot noise performance (N=sqrt (S)) over the entire dynamic range.

It does sound like a flat field issue, which solves this repeatitive pattern you are seeing

[blinky]

doe sound like a FPN problem....

this kind og noise scales linearly with signal..so stacking does not give a boost to SNR.

to effectively remove it, you must take a flat field....but the flat field needs to be taken so that the image is in the FPN regime of the photon transfer curve.

I have measured the PRNU pixel response non umiformity for my 300D, and got an answer of 7.7%, meaning the onset of FPN is at a signal level of 1/PRNU^2...approx 170DN...very low...meaning that the SNR is capped by the FPN very early on in the exposure. Not good.

In fact, it can be shown that the max SNR achievable for a given PRNU factor is SNR max= 1/PRNU...or 13.

So the flats will no doubt be taken in the FPN regime (a value of around 10,000DN) ought to be enough for the sensor to be in the FPN. Just make sure that you take enough (20 should do)

A successful flat will remove almost all FPN, yielding shot noise performance (N=sqrt (S)) over the entire dynamic range.

It does sound like a flat field issue, which solves this repeatitive pattern you are seeing

:scratch: :scratch: :scratch: :scratch: :) :scratch: :scratch: :scratch:

Can you de techie it a bit? :withstupid:

[beamish]

yeah ditto :scratch:

[RobH]

Hi guys,

I get rid of this sort of thing in a particular colour channel by selecting the stars etc, inverting the selection, and then applying neatimage pretty aggressively until it's no longer a problem.

As I image using filters, and my detail is in the luminance layer, I find that in practice this doesn't affect the final image.

I've seen this sort of patterning in the past when using my old uncooled DSI and not having correctly temperature calibrated darks....2 degrees difference with the DSI would create all sorts of trouble. Don't know if that's what it is though, Paul knows a lot more than me about the technicalities.

Cheers

Rob

[narrowbandpaul]

here goes...

there are several key noise factors in CCd imaging...read noise comes from the electronics used to convert the charge in the pixel to a voltage and then a digital number (DN) or ADU if you prefer. . There are a certain number of electrons corrseponding to 1DN, this number, called the gain, converts nmber of electrons to ADU (or DN) and vice versa. It has units of electrons/DN. Read noise is invariant of exposure length. it is only the noise added by reading out the detector.

Dark Signal comes from the generation of electrons within the silicon, even when no light falls on the CCD. the rate of emission depends on the temperature. Hence why CCD's are cooled. Dark signal is not the same as noise! Noise is related to the signal. think of it this way...during an exposure some dark electrons are generated. This yields a varying amount of electrons in each pixel, since this rate is actually random. Over the entire CCD there will be an average value, which could be removed without a dark frame. This average number of electrons (equal to the dark current rate x length of exposure) is the signal. But, because the generation rate is random, even after removing the average signal, some variation will be seen. This is obvious because some pixels had fewer than average electrons, and some had more. Think of the noise as the variation in the number of electrons.

The noise then, is not the average amount, but the VARIATION in number of electrons. This is true for all noise. What is interesting is the relationship that noise has with signal.

For dark noise, we find it is governed by Poissonian Statistics, which says that the noise is proportional to the square rot of signal..ie N_dark= sqrt (S), whre S is the average number of electrons generated.

For read noise we found that the noise was independant of signal

Another type of noise is Shot Noise...this is the absolute best you can hope to achieve. Shot noise arises due to poisson statistics. it is the noise present due to counting or recording photons from an astronomical object. so N_shot= sqrt (S); where S is the average number of electrons generated from photons coming from the object. Clearly this cannot be bettered, since we always record light from things. When we use dark frames and bias and flats this is what we are trying to achieve...shot noise performance

but there is one source of noise, often overlooked, that kills the image...Fixed pattern Noise, or FPN.

FPN is a variance of sensitivity from pixel to pixel..resulting in a deviation from a constant signal. And deviation about an average signal creates....NOISE! FPN is actually a charge collection problem, not sensitivity? (I will read this soon in janesick's Scientific Charge Coupled Devices). So how does the noise vary with signal?...not poisson, not independant...but linearly...Double the signal...double the noise. Infact the noise is a constant times the signal, ie

N_FPN= const*(S), this constant is called the Pixel response non uniformity (PRNU), and describes the extent of the fixed pater noise. Typical values are a few %

Lets put this in to context.

One number defines the quality of an image. It is the signal to noise ratio (SNR)...the higher the better. If we have a perfect detector, operating with shot noise performance (the best) we expect a curve of S^0.5

What happens when we add some read and dark noise (but no FPN yet)

not much, the read noise slightly limits the SNR for small signals (faint objects), but shot noise dominates for large signals (which is good).

Add some FPN now, say 10% (or 0.1) PRNU, and bad things happen...

The SNR looks the same for low signals (limited a bit by the read noise (the only major factor for small signals)), but going to larger signals, and the SNR plateaus and reaches a steady value...no longer shot noise performance (very bad)....this means (the key realisation) although you may make longer exposures and take more of them...you CANNOT increase the SNR (quality) of an image. This is purely because the noise is linearly proportional to the signal.

So to achieve the highest possible SNR, we need to remove the Fixed pattern Noise...done by using a flat field...

A brief detour in to the Photon transfer curve....

This is a very useful tool in quantifying importance performance parameters such as gain (e-/DN) and full well capacity. it also tells us the PRNU factor, and the switching point, where shot noise bows down and dreaded FPN takes over.

A PTC is a plot of Noise (RMS) vs Average Signal...

RMS is the measure of deviation from the average value. Its defined as the sqrt ( average of the squares- the square of the average)

What you do is this.

Find a evenly illuminated wall (dont use a lens...ie dont focus the image...blurred is better), take a series of images, from very faint to saturated (completely white).

Take a series of bias frame (the shortest exp possible), average them and subtract from each light image.

Now, for each image, define a small box (say 40x40) and within this box work out the average value of the pixels (DN) and the RMS (noise). Use the ame box each time. it may take around 40 images to go from faint to over exposed.

Plot these numbers on a log Y vs log X graph (easy to do in Excel). There should be 4 regions...

1) A slope of 0. This is the read noise. It is independant of signal

2) A slope of 0.5. This is shot noise. It varies as sqrt (S)

3) A slope of 1. This is the FPN. It varies in proportion with S

4) Full Well. There will be a sharp turning pont. As the pixels get full, they more or less have the same value, so for the same signal, the noise drops dramtically.

In terms of flat fielding, the 2nd and 3rd regimes give us useful data.

Firstly, the point at which FPN dominates over shot noise is defined as 1/PRNU^2. So measure the value of this turning point on the graph, and you will find the PRNU factor.

To remove FPN, your flat field will need to be in the FPN part of the graph.

In practice, this is actually easy, because most cameras have a value of around 7%, yielding a turning point of 204. This is actually a low number...flat field will be taken much higher than this. But if you had a very good sensor, then the switching point would be far higher, seveal thousand.

So you need to take around 20 flats, each exposed such that they are in the FPN regime.

If you go through the equation of SNR, and let the signal be very large, it is shown that the limiting SNR when a pixel response non uniformity is present is given by..SNR max= 1/PRNU...for 0.07, this equates to max of 14. No matter how many exposures i take...my max quality is 14.

You can check if your flats are working, by doing another PTC, with flat fielded exposures...ie apply the flat field to your original set.

The result should be, more or less shot noise (gradient of 0.5) over most of the dynamic range (faint to full well, or ovr exposed)

This hopefully explains the ideas behind flat fielding, and noise sources.

If you do go ahead and make a PTC for your camera, also plot signal/noise vs signal...and observe the plateau of SNR...this is due to the FPN. As mentioned earlier, SNR max= 1/PRNU

I have done this for my canon 300D, and the results so far are...

Full well =2400 e- (a bit low, but i think its because of high iso)

PRNU =0.077=7.7%

Hope this explains flat fields

paul

[Psychobilly]

Yeah its what Paul said.....

I think thats one to print out and read... and re-read... :scratch: and mark up.... :scratch: and still be in the dark... :scratch: :withstupid: or dark flat.... :scratch: :withstupid: :crybaby:

Respect though Paul for your knowledge on the subject...

[narrowbandpaul]

thanks psycho...a lot of this I learned from richard crisp, and Janesick... both of whom have taught me lots

I hope my explanation was coherent

[Psychobilly]

Hi Paul,

flats are the next thing i need to workj on and your thourough explanation may help me avoid soem of the pitfalls...

Saying that the Megrez 72 with the FF III is pretty bloomin flat....

Billy...

[narrowbandpaul]

hi rob...

in practice a two degree difference is actually significant. Normally the noise halves for a 7 degree drop in temperature, but as you cool the camera, the dark current doubling temerature drops, so a small difference is even more amplified.

A two degree difference might provide 1/3 as much/ or less depending on whether the camera is warmer or cooler.

If its warmer, then the dark has more noise (since more signal) than your lights, and the lights are over corrected. This actually adds more noise, since now the variations are the other side of the average.

If its cooler then not enough variation is removed, and image has residual noise.

But i am not sure how that would create a fixed pattern problem. Yes the image would still have noise, but it is noise that can be reduced by stacking.

I am not sure that your problem is a fixed pattern issue, but I may be wrong...

Dont think that helped much

Paul

[narrowbandpaul]

ahhhh, but is your camera....remember FPN may be quite high (my SLR was)

yes hopefully that explanation will help people avoid some pitfalls

[Psychobilly]

Pity my trial of CCDinspector has expired i was hoping to have analysed the first light images from the Megrez 72 FF III combo...

Whats your recommened method for taking low noise flats then Paul?

[winfij]

Hi Glen,

Were these subs guided?

Does the target drift across the field from one sub to the next - i.e. if you were to stack the subs without aligning them, would the result be smeared out in the same diagonal direction as the noise pattern?

Assuming the answer is yes, then this is a fairly common problem - the fixed pattern noise in the pixels is spread out once you align the subs to compensate for the drift, resulting in these short streaks.

If the subs were perfectly guided such that you could stack them without aligning and they all lined up, the noise would be far less noticeable - it's just that the brain is better at picking out the extended lines compared to individual (apparently random) pixel noise.

Reducing the amount of alignment correction needed during stacking (by tighter guiding) is a good way to reduce the effect, but it's easier said than done...

John

[Tim]

i too think it is a problem with slightly off alignment, causing the streaks in one direction. Im hoping to have fixed that with drift aligning, will soon know

If the whole stack was long enough, the streaks would show slight curves form field rotation I would think?

[Tim]

Peter, btw, the flats are dead easy to do.

Just switch your cam to av mode, use the iso you are using, and point at an evenly illuminated area. I usually take 5 , maybe 20 would be better.

It gets rid of vignetting, and provides better contrast in the image, you'll also want bias frames too, so just set the cam to 1/4000th and take a few of them!!

[narrowbandpaul]

you can take darks for your flats too, just as normal.

But take a few, so you have esxcellent noise control

[Psychobilly]

Thansk TJ.

I actually used the cloudy parts of Friday night to take some mid sequence darks and bias frames I normally use "library" ones and i think its made a difference i did this as the temperture was a lot different fron the library set.

I have made a flats target out of two layers of a white cotton T-shirt stretched across a either side of a 1" thick wooden frame. I hang this on the wall in the obs and take pics of it.

I move the scope slightly between shots just incase i pick up a pattern - I know it shouldnt as the scoeps focussed at infinity and the targets only a matter of a few feet from the scope.

Paul - thansk for the info - I have only used dark flats once - it took me a while to get my head around them....

Billy...

[narrowbandpaul]

the dithering technique should work well...do you median combine?

[Psychobilly]

Hi Paul,

Yes I median combine.

The inside of the observatory is white so witht he doors an aperture oupe i getr fairly even illuminaion of the "target".

My source of the latest generation Electoluminescent materials still hasnt "delivered" that looked fairly promising for backlightign the target for night use.

Billy...