Walking Noise is coming from my flats!

[Stub Mandrel]

Please excuse the lack of images to back this up, but please take me on trust!

My images from my last two sessions have had some 'walking noise' - not lines of hot pixels, but a 'brushed' appearance to the dark background.

With the two worst offenders across two lights I found the culprit - my flats!

No flats I get a gradient, but no 'walking noise'.

I think I have found the reason to.

The flats were taken uncooled while the lights were taken with extreme cooling.

My theory is that the flats have residual noise not in the cooled darks and drift due to imperfect PA/tracking (only the first set of lights were guided) has walked this added in noise across the frame.

A stopgap solution I will try is to 'denoise' my master flat or blur it by about five pixels (the noise streaks are typically 2-3 pixels across).

The ideal solution is awkward but should be possible to implement - recreating my setup and taking cooled flats - won't be perfect but should be better.

[Thalestris24]

Hmm...

I'd have thought if you take sufficient flats (30 or so) and stack them with kappa-sigma clipping then there shouldn't be any such noise in the master flat as you describe? Um, probably with a cooled camera you probably need to take flats at around the same temperature - I think! - I would!

Louise

[vlaiv]

Yes, improper calibration of lights + slight shift between frames would cause walking noise.

There seem to be hot pixels not corrected by flat darks (difference in temperature) that cause "over correction" of lights - you can check this if your walking noise is darker than "ambient'.

There is one thing that you might try before attempting to redo flats - do flat darks at approximately same temperature as original flats and of same duration (btw, don't use the same darks you use with lights for flats unless you have same exposure length for both flats and lights - if you have any sort of amp glow it will have an impact on calibration).

[spillage]

I had massive issue with my flats until I included some dark flats then it was all okay. I also found that using normal darks caused more issues than it solved.

[Stub Mandrel]

I've done somes dark flats and will see what happens

[Stub Mandrel]

Oh dearie! Dearie! Woe is me!

These are two tiny clip-out from 58mm wide field images of the whole orion area.

The first one uses bias, flats and dark flats, the second one is stacked with only bias. They both have similar stretches, in fact eh no-flats one is stretched further. I didn't figure darks were much use using a cooled DSLR when its -4C already.

It's indisputable that the flats are ruining my beautiful data... The pattern does show a little on the no-flat version, so I think the origin is as 'walking noise' but you can see the no-flat version is showing far more detail with less noise. (I know there's a saturation difference, reducing it doesn't get rid of the noise).

 

 

[Thalestris24]

Weird! Can you post a master flat (as jpg)? I wonder how you are taking them and whether this is a new problem?

Louise

ps do you dither your lights?

[Stub Mandrel]

Here's a DSS master flat as PNG, typically DSS it is very dark.

I didn't dither the example above, it's unguided (30s subs at 58mm). Guided ones from a few nights ago show the same effect though.

[sharkmelley]

Which camera are you using?  Are your flats well exposed?  i.e. if you are using a DSLR then the histogram should be well across to the right. 

Bear with me here - those walking noise hot pixels must be caused by dark pixels in your flats.  So your flat darks are not matching your flats very well and so subtracting the dark flats is creating those dark pixels in the flats.  But on well exposed flats, the thermal fixed pattern "noise" should not be an issue at all.

Mark

[Stub Mandrel]

Here's my latest 130P-DS flat, stretched to within an inch of its life, again looks pretty even aside from mild vignetting and just that one dark spot.

Hmm Confucious he say man with clean sensor think twice about flats and use Gradient Exterminator instead...

[Thalestris24]

Hmm... that looks much too dark to me. Should be more like a mid grey. Plus it looks full of rectangles??

Louise

ps was referring to the previous dark image

[Thalestris24]

Yeah which camera? I guess the rectangles are pixels? Zoomed in? What is the flat like as a whole?

 

[vlaiv]

That looks quite ok for DSLR - rectangles that we see are just consequence of resampling artifact (image is bigger and there is bayer pattern so when browser reduces image without some sort of filtering that is to be expected).

I'll have a look at flat itself and see if I can spot anything peculiar.

What worries me is that dark you posted - it also shows rectangles and it should not do that - from darks one should not be able to tell if there is bayer pattern in image.

Btw, how did you calibrate those two images exactly (number of calibration subs and procedure)?

What I see in first image is not walking noise, well it's not what I would call walking noise, it happened to me before without using flats - it has something to do with hitting read noise floor and happens on OSC images (might have to do something with used debayer algorithm and read noise to produce such effect).

[vlaiv]

 

5 minutes ago, vlaiv said:

What worries me is that dark you posted - it also shows rectangles and it should not do that - from darks one should not be able to tell if there is bayer pattern in image.

Oh, ok my bad, that was not dark it was unstretched flat.

[Stub Mandrel]

1 minute ago, vlaiv said:

What worries me is that dark you posted - it also shows rectangles and it should not do that - from darks one should not be able to tell if there is bayer pattern in image.

Btw, how did you calibrate those two images exactly (number of calibration subs and procedure)?

What I see in first image is not walking noise, well it's not what I would call walking noise, it happened to me before without using flats - it has something to do with hitting read noise floor and happens on OSC images (might have to do something with used debayer algorithm and read noise to produce such effect).

That's a DSS master flat, not a dark, I just forgot to stretch it. Stretched it looks like the 130P-DS one, but without the vignetting.

72 lights, 21 flats, 19 dark flats, 1 master bias from about 30 frames; procedure was just to feed them to DSS using default settings (I had some really odd results yesterday so reset to defaults).

I do wonder if it's something horrible in my DSS settings.

I'm trying again with set teh dark point to 0 switched off.

 

[Thalestris24]

1 minute ago, vlaiv said:

 

Oh, ok my bad, that was not dark it was unstretched flat.

Yeah - i thought it too dark for a flat

[vlaiv]

Well do excuse this rather poor comparison, but this was my very early effort with rather poor equipment and no processing skills (ST102 F/5 achro and camera used to record was QHY5LIIc - color version)

It is Cresent nebula in case you were wondering . Anyway, just wanted to show that hitting read noise floor can be mistaken for walking noise. Here green is obvious in generated streak pattern. But pattern is not oriented in RA.

It really resembles streak pattern in your first image and I believe source of it is the same.

Now I'm 100% positive I did not use flats back then, but to be honest I have no idea how I did calibration. Only thing I know is that it was uncooled camera.

 

[vlaiv]

It might be some problem with DSS and the way it handles flat files? It is rather strange that master flat from DSS is so dark - that makes me wonder if for some reason DSS is making flat real dark and read noise from flats is actually creating problems somehow.

[Thalestris24]

20 minutes ago, Thalestris24 said:

Yeah - i thought it too dark for a flat

I think I take it back - my master flats are very dark too... I was thinking of the individual flats that don't look so dark. My bad! I think dss converts the files to 32 -bit? so they have a big dynamic range compared to the originals

[Stub Mandrel]

Problem is my DSLR is cooled, here's a ten minute master dark from it taken at an ambient of about 15C:

 

I don't think there is going to be measurable noise on a 30-second image at an ambient temperatuire of -4 + cooling.

[Stub Mandrel]

3 minutes ago, vlaiv said:

It might be some problem with DSS and the way it handles flat files? It is rather strange that master flat from DSS is so dark - that makes me wonder if for some reason DSS is making flat real dark and read noise from flats is actually creating problems somehow.

I am wondering if something is wrong. perhaps if the flat is TOO flat it exaggerates the noise?

[vlaiv]

6 minutes ago, Stub Mandrel said:

I am wondering if something is wrong. perhaps if the flat is TOO flat it exaggerates the noise?

No, that would not be the problem. But I think there is something that you should check. Don't know what is the best way to do that with files from DSLR, if there is any program that can load cr2 files and give you statistics.

Check your bias files and flat dark files - what does histogram look like on each of those (individual subs, and master). What was ISO setting for those? It might be the case that histogram is clipping to the left?

[Stub Mandrel]

hitting the sack now... but I have a few programs I can use, paintshop pro gives more information than PS I will look tomorrow.

[carastro]

I'm probably going to be shot down for saying this, but I have had times when using bias mucks up an image, and quite often I just use lights and flats with no bias.  It's not consistent though, as sometimes using bias as well works OK.

I can't explain it, but just as an exercise, it might be worth stacking, flats and lights leaving everything else out.  I was told to try this at an SGL camp once and removal of the bias cured the problem.

Carole 

 

[wimvb]

2 hours ago, Stub Mandrel said:

The pattern does show a little on the no-flat version, so I think the origin is as 'walking noise' but you can see the no-flat version is showing far more detail with less noise

Have you considered the possibility that your calibration masters are not adding noise, but just altering the signal, and thereby the snr. This would show as an "increase in noise", where there is none, just a decrease in signal.

One instance where calibration frames increase noise (although not here), is dark subtraction. This subtracts the effect of dark current and amp glow, but leaves the associated noise unaltered. The result is a noisier image (decreased S in SNR).

It would be strange indeed if flats add walking noise, because that is the result of improper calibration AND (proper) star alignment. Calibration frames are never aligned, just stacked, so any hot pixels or fixed pattern noise is also stacked on top if each other.

Btw, does anyone know if master flats are scaled in some way before being applied? If not, the end result of calibrating a light frame with a darker vs a brighter flat would be different.

Just wondering.