Hot pixel/read noise streaks in my stacked images?

[pipnina]

I got some good results with my 8-minute exposure of the andromeda-cassiopeia region  but I wanted to improve on it so I went out another ngith and got 29 minutes worth of data with my DSLR.

For some reason, my intervelometer refuses to take images of the correct exposure (i asked for 15 second subs and got 14 seconds, but it was never exactly 14) so I had to do some fooling for DSS to stack them, but it worked eventually.

The only issue is, that I've realised ALL of my stacked images (from my first stacked image of orion to this new one) have weird noisy sreaks in them (highlighted better by the maximum-stacked version (highest value for each pixel) which actually revealed optential glare from a near by street lamp as well)

At first I thought they were hot pixels running up my image, they might be, but I'm not sure. I would have thought the Auto Adaptive weighted average that I used (or the kappa-sigma I used for the orion image) would have cut them out. Needless to say, it's rather annoying me as they look to be limiting what I can do with my DSLR (which I need to keep being able to use happily until I can amass ~£2500 for a proper imaging setup with a cooled CCD, and that might take a while)

I tried to take some spoof dark frames and use GIMP to remove all but the hot pixels but DSS threw a fit

Another note is that multiplying the integration by 4x didn't have much affect on the noise... I was expecting the difference to be a little more dramatic... But perhaps it's just because the read noise and dark current is being put into the image over and over and stopping integration from lowering the noise further.

New RAW data (29 minute exposure and a new moon) https://dl.dropboxusercontent.com/u/264557087/YEEEEEEEEE16bit.png

Old data (8 mins (also under full moon)) https://dl.dropboxusercontent.com/u/264557087/M31%20old.png

     ~pip

P.S. Don't question my file naming scheme.

[Pompey Monkey]

I'm in a bit of a hurry at the moment, but apparently "dithering" is one of the better methods of reducing this DSLR "pattern" or "walking" noise. APT and Backyard EOS have built in algorithms for doing this with your ascom guided mount. 

[pipnina]

I'm in a bit of a hurry at the moment, but apparently "dithering" is one of the better methods of reducing this DSLR "pattern" or "walking" noise. APT and Backyard EOS have built in algorithms for doing this with your ascom guided mount. 

I didn't know actual mounts could dither, but alas I don't have one... I guess I'll have to try dithering manually.

It's kinda funny, though, I thought I wouldn't need dithering since the earth's rotation sorta dithers for me... 

[sidelight]

HI, Have you tried adding bias shots into the mix?  Between darks and these they should help DSS identify and reduce signal noise.

Cheers

Ross

[pipnina]

HI, Have you tried adding bias shots into the mix?  Between darks and these they should help DSS identify and reduce signal noise.

Cheers

Ross

I'll give those a try, although hot pixels seem to be the issue, how many would you recommend? I've tried using them before and the master bias usually ends up being a couple of pixels that are slightly above black and mostly just black (e.g. no data that wasn't completely random) and that was after 20 shots...

I've also heard that they're temperature dependant.

[kalasinman]

Most DSLRs have read noise that dominates the image at lower ISOs. You shouldn't image below ISO 800 with a Canon.

Your streaks look a lot like star trails. After you stack in DSS after the Auto save is produced, stay right there in DSS and select register images, then cancel. You'll see a list of all your scores. Delete the low ones, then click on each sub and inspect it individually. There may be something bad in one image that's ruining the rest.

Make a dew shield to keep local sources from causing internal reflections.

You seem to have a fair amount of coma. Pick framing that allows a sizable crop.

It would be very helpful when posting and asking for help to always include image information:ISO, exposure time, f/stop, image train.

Just because a dark or bias frame looks dark to you has no bearing on it's usefulness for calibration. The data is still there. If you want to see it, try stretching either one, just don't use stretched ones for stacking.

No, with a barn door you can't dither and if the earth's rotation dithers for you, your barn door isn't working properly.

Imaging absolutely requires a good mount in order to produce presentable images. Before you consider saving for a CCD get a decent mount, implement guiding, and start software controlled dithering.

[Knight of Clear Skies]

Hi, there are a few problems with the images you've posted. If you look at your 8 minute Cassiopeia only the centre of the image has stacked properly. This is because the image from the lens is slightly distorted so the positions of the stars don't match up between frames, so DSS can't align then. Manual dithering isn't going to work with this setup for that reason. (Dithering will happen naturally when using an unguided mount, as over a few subs the image will drift by more than a pixel.)

The Orion image is a good effort from a fixed tripod, you've got some colour and a hint of the Flame Nebula too. The star shapes are a bit out but I'm not sure if this is tracking, coma or both as you haven't posted any exposure details. De-noising in post processing may help, as may dark bias frames. Darks are certainly temperature dependant but I've heard that dark bias aren't.

Stacking improves signal-to-noise ratio but there are diminishing returns for each sub added to the stack. I don't quite understand the math but after about 30 frames any improvement falls off rapidly and can be unnoticeable. As you suspected the problem is the noisy subs, longer subs on a tracking mount is the real solution.

Kappa-sigma clipping mode when stacking is very powerful and great for removing artifacts such as hot pixels and satellite trails.

Hope that's some help.

[pipnina]

Most DSLRs have read noise that dominates the image at lower ISOs. You shouldn't image below ISO 800 with a Canon.

Your streaks look a lot like star trails. After you stack in DSS after the Auto save is produced, stay right there in DSS and select register images, then cancel. You'll see a list of all your scores. Delete the low ones, then click on each sub and inspect it individually. There may be something bad in one image that's ruining the rest.

Make a dew shield to keep local sources from causing internal reflections.

You seem to have a fair amount of coma. Pick framing that allows a sizable crop.

It would be very helpful when posting and asking for help to always include image information:ISO, exposure time, f/stop, image train.

Just because a dark or bias frame looks dark to you has no bearing on it's usefulness for calibration. The data is still there. If you want to see it, try stretching either one, just don't use stretched ones for stacking.

No, with a barn door you can't dither and if the earth's rotation dithers for you, your barn door isn't working properly.

Imaging absolutely requires a good mount in order to produce presentable images. Before you consider saving for a CCD get a decent mount, implement guiding, and start software controlled dithering.

I haven't built the barn door yet, it's WIP

You do make a fair point, I probably should have mentioned that all my images are taken at f-8, usually 35mm but sometimes 18 or 55, (orion 35, 8-min stack 18, 29 min stack 55(I think)) my ISO is always 800 (except Orion, which was 1600ISO, I've moved to 800 since)

My white balance is direct daylight and I have noise reduction off.

Also, duely noted, I'll work towards a good mount before I rush to replace my camera.

Hi, there are a few problems with the images you've posted. If you look at your 8 minute Cassiopeia only the centre of the image has stacked properly. This is because the image from the lens is slightly distorted so the positions of the stars don't match up between frames, so DSS can't align then. Manual dithering isn't going to work with this setup for that reason. (Dithering will happen naturally when using an unguided mount, as over a few subs the image will drift by more than a pixel.)

The Orion image is a good effort from a fixed tripod, you've got some colour and a hint of the Flame Nebula too. The star shapes are a bit out but I'm not sure if this is tracking, coma or both as you haven't posted any exposure details. De-noising in post processing may help, as may dark bias frames. Darks are certainly temperature dependant but I've heard that dark bias aren't.

Stacking improves signal-to-noise ratio but there are diminishing returns for each sub added to the stack. I don't quite understand the math but after about 30 frames any improvement falls off rapidly and can be unnoticeable. As you suspected the problem is the noisy subs, longer subs on a tracking mount is the real solution.

Kappa-sigma clipping mode when stacking is very powerful and great for removing artifacts such as hot pixels and satellite trails.

Hope that's some help.

The straneg thing is that all the reviews of Nikon's distortion control say it completely removes all traces of barrel/pincushion above 35mm on my lens, but no matter what focal length I'm running at it never outputs with all the stars alligned! I'm wondering if distortion control isn't working in RAW files... but I tried stacking with converted TIFFs and the same issue arises.

I think the signal to noise reduction from stacking is logarithmic, so you have to double the frames to halve the noise (meaning to halve the noise from 30 you need to get 30 new frames, and to halve the noise from 60 you'll need to get another 60) hence why I was trying to quadruple my integration.

I've found Kappa-sigma and Auto adaptive weighted average both seem to do a verry good job, and median/average being less good but quicker and maximum being great if you want to see everything wrong with your image.

I might have to do some tests with my distortion control (off vs on in a stack maybe) and maybe some tests with noise reduction as well. (it was advised that I not use it when I started, some saying dark frames made their DSLR images worse, but you never know until you've done a (slightly) scientific study!

Thanks,

    ~pip

[kalasinman]

Another item, your camera. You've never listed what camera you use. You really should add your equipment list to your signature. Specific model and type of DSLR and any lens.

Yes, the idea of RAW or NEF is to bypass settings like distortion control , color balance, etc.

Turn off any sort of in camera noise reduction.

With either Canon or Nikon, some of the kit lenses are rubbish.

Fixed focus, manual lenses are perfect for astro. You can often pick them up used at a good price and are usually better optics than kit zoom lenses. Although Canon has changed lens mounts over the years. leading to potential compatibility issues with older lenses, Nikon has not. Any old F mount lens will work fine for astro, although the advanced features (which you won't need ) may not work.

Exposure is usually used as the length of a single sub. Total integration, the sum of the subs used in the final product.

A useful guide to exposure times to avoid star trails is 400/FL=exposure. This will vary with distance your target is from the Celestial Equator.  So that would be 12 seconds at 35mm.

This doesn't apply if tracking, of course.

[Knight of Clear Skies]

The straneg thing is that all the reviews of Nikon's distortion control say it completely removes all traces of barrel/pincushion above 35mm on my lens, but no matter what focal length I'm running at it never outputs with all the stars alligned! I'm wondering if distortion control isn't working in RAW files... but I tried stacking with converted TIFFs and the same issue arises.

That's interesting, I'd guess the stacking software doesn't understand this information. You could try converting the RAW files to 16 bit TIFFs and stack those, the Nikon software should adjust the image when converting.

F8 is very slow for AP. I've used a variety and stop them down to between f4 and f5.6 depending on the lens when using a tracking mount. When shooting from a fixed tripod I use them wide open or nearly so.

[Scott]

fixed tripod imaging, although a bit limiting can give some great results. I would suggest not going lower than iso 1600 and an f/stop no slower than f/4-f/5.6. (just be sure to keep the histogram at around the first third of the graph)

Try for regions of the milky way that are rich in nebulosity such as cygnus all the way down to sag/sco (or as low as is reasonably possible). whilst on a static tripod, i'd recomend keeping the lens length as short as possible.

After taking your subs (lots) then take flats (20-30). you can use an ipad screen that is whited out (if you're unsure how to do this then google is your friend). make sure NOTHING is changed on the camera for this. focus,settings and temp should be the same as your subs.

After this you can take your darks. focus is unimportant as you will have the lenscap on anyway. just be sure theres no light leakage such as your viewfinder. put a black towel over it or whatever it takes.

dithering is not an option so don't bother trying. also, remember that there is a 15 degree/hr rotation so every 15 mins or so, re-centre your target as best as possible. another big big help is getting to a dark site.

be patient, practice and it'll all come together

clear skies mate

[pipnina]

That's interesting, I'd guess the stacking software doesn't understand this information. You could try converting the RAW files to 16 bit TIFFs and stack those, the Nikon software should adjust the image when converting.

F8 is very slow for AP. I've used a variety and stop them down to between f4 and f5.6 depending on the lens when using a tracking mount. When shooting from a fixed tripod I use them wide open or nearly so.

I had found that wider open than f/8 vignetting becomes more of an issue (although my method of processing doesn't care about that so I might just switch to F3.5-5.6 (been looking at a rpime lens that can go to f1.8 so that might be cool.)

Thing is, even in the TIFFs it still causes the weird stars... I open the tiffs with and without the distortion control and there is a diference. Maybe the distortion control isn't as good as I've heard it is... But I'd heard it corrected for distortions that weren't even barrel/pincushion so i got no idea what's going on.

fixed tripod imaging, although a bit limiting can give some great results. I would suggest not going lower than iso 1600 and an f/stop no slower than f/4-f/5.6.

Try for regions of the milky way that are rich in nebulosity such as cygnus all the way down to sag/sco (or as low as is reasonably possible). After taking your subs (lots) then take flats. you can use an ipad screen that is whited out (if you're unsure how to do this then google is your friend). make sure NOTHING is changed on the camera for this. focus,settings and temp should be the same as your subs. after this you can take your darks. iso,f/stop and temp should be the same but focus is unimportant as you will have the lenscap on anyway. just be sure theres no light leakage such as your viewfinder. put a black towel over it or whatever it takes. dithering is not an option so don't bother trying. also, remember that there is a 15 degree/hr rotation so every 15 mins or so, re-centre your target as best as possible. another big big help is getting to a dark site.

be patient, practice and it'll all come together

clear skies mate

I've been meaning to image the MW but where I am houses are in the way and it just doesn't get dark early enough for my dad to drive me to our "Dark site" (really just means it's the darkest we travel to, plymouth's LP goes further than 25 miles so I can't really get away from it unless it's on a holliday.

Either way, thanks for the tips! clear skies to you, too.

[wxsatuser]

I think the signal to noise reduction from stacking is logarithmic, so you have to double the frames to halve the noise (meaning to halve the noise from 30 you need to get 30 new frames, and to halve the noise from 60 you'll need to get another 60) hence why I was trying to quadruple my integration.

Thanks,

    ~pip

The signal goes up linearly with the exposure.

The noise goes up as the square root of the exposure.

The snr increases as the square root of the exposure.

[lensman57]

Hi,

I have just read your post and you seem to have quite a lot of issues with your imaging. I used to be like yourself so there is light at the end of the tunnel but with AP problem solving you need to approach it systematically. One step at a time. I don't know what equipment you have but the first step and the most important one is to get the Polar alignment right and the rolling train starts from here. Let us know how we can be of help but one problem at a time. My only other point of advice is to ditch the intervalometer if at all possible and use a laptop. I use APT for Canon and Backyard Nikon for my mono modded 5100 ( I had to construct a device to enable long exposure capability but it does work ), this will sort out a lot of your problems.

A.G

[pipnina]

The signal goes up linearly with the exposure.

The noise goes up as the square root of the exposure.

The snr increases as the square root of the exposure.

Not sure I understand... I can get that signal goes up linearly (e.g. 1 second exposure = 1 "unit" of signal, 2 seconds = 2 "units" of signal) but I don't understand why noise increases with exposure, too.

I think I get the 3rd bit.

[pipnina]

Hi,

I have just read your post and you seem to have quite a lot of issues with your imaging. I used to be like yourself so there is light at the end of the tunnel but with AP problem solving you need to approach it systematically. One step at a time. I don't know what equipment you have but the first step and the most important one is to get the Polar alignment right and the rolling train starts from here. Let us know how we can be of help but one problem at a time. My only other point of advice is to ditch the intervalometer if at all possible and use a laptop. I use APT for Canon and Backyard Nikon for my mono modded 5100 ( I had to construct a device to enable long exposure capability but it does work ), this will sort out a lot of your problems.

A.G

Camera, tripod, intervelometer is my imaging equipment atm. That's going to change soon (building a barn door) but thanks for suggesting the laptop, once the barn door is built I might be able to use my dad's netbook to issue commands to the camera if I leave it on a table or something next to it.