Dithered/drizzled vs non dithered vs flats vs no flats!

[rofus]

Since I moved to autoguide I noticed more noisy/hard images from my setup, and after a while and thanks to some tests and suggestions here I discovered that my guide (very good indeed!) was so precise that did not help spreading the reading over pixels, process that helps noise being averaged out a bit.

So I was introduced to dithering and drizzling concepts

In parallel I also leart using Pixinsight for my postprocessing, and noticed that not using flats was somehow giving me better results because PI functions were better and final image less noisy.

So I decided to make some tests!

Having full moon it was a limited choice, but results are very interesting. I took 10 subs, 2 min each, 800 iso, of m81-m82. Added to this 5 darks, and optional flats. I did this twice, once using dithering and once without dithering, same temperature and basically same conditions. They were all aligned and stacked using identical parameters.

It's not a scientific test but gives some hints. To make drizzling (more) effective in terms of resolution I should have taken more exposures, but I thought was already a good test for noise reduction with dithering.

Below I attach three images (all 2:1 zoomed):

1) non-dithered sequence VS dithered & drizzled stack (same STF applied to both)

2) non-dithered sequence VS dithered & drizzled stack (same Histogram applied to stretch both images)

3) dithered sequence stacked with flats VS same dithered sequence stacked WITHOUT flats (I tried applying a somehow similar Histogram as they were very different)

The main and most important thing I wanted to test was the noise distribution using dithering and drizzling (without drizzling results were not as good). It's clear from images 1 & 2 that even with such a small set of images, it's vital to use dithering if long exposures and good autoguiding. Difference is huge and showing both in default STF view and stretched histogram ones. I also thing the drizzling helped a bit giving out a more detailed imaged even though less noisy.

The usage of flats VS no flats, on the same dithered/drizzled stack, is more curious even if it's not the first time I notice this. But that's less of a concern because PI does a much better job than flats

Any comment is welcome, as I said it was just a test for me to understand how much dithering is important, and I thought was useful to share.

[rofus]

Couple more details: shoot with a Nikon D90 on Skywatcher 150 Startravel.

Calibrated, registered, stacked etc. in PI.

[james_screech]

Interesting results with the flats. What happens if you do a mild blur of the flats before using them?

It could simply be that there is a lot of noise in the flats and this is being transferred to the final image. How many flats did you use?

James

[rofus]

Interesting results with the flats. What happens if you do a mild blur of the flats before using them?

It could simply be that there is a lot of noise in the flats and this is being transferred to the final image. How many flats did you use?

James

I used about 20 flats, so enough for the noise to be averaged out when PI is calibrating and creates its master flat.

Talking with others some time ago I realised I'm the only one experiencing this, and it was a common opinion that it's still useful with some old setups/specific ccd/dirty sensors (I always keep mine clean), but today with software is possible to do something more clean and precise without more elaborations during calibration.

To be honest Pixinsight does a much better job than flats, without introducing more 'signal elaborations' when calibrating but with more sophisticated maths to even out things.

I just wanted to confirm this and share...I almost never use flats usually and now with Pixinsight even less...

I think results with dithering and drizzling and without dithering are really interesting as well: even with such a small number of exposures the slightly added detail and the more noise-free image obtained with dithering and drizzle during stacking is incredible...enough for me to always use dithering from now on!

[glowingturnip]

totally agree re dithered and drizzled and see similar results on my own stuff.  I adopted it as my standard practice pretty much as soon as they released it

[ollypenrice]

Flats are essential but, of course, they need to be right. You cannot fake flats by using DBE, though it does an almighty good job of trying! If your flats are not working this is not because flats are not a good idea, it's because there is something wrong with your flats. You need to find out what that is. (Yves and I had a right old battle with just this problem.)

The numbers suggest that 40 flats would be better than 20 but more than 40 wouldn't be worth the bother. Flats need to be calibrated with darks-for-flats but a master bias should make a perfectly good dark-for-flats.

Finding out the right ADU peak for the flats is important and if you are struggling I'd suggest dropping it to below 1/3 maximum depth.

Olly

[rofus]

Hi Olly,

I agree obviously that I could do something maybe to get better results with flats, and that does not mean that are always this way, but I just think on the other side they're not always needed. I tried generating dark for flats, result was a perfectly black frame (obviously my D90 has all onboard processing/noise reduction etc. disabled).

With my D40  a few years ago and with the D80 later, both attached via smaller connection I had vignetting that I got rid of with flats, and they helped greatly (images were better with than without). Since I updated to big(ger) refractor, 2" connection and D90 camera, I found that without flats images are actually better and smoother, and I got rid more efficiently of unevenness and vignetting via PS astro actions, lens correction, and finally with PI. Mostly because I anyway cut/crop to around 70% of the frame to avoid focus/stars elongated around the frame, normal with my optics without a field flattener. Once done that the rest of the image does not need flats.

I just thought was useful to share because I know many others stopped using flats and actually started getting better results using PI and calibrating only with darks and bias. But as I said it probably depends on many factors, including obviously my equipment and the way I'm taking flats. Maybe someone else can make a similar comparison without flats vs with flats vs without flats BUT with PI, I think results would surprise many.

For me the most important stuff was the stunning quality difference between non dithered, dithered/drizzled with flats and then just dithered/drizzled...for sure the latter it's my new imaging workflow 

[ollypenrice]

I'll take some convincing. If you are chasing faint data (Integrated Flux, Tidal tail in the Leo Triplet) then only flats 'know' what is signal and what is uneven illumination. No software can know this. Rogelio Bernal Andreo says that, in chasing the Integrated Flux nebula, he didn't dare use DBE because he feared it would fail to distinguish between signal and background. The importance of flats does vary between targets but for some I don't believe there can be any way around them. But, boy, they can be a pain when they won't play.

One problem can be unexpectedly reflective surfaces withn the light path. Not all matt blacks are really matt black in IR. https://www.sbig.com/about-us/blog/flat-fields-the-ugly-truth/

Olly

[rofus]

Hey Olly,

thanks for that link, really useful I'll have another good read and will use it in the future. Surely I agree on very faint objects it could help, and as I said it depends on equipment (my 150" short tube refractor is not ED or Triplet and so I expect is not made to perfection).

I'll do some more tests about flats in the future maybe, and will use that link.

For now I feel I want just to wait another clear night and image some more with dithering

[dph1nm]

Remember flats also remove pixel to pixel variations in sensitivity which would otherwise add to the noise in your final image. This cannot be taken out in software.

NigelM

[IanL]

What light sources are you using for your flats?  With a DSLR / OSC artificial 'white' light sources may be anything but.  If your source is giving out insufficient light in one channel (often red) it can end up much noisier than the other two.  LEDs and some flat panels suffer from this problem.  Your best bet is a cloudy daylight sky with a diffuser over the objective or incandescent bulbs in a light box or with a screen, as these give out a continuous spectrum.

You can check your flats using the PI statistics process.  Just open it up and select your flat in the drop-down.  Click the 'spanner' icon and make sure that MAD (median absolute deviation) is selected. Compare the MAD values for each of the RGB columns.  If one is significantly higher than the others, that channel will have far more noise and could be the culprit.

[rofus]

Hi,

for my flats I use an iPad Air screen white app specifically made to provide neutral light. As a photographer I tested it already and there are no colours dominant in it. Consider also that I shoot with a Baader Semi Apo always in, so both my lights and flats are done with it.

Using this same method never created a problem, but as I said was working well (making things better) with my Nikon D40, slightly less with my Nikon D80 (in both cases with a Startravel 102 with a 1/4 adapter. It solved heavy vignetting and other imperfections in the image.

Since I moved to a Nikon D90 on the Startravel 150 all on 2" I stopped using flats because I noticed it was just worse. I used instead PS to solve vignetting (or just crop), and now with PI much better. It probably depends on equipment configurations, subject of the photograph etc., but I don't think is too much about my flats technique.

I attach one of my last flats HEAVILY STRETCHED in Photoshop, and I mean really heavily....it just shows as you see some vignetting (really pushed) that is easily fixable in sw, but nothing else...my old flats were showing many more imperfections.

[james_screech]

That flat is a lot of noise in that flat. What are the stacked flats like? What ISO are you using? Flats can be taken at a lower ISO than the lights and darks. You could also try a mild blur of the masker flat to remove some of the noise.

James

[rofus]

That flat is a lot of noise in that flat. What are the stacked flats like? What ISO are you using? Flats can be taken at a lower ISO than the lights and darks. You could also try a mild blur of the masker flat to remove some of the noise.

James

Yes there's a lot of noise because it's just ONE flat (not the master flat) and because as I said I literally stretched it to roughly 90% of the visible, meaning that is VERY stretched, so a single/non stacked flat that stretched is normal to appear with that noise.

Flats afaik are better to be shoot at same iso, and it's just 800, noise with that extreme stretch is just normal imo. I can try blurring etc. but then again a flat is nothing but to remove vignetting, that is anyway not an issue with my setup because is very limited and not present in the crop I do anyway (elongated stars around 20/30% of the view because I use just an achromat and without any field flattener).

The point was just to show that is a very 'flat' flat, and as such not really useful to remove any unwanted info apart from vignetting that I much better remove via software and it saves a lot of unnecessary work. Don't know probably is my specific setup, it just seems I don't need any flat, that indeed just adds artifacts while helping for vignetting, and I get the same via sw...

Will try as well more tests...it's just a behaviour I noticed upgrading the quality of my DSLR and that I saw in others using large scopes and more modern/larger sensor cameras...