Dithering Vs Darks (DSLR) - how many Darks anyway?

[Notty]

Hello, I've just started playing with dithering (APT-PHd) during my last session, as I've been struggling with the concept of wasting huge amounts of imaging time taking Darks.  I was taking 600s subs, and I take flats and dark for the flats also.  Not sure how it would have compared had I taken Darks instead, but it raised a couple of other questions, given a finite imaging window overnight:

1. Would more data (dithered) with the ensuing higher s/n ratio always result in a better image than less undithered data but supported by Darks? 

2. I assume you cannot correct dithered data with Darks, is this right? It's either one or the other?

3. For the given imaging timeframe, what proportion of data/darks would you take?  E.g for 6x600s exposures what's the minimum number of dark frames that would make it worth doing?

4. I am aware of the concept of building up the Dark library, (I cannot see me having the time to organise one) but I know they have to be taken at the same temp as the sensor as it records the lights.  This is a moving target though as the sensor warms up over the course of the session, so if you take them at the end of imaging the you're carrying up the upwards movement of the temp.  How close do they have to be anyway?  Within 2 degs? 5? 10?

If I invest in a cooled CCD will this all go away anyway?  Many thanks for any advice.  Here's my dithered Elephants Trunk Neb, over processed. (My search for an alternative to Photoshop continues)

 

[MattJenko]

I think you are finding that darks and DSLRs are a dodgy combination. Lots of dithered subs and maybe a hotpixel map for rejection will work considerably better. Flats are the key calibration frames to take though. and they don't have to be at the same temp. With a cooled CCD, you can take all the darks during the day at the set cooled point and because all the subs are taken at that temp as well, darks have a chance of working properly. I have an Atik 414ex and still don't use darks...

If you are prepared for a learning experience and an upfront cost, you could do significantly worse than PixInsight. 

[Notty]

Thanks  Matt.  CCD is definitely the way to go when I can afford it, and I'm glad you're confirming my suspicions that darks might be not as helpful as theory suggests with my DSLR.  I've looked at PixInsight but having read about it I just don't think I have time to take it on.  It's a shame, 90% of my processing is levels and curves which you would have thought are the basics in any package but I've tried Gimp (8 bit only so no good) and I've got AA6 (just can't get to grips with manipulating them ) nothing else gives me the usability and results of Photoshop - yet I am desperate to be free of their rental model given I can go 3-4 months between imaging sessions!

[MattJenko]

There can be excellent results obtained with DSLRs, so don't be too hasty to dismiss them, you just need to play to their strengths. As with most astrophotgraphy, life becomes much easier when you have lots of subs, although it takes longer to stack them all!

I thought there was a dev branch version of GIMP which handles 16bit images?

[Notty]

Crikey you're right, in fact the next stable release 2.10 will support 16/32 bit!  Installing from all the developer source looks a bit daunting but once they've packaged it up as a stable release I'll be all over it yippee!

[ollypenrice]

If darks are not perfectly temperature matched, and without set point cooling they won't be, they will probably do more harm than good and wasting live imaging time would be insane!

If I were you I'd do a large scale dither with a DSLR, 12 pixels or so, use a master bias as a dark, and sigma clip for stacking. Use a hot pixel filter and, if your software supports it, a bad pixel map. The calibration files are applied before stacking so there is no issue with dithering. You can also use a master bias as a dark for your flats. This will be no different, statistically, from a dedicated dark for flats and will save you time.

Olly

[mftoet]

1. Dithering is of upmost importance when imaging with a DSLR. Be sure to make relatively large dither steps (> 10 pixels of the main camera), otherwise dithering won't be  effective enough and hot pixels might leave trails. Dithering is more important than using darks, but the best way is to use both.    

2. No it's not either one or the other. Your sensor (and thus the effects of dark current) is fixed - the field of view moves. That's the concept of dithering; when registering your subs, noise and sensor artefacts while move relatively to the stars and get cancelled out with outliers rejection (sigma clipping).     

3. Use at least as many dark as lights. It's even better to use twice the amount of darks. NEVER use less darks than light; in that case you'll be injecting noise.

4. Depending on the software you use, dark frames can be scaled. I use PixInsight and have recently used scaled darks that were taken at -1º for calibrating lights that were taken during a night where the ambient temperature dropped from approximately 19º to 13º. I've compared the differences by calibrating the lights with and without the -1º darks and the dark-calibrated lights definitely show less noise.

EDIT: of course it's best to match the temperature of the darks with that of the lights, but don't be panic about several degrees of temperature difference. I have to confess that extending my dark library has been standing on my list of "things to do" for quite a while now...

[Notty]

1 hour ago, mftoet said:

 

3. Us at least as many dark as lights. It's even better to use twice the amount of darks. NEVER use less darks than light; in that case you'll be injecting noise.

I never made that link, and may have been caught out before by doing just that!  That might help me I think, but again, with 6-10 600s subs there's no way I can spare that much time again for darks.

One further question on dithering though, from APT settings I have a dropdown of dithering "amounts" (I don't think it specifies the units) but it only goes up to 5.  Do you know where/hoe else I can tell it to up the movement?

[dph1nm]

3 hours ago, Notty said:

Crikey you're right, in fact the next stable release 2.10 will support 16/32 bit!  Installing from all the developer source looks a bit daunting but once they've packaged it up as a stable release I'll be all over it yippee!

Gimp 2.9 installers are available at http://www.partha.com/

NigelM

[sharkmelley]

Some of my own thoughts on this:

1) You asked how close in temperature the darks have to be.  Dark current (and therefore pixel brightness) doubles for a 6C rise in sensor temperature.  Therefore temperature matching of darks to lights must be done quite precisely to be useful - say to within 1 or 2C.  In other words building a master dark library is a waste of time unless you are certain you can perform temperature matching with precision.

2) Don't trust the temperature you see in the EXIF data.  In other words building a master dark library is a waste of time (did I already say that?). 

3) For dark subtraction it is much better to buy software (e.g. PixInsight) that uses mathematical matching algorithms to scale the master dark to each light.  I use a single master dark taken during Summer ambient temperature all year round - this single master dark is generated using 4 hours of total integration (dark) time.  I update the master dark once a year because new hot pixels will always gradually appear on the sensor.

4) I use both dithering during acquisition and scaled dark subtraction during processing.  I regard both as essential if you intend to apply massive stretches to your data.  Sigma rejection is essential during integration of registered light frames.

5) If you can do large amounts of dither (as suggested by Maurice) then you might get away without darks completely if your sensor has state of the art dark current suppression (e.g. Sony Exmor or Canon 7D mkII).  Personally I use PHD2 for guiding/dithering which doesn't give me get sufficient amounts of dither with my guide scope and camera - your mileage may vary.

Mark

[michael8554]

Olly advised dslr dithering plus "bias as darks" some time ago, and I've used it ever since,  not shooting multiple 10 min darks saves ages.

Michael

[Ouroboros]

How is amp glow removed if darks are not used? 

[ollypenrice]

47 minutes ago, Ouroboros said:

How is amp glow removed if darks are not used? 

Fair point. On long subs this sometimes slightly affects my CCD images in one corner. I just remove it in Photoshop. It takes seconds, but if it's a problem with a specific camera then it might oblige you to use darks. It seems to me that modern cameras have far less amp glow. I don't know if this is really the case?

Olly

[wimvb]

If amp glow is weak and very gradual, than gradient removal (such as DBE) can take care of it. For a DSLR the same should apply to amp glow as it does to hot pixels; it may not cancel out if the temperature is not correct.

With my DSLR I have a light area on the left hand side of each frame, which I've never been able to remove properly with darks. Nowadays I can reduce it with DBE and I crop the image to remove what's left of it.

[Ouroboros]

3 hours ago, ollypenrice said:

Fair point. On long subs this sometimes slightly affects my CCD images in one corner. I just remove it in Photoshop. It takes seconds, but if it's a problem with a specific camera then it might oblige you to use darks. It seems to me that modern cameras have far less amp glow. I don't know if this is really the case?

Olly

The odd thing about my aging 450D is how inconsistent amp glow can be.  Sometimes it is obviously present and sometimes it isn't - so darks may not always help. 

[Alien 13]

21 minutes ago, Ouroboros said:

The odd thing about my aging 450D is how inconsistent amp glow can be.  Sometimes it is obviously present and sometimes it isn't - so darks may not always help. 

Some Amp glow can actually be the camera battery getting warm, this could be different for any that you use. The fix is to use a battery grip or ext PSU with adapter.

Alan

P.S. It is easy enough to confirm by looking at the recorded amp glow relative to the battery position remembering that the sensor top and bottom are reversed.

[Notty]

18 hours ago, michael8554 said:

Olly advised dslr dithering plus "bias as darks" some time ago, and I've used it ever since,  not shooting multiple 10 min darks saves ages.

Michael

Whoa that's a new one on me!  If that's possible then that would be most interesting.  Incidentally Ive been trying to use Astroart for my pre and post processing.  I was having very bad results until I was told by the AA designer not to use biases at all!  Once I removed them it stacked them properly, now i don't know if this is an Astroart thing or not.  So now I'll have a try again and put the biases in as Darks.

[Ouroboros]

19 hours ago, michael8554 said:

Olly advised dslr dithering plus "bias as darks" some time ago, and I've used it ever since,  not shooting multiple 10 min darks saves ages.

Michael

I've been puzzling about this suggestion for some time.  In the DSS calibration process (described here) the master bias is subtracted from the lights anyway. So wouldn't using  bias in place of darks be subtracting bias twice in the DSS calibration process? 

[Stub Mandrel]

Can we have an explanation of this 'bias as dark' process? I just tried using a master bias as a 'master dark' and the result was, frankly, appalling - it suggested that something had gone seriously wrong with the data.

[ollypenrice]

You have to be careful not to double subtract bias. Bias is contained in a dark so don't subtract it twice. I don't use DSS but the thing is not to get hung up on the terminology. If you are going to use a bias as a dark you should probably call it a dark and certainly put it where darks go in your stacking software, not where bias go. One of the many reasons I like AstroArt is that you can see exactly what it is doing. If you put any file in the section for, say, darks, it will be treated as a dark. 

Olly

[redfox1971]

I just tried loading a master bias in pixinsight for the master dark but the batch processing script will not allow it to be selected as was already in use for master bias so to get around it I selected a load of single bias frames for the darks which it excepted the resulting image did not have any considerable difference.

[ollypenrice]

2 minutes ago, redfox1971 said:

I just tried loading a master bias in pixinsight for the master dark but the batch processing script will not allow it to be selected as was already in use for master bias so to get around it I selected a load of single bias frames for the darks which it excepted the resulting image did not have any considerable difference.

PI got it right. Don't double subtract the bias. PI will be asking for a bias so it can scale the darks for temperature mismatch. It rejects the bias as dark because it sees it can't scale it since it's the same thing. (I suppose.) I like AstroArt because it does not try to second guess me. It lets me put the calibration files I have into whatever role I wish to try them. I use bias as dark, bias as dark for flats, flats, bad pixel map, column repair and hot pixel filter. I can put any sub length into this system, from 5 minutes to 30 minutes, and it is happy. Believe me, with the volumes of data we get through here that is no mean advantage! In our last image, with 51 hours of data over several panels, I have cleaned precisely no errant pixels whatever and applied, literally, no noise reduction anywhere in the processing.

The trouble is, I'm describing what works for me and if it doesn't work for you then it doesn't.

Olly

[Stub Mandrel]

10 minutes ago, ollypenrice said:

You have to be careful not to double subtract bias. Bias is contained in a dark so don't subtract it twice. I don't use DSS but the thing is not to get hung up on the terminology. If you are going to use a bias as a dark you should probably call it a dark and certainly put it where darks go in your stacking software, not where bias go. One of the many reasons I like AstroArt is that you can see exactly what it is doing. If you put any file in the section for, say, darks, it will be treated as a dark. 

Olly

DSS asks for both darks and bias frames; it works so presumably it deals with them properly in its own way the help doesn't say what transformations it uses (but at least looking showed me where to find Bayer Drizzle at last!).

I put a master bias in the dark section and it treated it as a master dark, and I also put it in the bias section where it was treated as a master bias (I did have to rename a copy as if they have the same name it won't load it twice).

This is all getting me so confused - I just discovered I ran through a test image and used the wrong ISO master frames :-0

[Ouroboros]

So in DSS would the equivalent be to not add bias files in the normal way, but to only include a master bias as a dark? 

I must say I'm struggling to see why this is different from simply allowing DSS to subtract the master bias in the normal way and not include anything as darks. But maybe there are subtleties about the calibration process in DSS http://deepskystacker.free.fr/english/theory.htm#CalibrationProcess that I don't understand. 

[ollypenrice]

51 minutes ago, Ouroboros said:

So in DSS would the equivalent be to not add bias files in the normal way, but to only include a master bias as a dark? 

I must say I'm struggling to see why this is different from simply allowing DSS to subtract the master bias in the normal way and not include anything as darks. But maybe there are subtleties about the calibration process in DSS http://deepskystacker.free.fr/english/theory.htm#CalibrationProcess that I don't understand. 

No, you are probably right. As I say, I have used DSS as an experiment but I can't say it appealed to me, thought that is not necessarily the fault of DSS! There is only so much software you can get your head round, and for me that is not a lot of software!!!

Olly