Master dark question

[Anthonyexmouth]

Now I've got a cooled camera I set it off making a dark library. What's the best process for creating a master dark? Do you just stack them as you would lights? 

Is it worth making a master dark?

[ollypenrice]

1 hour ago, Anthonyexmouth said:

Now I've got a cooled camera I set it off making a dark library. What's the best process for creating a master dark? Do you just stack them as you would lights? 

Is it worth making a master dark?

I follow the advice of the makers of AstroArt, my stacking program. I take about 20 and stack them without alignment using the simplest algorithm, 'Average.' I've found that the most important thing is to exclude light so I don't do them on the telescope, I take the camera off and put its screw-fit metal chip cover on. It's certainly easier to use one master dark than a whole set of individual ones each time.

Olly

[steppenwolf]

58 minutes ago, ollypenrice said:

I follow the advice of the makers of AstroArt, my stacking program. I take about 20 and stack them without alignment using the simplest algorithm, 'Average.' I've found that the most important thing is to exclude light so I don't do them on the telescope, I take the camera off and put its screw-fit metal chip cover on. It's certainly easier to use one master dark than a whole set of individual ones each time.

+1 👍

[vlaiv]

I would add following:

Get as many of dark as possible. Don't settle for 20. I sometime used more than x10 that number. Remember, each dark carries both dark current noise and bias noise in it. Stacking 20 of them will reduce that noise by about x4.5 (square root of 20). You end up "injecting" that back into each sub when doing calibration. If you don't dither between exposures, this can translate into injecting that much noise in final image (if all lights are perfectly aligned).

Don't be afraid to get as many darks as possible, it will make a difference.

Depending on your environment, you might consider using different stacking algorithm than average. Average is good if you have "clean" environment, but if there is any chance of cosmic rays, or radiation or anything, then use sigma clip stacking instead. This is especially important if you use long exposure subs and you take many darks - at some you could have a cosmic ray hit or radiation hit. It happened to me more than once, I had one set of darks where I had such "event" on almost all frames at least once - it was due to wood ash near camera. Camera was placed next to fireplace (not burning) and there was residual ash that emits a bit of radiation.

Don't have such issues, or it happens very rarely if I take my darks in basement.

Master dark is very worth making, and not only one master dark - in fact do several master darks - at different temperatures (summer darks / winter darks) as well as of different exposures (if you do short exposures "fill-ins" to capture star color that would otherwise saturate in regular exposure - this might be needed for very bright object sometimes as well).

 

[Anthonyexmouth]

Thanks, what I've done so far is take 50x at 60s, 90s, 120s, 240s and 300s at unity gain and -10°. It's been warm and Tec was running at 100% to maintain -15 so thought -10 for summer and -15(20) for winter. 

Are cosmic ray hits easy to see or does the stacking sort it? 

From what I've seen not many are going much longer with the 294 but if I do I'll make those darks later

Is DSS ok for making master darks? 

What about bias frames? With APT it won't give me the option for really short subs

[vlaiv]

Don't do bias frames with CMOS sensors - most have unusable bias frames for some reason or another. You don't need bias frames for proper calibration if you match temperature and exposure length of matching frames (lights and darks, flats and flat darks).

33 minutes ago, Anthonyexmouth said:

Is DSS ok for making master darks? 

I think it should work well.

34 minutes ago, Anthonyexmouth said:

Are cosmic ray hits easy to see or does the stacking sort it? 

Not sure what you are asking - they are easy to see in a sub, especially if you have multiple subs - just blink them and you will see if there are any.

You will recognize it by:

- looks like hot pixel, but it is not a single pixel, but multiple adjacent pixels - most of which are saturated

- it does not look like star (no gaussian profile)

- if it is "round" it means that impact was close to perpendicular on sensor surface, if it is a streak, it means that impact was closer to parallel to sensor surface.

Depending on number of subs you are stacking - you can see them also in final stack if you don't use sigma clip stacking, but it usually takes stretching the data to notice them (much like hot pixels). Although signal saturates in single frame - it is only one of frames and more frames you stack, less contribution those saturated pixels will have. Stack enough frames and you won't be able to distinguish it from noise.

[Anthonyexmouth]

12 minutes ago, vlaiv said:

Don't do bias frames with CMOS sensors - most have unusable bias frames for some reason or another. You don't need bias frames for proper calibration if you match temperature and exposure length of matching frames (lights and darks, flats and flat darks).

I think it should work well.

Not sure what you are asking - they are easy to see in a sub, especially if you have multiple subs - just blink them and you will see if there are any.

You will recognize it by:

- looks like hot pixel, but it is not a single pixel, but multiple adjacent pixels - most of which are saturated

- it does not look like star (no gaussian profile)

- if it is "round" it means that impact was close to perpendicular on sensor surface, if it is a streak, it means that impact was closer to parallel to sensor surface.

Depending on number of subs you are stacking - you can see them also in final stack if you don't use sigma clip stacking, but it usually takes stretching the data to notice them (much like hot pixels). Although signal saturates in single frame - it is only one of frames and more frames you stack, less contribution those saturated pixels will have. Stack enough frames and you won't be able to distinguish it from noise.

Ah ok, got it. thanks

will need to hammer the credit card again and get the full version of PixinSight. guessing its pushing my luck to as for a 3rd trial.. lol

[Singlin]

I have cosmic rays in about 50% of my darks.

What should I do?

[vlaiv]

2 minutes ago, Singlin said:

I have cosmic rays in about 50% of my darks.

What should I do?

Create master dark with sigma clip stacking instead of simple average. There is a minimal chance that cosmic ray will hit two frames at exact same place, so you can set reject to fairly low value. In principle it should reject only one pixel, maybe two with such low probability.

[ollypenrice]

9 hours ago, vlaiv said:

I would add following:

Get as many of dark as possible. Don't settle for 20. I sometime used more than x10 that number.

 

I often use 30 minute subs...  

I did once try 20 versus 60 darks and could find no difference in the output image but, if you are using shorter subs, more certainly won't do any harm.

Olly

[Singlin]

I have produced darks for every temperature that I image, 300-600-900 and 1200 sec.

I thought that this is what was required 😀

[Singlin]

Do I use Average Sigma clipping or windsorised or just simple Sigma clipping ?

[Singlin]

In  Pixinsight can you make produce a Master dark in Batch processing without adding all the other file etc Flats,Biad and Lights?

I get an error saying that they are missing when I just try and do Darks alone.

[discardedastro]

You can't, no, but making one using ImageIntegration is pretty trivial. Just disable normalization (in both the main settings and the pixel rejection settings), disable noise evaluation, use whatever rejection works best for your setup and quantity of dark frames (average sigma will work for most, linear fit if you've got loads of frames).

You can then either load it as a master dark in the Batch Processing script, or use ImageCalibration yourself with your light frames prior to manually aligning and stacking.

[Anthonyexmouth]

Ok, kinda related question. Imaged the East Veil last night for a little while. This morning I stacked the subs in DSS with my darks and the output is only 700ish kb, if i stack without the darks its 100mb. 

any ideas? 

starting to miss my Canon. 

[vlaiv]

11 minutes ago, Anthonyexmouth said:

Ok, kinda related question. Imaged the East Veil last night for a little while. This morning I stacked the subs in DSS with my darks and the output is only 700ish kb, if i stack without the darks its 100mb. 

any ideas? 

starting to miss my Canon. 

No idea of what might be happening there, maybe if you post image of stacking results, both with darks and without, then we could be able to tell.

[vlaiv]

3 hours ago, ollypenrice said:

I often use 30 minute subs...  

I did once try 20 versus 60 darks and could find no difference in the output image but, if you are using shorter subs, more certainly won't do any harm.

Olly

It is mostly about read noise. This is important for CCDs because they tend to have larger read noise. Dark current is fairly low in most sensors, so read noise is dominant term.

One can lessen the problem if using dithering, and of course one should use dithering because of this and other benefits.

You can see this effect if you have data that was not dithered much. Do a simple experiment and calibrate with 10, 20 and large number of darks and measure background noise levels - just standard deviation in background patch of the image. You will see a difference in noise levels.

 

[Anthonyexmouth]

Lights Only.tifWith Darks.tif

 

No settings were changed between stacking

[vlaiv]

3 minutes ago, Anthonyexmouth said:

Lights Only.tif 97.89 MB · 0 downloads With Darks.tif 731.09 kB · 0 downloads

 

No settings were changed between stacking

Hm, it looks like there is a mix of things going on to produce this result.

Tiff format uses zip compression, so it will reduce actual file size, and "With Darks" one looks like there is quite a bit of "clipping to the left" - meaning most values are below 0, and it could be the case where for some reason (maybe file format or something) these ended up being 0. So you get image full of zeros and star here and there - that is easily compressible and ends up in small file.

Can you do the stacking again and save FITS output (32bit float point precision) so we can inspect those?

If most values are below 0 then there is some calibration error - "darks" are "too strong" (contain higher values than they should to be matched to lights) - can happen if there is difference in exposure time or gain or offset or something.

It would be good to include one light FITS and one dark FITS to do analysis and see what is going on.

[Anthonyexmouth]

L_2019-07-21_00-30-12_Bin1x1_240s__-10C.fitD_2019-07-20_10-29-37_Bin1x1_240s__-10C.fit

heres a light and dark to start with.

its very possible i screwed up the darks being new to the cooled cmos scene. 

[vlaiv]

35 minutes ago, Anthonyexmouth said:

L_2019-07-21_00-30-12_Bin1x1_240s__-10C.fit 22.31 MB · 0 downloads D_2019-07-20_10-29-37_Bin1x1_240s__-10C.fit 22.31 MB · 0 downloads

heres a light and dark to start with.

its very possible i screwed up the darks being new to the cooled cmos scene. 

Everything appears to be in order but something is seriously wrong

Header information is as it should be. Both show same temperature, gain settings, exposure length - one is labeled light frame, other is labeled dark frame (by APT).

However, data in dark suggests it is more like flat file rather than dark. How did you take your darks?

Look at this:

This is histogram of light frame in full 16bit range - exactly what you would expect - it's bunched up to the left but not clipping, it has 3 peaks because it is OSC camera (slightly different sensitivity to sky background between R, G and B ) so it's looking like proper light frame.

Now this is histogram of the dark frame - same range of values (16 bit or 0-65535):

This should not be like that, this is how flat histogram would look like (under funny light source, with quite a bit of vignetting).

This would mean that you either mistook flat for dark, but I don't think so, as stretched dark does not show usual things flat shows - dust shadows and proper vignetting. It shows very strange type of vignetting, so I'm guessing camera was not on the scope when you took darks. Here is stretch of dark to show what sort of vignetting it has:

So my guess is that you took darks with camera detached from the scope, but you had some sort of light leak - camera was not properly covered, and judging by histogram it is most likely IR type of light.

This is a good news. Your lights are fine, you just need to redo darks properly and everything should be fine.

[Anthonyexmouth]

to do the darks I had the camera cap on and sensor down on a table, thought that would help stop light leakage, must have been wrong.

i'll pop the camera off later and redo a few darks. 

 

[vlaiv]

1 minute ago, Anthonyexmouth said:

to do the darks I had the camera cap on and sensor down on a table, thought that would help stop light leakage, must have been wrong.

i'll pop the camera off later and redo a few darks. 

 

That is how I do it, never had any issues.

What sort of table is it? Some materials are transparent to IR - like plastic. That is why there is a risk of just using plastic cap on sensor. If table is plastic or glass type - that won't help much. Look for thick wooden table for this. Or better yet put a piece of aluminum foil over camera cap.

If aluminum foil can stop someone messing with your brain waves, it can certainly fend off some IR radiation

 

[Anthonyexmouth]

3/4" wood table top. 

just letting it cool and i'll post up a new dark, more light screening this time. 

[Anthonyexmouth]

Ok, heres a new dark. 

D_2019-07-21_12-36-16_Bin1x1_240s__-10C.fit