I need your help evaluating those darks!

[R26 oldtimer]

Good morning, I've just finished my DIY conversion of a mirrorless camera.

The mods include:

Removal of the IR filter (single), making it a full spectrum mod.

The soldering of cables directly to the shutter button, since the camera lacked any remote shutter connection.

The cooling of the camera back using peltier elements

Everything works ok, and since its completely cloudy over here, I took some test darks to see how the camera works. So I took two sets of 900sec dark frames, one set  inside the house at 22deg celcius with the camera uncooled and cooled and another set outside at 12deg celcius, again with the camera uncooled and cooled. The cooled darks look a bit better to me, but I've noticed that the uncooled dark when shot outside at 12deg ambient looks a bit better than the cooled one at 22deg, suggesting that the cooling effect is actually less than delta 10deg.

I would like your opinion on those darks (maybe using some fancy ccd analysis tool?), because if the cooling effect is minimal, then I might as well remove the peltier cooler.

Thanks in advance!!

[vlaiv]

Best way to asses darks would be to post them in raw format without any sort of transformation / debayering done to them.

Here is quick analysis of first posted png:

Due to debayering performed (or maybe as a consequence of light leak?), statistics are not looking good:

With regular raw you would expect "RGB channels" to have same mean value and roughly the same StdDev (same as Mode). Here we see that it is not the case - green is significantly lower than other two. This could be due to debayering or even some sort of light leak (different pixel sensitivity to light leak because of R, G and B filters on them).

We can see what stretched channels look like to see if there is any sort of pattern (amp glow or bias pattern or whatever):

You can see that green is much darker (this is same linear stretch on each channel).

Here are channels stretched roughly to same visual intensity (different linear stretch for each but showing roughly same information), together with histograms:

Channels look distinctly different, and in principle proper dark for color camera should not be different than that for mono camera - channels should look roughly the same, in fact you don't even need to split to channels for dark calibration - you do it while still raw.

In any case, I would be happy to run this sort of thing on your darks again if you provide me with RAW frame rather than debayered one (to see if there is indeed light leak or all issued above were caused by debayering process).

[R26 oldtimer]

Thanks Vlaiv for the quick response!!

I don't know if the fact that the frames were shot with a custom white balance makes any difference in the channels being different?  I've also noticed that gradient which looks like a light leak or internal light pollution or amp glow, so I've tested taking the darks in a box inside a cupboard with no difference, so if its not amp glow, then it must be some internal ir leak. Oh and the ISO was set at 800.

The camera is a sony c3 mirrorless so the raw files are .ARW, here they are:

_DSC0261.ARW

_DSC0263.ARW

_DSC0265.ARW

_DSC0269.ARW

 

 

Edited December 15, 2019 by R26 oldtimer
Spelling

[vlaiv]

15 minutes ago, R26 oldtimer said:

I don't know if the fact that the frames were shot with a custom white balance makes any difference in the channels being different?

Not an expert on DSLR and the way they work, but I would imagine that white balance information is not applied to raw files, but rather in post processing (or in jpeg conversion step in camera). At least that is what I believe since I can change quite easily white balance in software with Canon DSLR and its raw images (I don't use it for AP but sometimes for daytime photography).

Let's see if I can find software to convert ARW to Fits without any "distortion" and then examine those.

[R26 oldtimer]

Thank you again Vlaiv, I'll be waiting for your verdict!!

[vlaiv]

Ok, this is rather nice now

I used FitsWork to convert raw (ARW) files to fits. This is first time I've used it and it looks ok, I just disabled debayering and color scaling in options to prevent any fiddling with data.

Here are results of stats on each sub (1, 2, 3 and 4 being DSC0261 - 0269 respectively):

First few pointers - camera is obviously 12 bits and it shows as min value is -128 and max value is 3965 (that is 0 - 4093 range and max number in 12 bit is 4095). FitsWork applied automatic offset, so minimum value is -128 which is fine, while mean value is above 0 which is again fine and should be expected.

Mean value increases in each subsequent frame (this can be interpreted as higher dark current noise, or higher signal in general) but standard deviation does not increase like that (and it should if signal increase was due to dark current alone). Now standard deviation number here is not measure of noise since we did not remove bias from above subs (you need to remove bias signal to get pure random noise to measure it's value).

All 4 darks look fairly nice - same amp glow + bias pattern and 0269 looks probably the cleanest of the bunch.

Histograms also look nice - no clipping of any kind and they look the same for all four subs:

Now, in order to see what sort of noise each one really has - we would need to remove bias (maybe shoot small number of bias subs, like 16 or so and post those as well)?

For the time being I'll just play a bit more with the data to see what I can come up with:

Here is last two subs subtracted (0269 and 0265):

With set point cooled camera when you have exact temperature mean should be 0 - here it is 1 which is to be expected as respective means differ by 1 (17.955 and 18.977 in first table posted - stats for each sub). But this time noise went down and it is now 13.08 (compared to individual sub results in range of 40-50) - as this is true noise without bias signal. In fact, difference sub looks rather nice:

Smooth noise and not much else there (except that nasty offset of 1 - but that is either because of bias instability or as consequence of cooling, but I'm not sure which - we need a set of bias to figure it out).

Histogram of this difference between two darks also looks nice (bell shaped):

and FFT of difference also looks ok:

it is nice uniform noise - except for that vertical line which means that there are horizontal bands in resulting image - meaning not all bias was removed because of difference of temperatures (that is why you need to match the temperature to remove bias features if doing just dark calibration without bias or you need bias removal + dark scaling to get good calibration).

Doing the same between first and second darks (0261 and 0263) reveals result with more noise:

But cleaner looking FFT (less pronounced vertical line - smaller mean difference and bias better removed):

Which number was shot under which conditions?

Another important thing that I just noticed - exposure time is rather different for each of these subs. How did you set exposure time?

914s, 909s, 931s and 970s

This could explain why there is "inversion" with subs - less mean level but higher noise (shot at higher temp but for less time than other subs).

[R26 oldtimer]

Vlaiv, I must say you are awesome!!

I'll just need a hour or two to digest (comprehend) all of the above, and I'll get back!

[R26 oldtimer]

Ok, here is a master bias (I think) shot at room temperature. Are bias frames temperature dependent?

bias-BINNING_1.fit  bias-BINNING_1.xisf

8 hours ago, vlaiv said:

Another important thing that I just noticed - exposure time is rather different for each of these subs. How did you set exposure time?

914s, 909s, 931s and 970s

I didn't use an intervalometer just a stop watch to count approximately 900sec, so that's why there is a variation.

8 hours ago, vlaiv said:

Which number was shot under which conditions?

DSC263 was at room temperature ( around 22degrees)

DSC261 was at room temperature ( around 22degrees) but with the peltier cooler on.

DSC265 was shot outside (at temperature of around 12degrees)

DSC269 was shot outside (at temperature of around 12degrees) but with the peltier cooler on.

I couldn't get a temperature reading out of the sensor

8 hours ago, vlaiv said:

With set point cooled camera when you have exact temperature

Unfortunately this isn't set point cooling, just the peltier running to it's max (obviously not enough to freeze the sensor)

 

And now a couple of more questions!

Since I won't be able to dither with this camera, I will have to use calibration frames (darks and flats). Is there a need for bias?

As the cooler is not set point, it will not help with calibration frames, it will just help with the noise especially during summer (temperatures above 30deg celsius), is that right?

You've used Fitswork to convert the .ARW files to .fits. Should I do the same before stacking them or feed DeepSkyStacker with the .ARW files?

And finally, do you see any real life advantage in using the cooler, or the benefits are minimal (as in wait a couple of hours for the ambient temperature to fall 5-8 degrees later in the night). Take into account that I am still new at AP, and far from chasing the tiniest detail to improve my pictures.

Again I have to thank you for your precious time to respond to my queries!!!

[vlaiv]

Unfortunately I can't do anything with that master bias as its somehow scaled 0-1 by stacking application or when raws were read (differently than FitsWork). Bias no longer represents ADU values.

21 minutes ago, R26 oldtimer said:

I didn't use an intervalometer just a stop watch to count approximately 900sec, so that's why there is a variation.

If you want to have get good results - you better find a way to make your subs last the same time (or as close as possible - less than a second would be good).

23 minutes ago, R26 oldtimer said:

Since I won't be able to dither with this camera, I will have to use calibration frames (darks and flats). Is there a need for bias?

Dithering is good regardless if you use calibration frames or not. Why do you feel that you won't be able to dither? You can manually dither PHD2 (if that is what you are using) - there is a script that will trigger dither. If you are already timing your exposures and sitting next to your rig - why not do manual dither between each exposure?

If you want to do proper calibration with this camera (and let's hope it is possible) - you will need darks, flats and bias (you could even use flat darks - but that is just overkill in your case I think - you can calibrate master flat with bias).

What you need to do is make sure you use dark optimization - it is algorithm that will try to guess scaling factor for darks if temperature is not exact (in fact - it will work if you miss exposure time by a few seconds as well). But for it to work properly, bias must be subtracted - therefore you'll need master bias.

27 minutes ago, R26 oldtimer said:

As the cooler is not set point, it will not help with calibration frames, it will just help with the noise especially during summer (temperatures above 30deg celsius), is that right?

Yes, that is correct - you can already see that it helps with noise a bit - in first measurement StdDev was smaller with cooled subs. To see how effective cooler really is - we need usable bias (to me at least).

Would it be too much trouble for you to do another master bias - but this time using FitsWorks to produce fits from raw, just make sure you turn off certain options like this:

Then do simple average on those 32bit fits with any program of your choice.

34 minutes ago, R26 oldtimer said:

You've used Fitswork to convert the .ARW files to .fits. Should I do the same before stacking them or feed DeepSkyStacker with the .ARW files?

Whatever you choose - stick with single approach - when mixing things you might get unusable results - like above, I used FitsWork you used something else - and bias won't match darks.

I would personally stick with FitsWork - because it seems nice and there is batch option (I just realized that).

36 minutes ago, R26 oldtimer said:

And finally, do you see any real life advantage in using the cooler, or the benefits are minimal (as in wait a couple of hours for the ambient temperature to fall 5-8 degrees later in the night). Take into account that I am still new at AP, and far from chasing the tiniest detail to improve my pictures.

Depends how much trouble it is for you - if it's not much trouble, then yes, keep using cooling - why not? (there will be no difference in processing if you want best results, and you are always chasing for finest detail to improve your images - you just don't know it yet - even as beginner you appreciate if your image is a bit better).

[R26 oldtimer]

13 hours ago, vlaiv said:

Would it be too much trouble for you to do another master bias

Would it be much trouble to start my car a few times, so an that an excellent mechanic with deep knowledge, would fine tune it for free?? 😊 No trouble at all!!

So, I've downloaded Fitswork and following the settings you've mentioned, I converted the raw bias frames to fits. Then I downloaded a trial version of PI and tried to do a simple average on them (no weighting).

integration pi.fit

integration1 pi.fit

In case I've messed up something in averaging, here is a link to Google Drive with the individual .fits bias frames

[vlaiv]

Just for your information - such things as simple average of fits and similar can be performed with ImageJ - free software for scientific image manipulation (there is Fiji version of it - loaded with plugins, and there is also AstroImageJ - loaded with plugins for astronomy use like plate solving and photometry and such).

[vlaiv]

10 minutes ago, R26 oldtimer said:

In case I've messed up something in averaging, here is a link to Google Drive with the individual .fits bias frames

Ok, PI just messes things up - again we have bias that is fractional and in range 0-1 for some reason.

I'll try to make it work, as I suspect that PI is just mapping (min, max) range to (0, 1) range - but I'm not 100% sure about that - I'll just try to see what happens. Btw - you did not paste link to google drive - but fits file twice

 

[vlaiv]

I don't think it worked well - I did get bias to -128 - 3965 range - same as darks, but mean value of bias is -14 (nothing wrong with that on its own - but quite a bit far from dark average values and I suspect it's not correct for that reason).

Once I remove bias from darks - there is residual bias signal there - which means that this bias is not good (either because of PI way of handling things or because bias is not stable).

Bias stretched:

Original dark (third one - 265):

Same dark with bias removed:

Almost good - bias is just a bit mismatched - otherwise that looks like rather clean pure dark - not much hot pixels, there are some temperature gradients - but at the bottom edge you will notice same pattern that exists in bias - this means that bias is not completely removed (bad for dark scaling - which is your aim since you don't have set point cooling) and can be because of scaled bias or bias instability in general.

 

[R26 oldtimer]

2 hours ago, vlaiv said:

Btw - you did not paste link to google drive - but fits file twice

Oops, my bad. Here is the link

https://drive.google.com/folderview?id=1EICLKoR6-NjwLYpae8W-ewZk2wbNdr8e

P. S. :Those bias frames were shot at room temperature, same as DSC263. Should they match the temperature of the darks (and lights in the future), or it doesn't matter?

[vlaiv]

15 minutes ago, R26 oldtimer said:

P. S. :Those bias frames were shot at room temperature, same as DSC263. Should they match the temperature of the darks (and lights in the future), or it doesn't matter?

In theory bias signal should not depend on temperature as it is not time dependent (shot at shortest possible exposure time) so there is no build up of thermal things. But in practice ambient temperature can have impact if it can influence readout electronics - it will probably only impact noise levels slightly and not bias signal.

[vlaiv]

I'm not sure what is going on here, but your "bias" subs are all over the place.

I don't know if info in fits header is correct or not, but you have "bias" subs with ISO12800 and 3s exposure mixed with ISO800 and 900+ seconds of exposure (that is really a dark sub not bias).

Now I would normally think that fits header info is simply not correct as bias needs to be taken at same ISO settings as darks and needs to be shortest possible exposure length - milliseconds, but stats on these subs kind of confirm that these subs are all over the place:

[R26 oldtimer]

The bias were at iso800 1/4000, but I must have done something wrong converting them to fits.

Would you mind if I upload another Google drive link in a couple of hours with the . arw  bias files?

I know this must be getting annoying for you, so I'll understand if you wanna call it a day.

[vlaiv]

1 minute ago, R26 oldtimer said:

The bias were at iso800 1/4000, but I must have done something wrong converting them to fits.

Would you mind if I upload another Google drive link in a couple of hours with the . arw  bias files?

I know this must be getting annoying for you, so I'll understand if you wanna call it a day.

No, not at all, yes, please upload .arw files and let's see what sort of information and master bias I came up with.

It now looks like you won't be able to use dark scaling efficiently - I had a look at two subs that have roughly same parameters (ISO12800 and exposure 3s - not sure if that is true or not - you will know if you ever took such subs and kept them) - and these two look rather similar and pretty much the way you would expect - but problem is that they have very different mean value for some reason. This could mean there is bias instability (different mean adu value for bias level each time you take exposure - this happens when sensor does some sort of internal calibration).

 

[R26 oldtimer]

Ok, I'm back. Here is the google drive link to a folder of fresh bias frames shot at iso800, 1/4000, .arw.

https://drive.google.com/open?id=1xU836mlPuEo7TlNXZcawPaubdt3_86Lk

[vlaiv]

2 hours ago, R26 oldtimer said:

Ok, I'm back. Here is the google drive link to a folder of fresh bias frames shot at iso800, 1/4000, .arw.

https://drive.google.com/open?id=1xU836mlPuEo7TlNXZcawPaubdt3_86Lk

Ah, lovely - these are really nice looking bias subs - very uniform and very much like good bias subs.

Look at their stats (again - FitsWork used to convert):

That is so nice - mean value is really nice at somewhere at ~0.3 with very small deviation sub to sub.

Master bias looks really uniform and flat:

and if we zoom in we can see vertical stripes that are common in master bias (either vertical or horizontal stripes are common).

What is not so good about this is that there is certain "signature" to dark subs that could be due to amp glow - which could be problematic when doing dark scaling - but you really need to try dark scaling to see if it will work.

Let's see if we can make sense from those initial darks now that we have master bias:

Quote

DSC263 was at room temperature ( around 22degrees)

DSC261 was at room temperature ( around 22degrees) but with the peltier cooler on.

DSC265 was shot outside (at temperature of around 12degrees)

DSC269 was shot outside (at temperature of around 12degrees) but with the peltier cooler on.

I just repeated what you written about conditions for initial darks so we can try to make sense out of it and to see if we can determine how good your cooling is.

DSC261:

Average signal level (dark signal after bias removal) is 0.018281393 ADU/s - I accounted for 914s exposure and that is why dark current is given in ADU per second (we can compare those values)

DSC263:

Dark signal is 0.018862519 ADU/s

DSC265:

Dark signal is 0.018966408 ADU/s

DSC269:

Dark signal is 0.019258152 ADU/s

These numbers seriously don't fit your information on how these darks were generated. Either that or there is something going on here that is "non standard" thing. Dark current doubling temperature is about 6C - so if we only observe subs that were without cooling and we look at 22C and 12C ambient - it is not unreasonable to assume that sensor was at least 6C cooler in lower ambient temperature - which means it should have about half of dark current in "colder" frames than in "hotter" - but difference between all is very small - few percent at most.

One explanation might be:

There is "amp glow" that is quite serious but does not behave like regular thermal noise - it is added in readout but does not exist in bias subs (which would be rather strange). Maybe it is light leak? If camera is modified and IR filter removed - any chance you had IR leak into camera while taking darks? This could happen if you had only plastic cover on camera - some types of plastics are transparent to IR radiation.

If you suspect this to be the case - put aluminum wrapping around the camera and redo darks to see if there is difference (do one shot with and one shot without IR protection - make sure you have same exposure length and compare mean ADU value in both - if significantly different - you have light leak).

Other explanation might be:

Camera does "internal calibration" on long exposure shots and removes "predefined constant value" from image based on exposure length - that is bad thing as there is no way you can do proper calibration then. Best that you can do is take darks of same duration as lights each time you image at the end of the session (or better yet - half at the start and half at the end - to even out any temperature difference).

[R26 oldtimer]

2 hours ago, vlaiv said:

DSC261:

 0.018281393 ADU/s

DSC263:

Dark signal is 0.018862519 ADU/s

DSC265:

Dark signal is 0.018966408 ADU/s

DSC269:

Dark signal is 0.019258152 ADU/s

So what you're saying is that if DSC261 (room temp 22°C) has a dark signal value (after bias substruction) of 0.018281393 ADU/s then the expected value of DSC265(outside temp 12°C) would be at least double that or more than 0.036ADU/s (given the temp difference of more than 6°celsius, where noise doubles) instead of 0.019258152 ADU/s?

[R26 oldtimer]

Well, next week over the holidays, I'll have the chance to shoot some subs under the stars, and get some darks and bias with the camera properly shielded from outside light, so we can come to a conclusion as where this anomaly comes from, external light or internal ir light or some strange internal calibration.

Vlaiv I want to thank you again for all the help you're giving me.

[vlaiv]

12 minutes ago, R26 oldtimer said:

Well, next week over the holidays, I'll have the chance to shoot some subs under the stars, and get some darks and bias with the camera properly shielded from outside light, so we can come to a conclusion as where this anomaly comes from, external light or internal ir light or some strange internal calibration.

Vlaiv I want to thank you again for all the help you're giving me.

You are welcome. Once you shoot all of those subs, should you wish for me to take a look at them again - just mention me here or send me a PM - I'll be happy to help.