Help please - binning and exposure times....

[geoflewis]

46 minutes ago, ollypenrice said:

Vlaiv's explanation of the guide RMS/image scale rule of thumb is exactly the same as my understanding.

Olly

Thanks Olly, I think it's an example of great minds think alike 😎

Cheers, Geof

[geoflewis]

7 hours ago, vlaiv said:

I've taken a look at subs you attached, and they look pretty much ok, so I'm certain they are not to blame for zeros that appear in calibrated frame.

I do have couple of recommendations and one observation.

- Use sigma clip when stacking your darks, flats and flat darks. You are using quite a long exposure for all of your calibration frames and chance you will pick up stray cosmic ray (or rather any sort of high energy particle) is high. It shows in your calibration masters. Sigma clip stacking is designed to deal with this. Here is an example in master flat dark:

Now if that were hot pixels (like surrounding single pixels that are white) then it would show in the same place on master dark, but it does not (master dark has few of these, but at different locations / orientations).

- There is quite a lot of hot pixels that saturate to 100% value - these can't be properly calibrated, so you need some sort of "cosmetic correction" of these in your master calibration files. Such pixels will be removed from light stack by again using sigma clip stacking, but you need to dither your light subs (and I'm sure you do).

- I've noticed something that I find strange in your master flat, a sort of linear banding. Not sure why it is there, or if it has a significant impact on the image. Probably not, and it is probably related to manufacturing of sensor - sensor has slightly different QE in that area because of something.... If you did not notice this pattern in your final images, then it is all fine. Here it is what I'm talking about:

Does not look like regular vignetting as it has sort of "straight" edges (and not round) although it is in the corner where you would expect vignetting.

It is probably nothing to be concerned about - I just noticed as it looks interesting and I have not seen anything like that before.

Now on ImageJ - I've already written a bit on how to use it to do calibration, so maybe that thread would be a good place to start. I wanted to do a thread where I would describe full workflow in ImageJ, but it sort of did not draw too much interest so it slipped my mind after a while, and I did not post as much as I wanted, but I believe it has part on calibration. Let me see if I can dig it up.

Have a look at that thread - there is a lot written about calibration, and also a plugin included that will do sigma reject (which you will need for your subs).

You can try it out to see what sort of results you will get, but in long term - do look at specific software for doing it automatically like APP for example.

If you have any trouble following tutorial, please let me know and we can go into specific details.

Thanks Vlaiv,

Sorry for my slow reply, but being on holiday I was out with my wife much of the day and your analysis is too complex for me to review on my phone, so I'm only just now reading it on my laptop.

Good points about using sigma clip for my calibration masters, something that I never paid any attention to previously, just using the default in my software of average. I will certainly make that change in future and indeed I have resacked them that way and now there is no cosmic ray hit showing in the mast flatdark. I already use sigma clip combine method for my calibrated light frames, and yes, I do dither my lights, but I also use an additional filter in the software to correct any residual hot pixels not fixed during calibration, so usually my final stack lights are good.

Regarding the banding in the corners of my master flats, yes, it is something I have noticed, but I don't think that it shows in the final processed image, not least because that reagion is usually cropped out. I don't know what causes it. Maybe it is something to do with my LED light panel, which has a slight flicker when on the low intensity that I need to shoot flats at my target range of 25k ADU. I have been thinking of increasing the ADU to say 30k to see if that makes any difference. The panel has variable intensity, but for all the LRGB flats I have to use the lowest intensity setting or the camera shutter is captured during the exposure. I could increase the intensity for the Ha flats to reduce the required exposure from the rather high 90 secs, but I just used the same intensity as set for the LRGB filters. I will definitely exeriment some more after I get home the week after next.

Thanks for the link to your notes on ImageJ calibration, but I think this is going to be too complex for me to follow. As you advise, I think the better way forward is to trial something like APP, again something that I can do once I get home.

In the meantime, I intend to write to the developer of ImagesPlus to ask him about the auto calibation process and why it is black clipping so may pixels. The software does include a manual calibration process, which uses the same calibration frames as auto. The manual calibration set up option also includes a dark scaling setting with the default set at 1.0. I found that when I set this to say, 0.75 then the number of clipped pixels was dramatically reduced to ~100. Even with dark scaling factor at 0.9 the 0 pixels was a few thousand rather than the sevearl hundred thousands at scaling of 1.0. I really don't understand what this dark scaling factor is as I can't find any explanation for it in the limited documentation that I have for the software, so I'm going to ask about that too. I wondered whether the setting in the manual calibration set up might carry through to the auto image set processing calibration, but that seems not to be the case the dark scaling factor getting reset to 1.0. Do you have any ideas what the dark scaling option might be Vlaiv? Here is a screen shot of the set up screen fyi...

Many thanks, Geof

 

 

 

 

 

[vlaiv]

2 minutes ago, geoflewis said:

In the meantime, I intend to write to the developer of ImagesPlus to ask him about the auto calibation process and why it is black clipping so may pixels. The software does include a manual calibration process, which uses the same calibration frames as auto. The manual calibration set up option also includes a dark scaling setting with the default set at 1.0. I found that when I set this to say, 0.75 then the number of clipped pixels was dramatically reduced to ~100. Even with dark scaling factor at 0.9 the 0 pixels was a few thousand rather than the sevearl hundred thousands at scaling of 1.0. I really don't understand what this dark scaling factor is as I can't find any explanation for it in the limited documentation that I have for the software, so I'm going to ask about that too. I wondered whether the setting in the manual calibration set up might carry through to the auto image set processing calibration, but that seems not to be the case the dark scaling factor getting reset to 1.0. Do you have any ideas what the dark scaling option might be Vlaiv? Here is a screen shot of the set up screen fyi...

Yes, it would be a good idea to ask about calibration process.

Dark scaling is something that you don't need but can use if you want to. It is used to calibrate with different exposure time darks. For it to work properly you need bias calibration as well.

Idea is as follows:

dark contains: bias + dark_current(time)

I put time in brackets with dark_current, because dark current intensity depends on time (linearly) - meaning well behaved dark current accumulates with time in the same way light signal does - longer the sub, more of there is (and in fact for doubling of the time - dark current doubles as well, so it is linear dependence).

If you for example shoot one minute lights and have master dark of four minutes - you can still calibrate your lights provided that you have master bias. Process would be as:

calibrated light = (light - ((4min_dark - bias)*0.25 + bias) ) / master_flat

Or in simple words - you take 4 minute master dark, subtract bias from it, divide what remains (only dark current) with 4 because you want to go from 4 minutes down to 1 minute, and "return" bias in by adding it.

Note factor 0.25 in above calibration equation - that is dark scaling factor, and it's value is ratio of exposure times - that of dark you have and lights you have taken and want to be calibrated.

If you use that factor without removing bias (without master bias) or on master dark of proper duration (or in fact if you use other number then exposure ratios) - you will get bad calibration.

In your case, you were just scaling down proper duration master dark (without bias removal) and you thus made it smaller. So if pixel in master_dark was for example 500ADU, and in original light was 490ADU (remember - noise can make it go up and down), without this scaling you will get -10ADU as calibration result (or rather you should get) - but software clips that to 0. If you scale your master dark by 0.9, then you have 490 - 500*0.9 = 40. This value is positive and it won't be clipped to 0. That is why you are seeing less zeros when using dark scaling, but I'm afraid such usage of dark scaling only leads to poor calibration and does not solve problem with histogram clipping.

[geoflewis]

20 minutes ago, vlaiv said:

Yes, it would be a good idea to ask about calibration process.

Dark scaling is something that you don't need but can use if you want to. It is used to calibrate with different exposure time darks. For it to work properly you need bias calibration as well.

Idea is as follows:

dark contains: bias + dark_current(time)

I put time in brackets with dark_current, because dark current intensity depends on time (linearly) - meaning well behaved dark current accumulates with time in the same way light signal does - longer the sub, more of there is (and in fact for doubling of the time - dark current doubles as well, so it is linear dependence).

If you for example shoot one minute lights and have master dark of four minutes - you can still calibrate your lights provided that you have master bias. Process would be as:

calibrated light = (light - ((4min_dark - bias)*0.25 + bias) ) / master_flat

Or in simple words - you take 4 minute master dark, subtract bias from it, divide what remains (only dark current) with 4 because you want to go from 4 minutes down to 1 minute, and "return" bias in by adding it.

Note factor 0.25 in above calibration equation - that is dark scaling factor, and it's value is ratio of exposure times - that of dark you have and lights you have taken and want to be calibrated.

If you use that factor without removing bias (without master bias) or on master dark of proper duration (or in fact if you use other number then exposure ratios) - you will get bad calibration.

In your case, you were just scaling down proper duration master dark (without bias removal) and you thus made it smaller. So if pixel in master_dark was for example 500ADU, and in original light was 490ADU (remember - noise can make it go up and down), without this scaling you will get -10ADU as calibration result (or rather you should get) - but software clips that to 0. If you scale your master dark by 0.9, then you have 490 - 500*0.9 = 40. This value is positive and it won't be clipped to 0. That is why you are seeing less zeros when using dark scaling, but I'm afraid such usage of dark scaling only leads to poor calibration and does not solve problem with histogram clipping.

Thanks Vlaiv,

I thought it would be too easy a solution 🤔.

I have posted the question to the ImagesPlus User Group and will see what answers, if any, that I get. The software developer used to be very active on the user group and in the early days even directly supported me via email, with me in turn helping him with beta testing. However, this has not been so in the past couple of years, so I'm not sure that he is actively supporting the product anymore. Also there have been no development releases for maybe 3 years, or more. If there is no response via the user group, then I will write to him directly, but I'm increasingly reaching the conclusion that it is time for me to invest in an alternative like APP or PI, where there are more users worldwide, more ongoing development and more active support is available. Up until now I have been put off by the steep learning curve of a new processing application, but there comes a point......

Regards, Geof

[dph1nm]

On 27/09/2019 at 22:47, vlaiv said:

However, like I said, factor of x5 is arbitrary - which means that above calculated exposures are not "optimal" or something like that - they are just good guide line

Seems very over cautious to me! 8.7e- RN is equivalent to 75e- signal. A sky 5x this signal (not noise) - i.e. 375e- will only see the overall noise increase by 10% or so due to the RN contribution and I am pretty certain most people could not spot such an effect

NigelM

[geoflewis]

1 hour ago, dph1nm said:

Seems very over cautious to me! 8.7e- RN is equivalent to 75e- signal. A sky 5x this signal (not noise) - i.e. 375e- will only see the overall noise increase by 10% or so due to the RN contribution and I am pretty certain most people could not spot such an effect

NigelM

Hi Nigel,

I’m interested in your observation, so what would you recommend please?

Regards, Geof

[vlaiv]

1 hour ago, dph1nm said:

Seems very over cautious to me! 8.7e- RN is equivalent to 75e- signal. A sky 5x this signal (not noise) - i.e. 375e- will only see the overall noise increase by 10% or so due to the RN contribution and I am pretty certain most people could not spot such an effect

NigelM

You are in fact saying that one should make sky signal being x5 "read noise equivalent signal", rather than sky noise being x5 larger than read noise. You are in fact making LP/sky noise sqrt(5) = ~x2.236 larger than read noise.

On 375e sky signal will have associated noise of ~19.36 (square root of signal). Sum of read noise and sky noise is in this case 21.23, so you are right, it is about 10% increase.

With other approach (making noise 5 times larger), one will in fact have: sqrt ( (1 unit)^2 + (5 unit)^2) = ~5.1 unit (where unit is read noise, and LP noise is 5 times that), increase in percents will be: 5.1-5/5 = 2% increase.

Like I said - it is arbitrary, but I would rather go for 2% increase then 10% increase, but you are quite right, a 10% increase in the noise will be very hard to spot for most people. Here is an example:

This is montage of two images of pure gaussian noise. One with sigma 1 and other with sigma 1.1. Data was linearly stretched (from lowest to highest value) after montage, so both images are stretched the same.

What do you think, is it left part of the field one with more noise or right part of the field?

 

 

Edited September 30, 2019 by vlaiv
typo ...

[geoflewis]

Hi Vlaiv,

On 29/09/2019 at 21:59, vlaiv said:

Yes, it would be a good idea to ask about calibration process.

 

On 29/09/2019 at 22:32, geoflewis said:

I have posted the question to the ImagesPlus User Group and will see what answers, if any, that I get.

I didn't get any reply from the software developer, but I did hear back from one of the more experienced users, who also happens to use the same model camera (QSI583) as me. We had a few email exchanges, too long to copy it all here, but here are a couple of extracts from what he told me....

Geof,

Just because a pixel has a value of 0 does not mean that it is clipped.

If the master dark for a given pixel has a value of 300, with no photons reaching the sensor, then the "dark subtraction" process reduces the value of that pixel in the light frame by 300, assuming that 300 ADU of the light value is due to a thermal noise contribution.

Geof,

I reported the pixel values that are displayed in the IP status bar, at the right-hand end. They are 8 bit values. Below is the histogram of the 25 minute master dark. It shows 99.9% of the pixels with 8-bit values of 0 or 1 and an average 16-bit value of 261.



Geof,

5 minutes for Ha is not very long, so it is not surprising to me that you have a lot of 0 data after calibration. I usually shoot twice that for luminance data 1x1 and 8 minutes for 2x2 color.

I wonder if you tripled that time where your data would be.

 

So nothing conclusive there other than I've been shooting way too short exposures, so I'm wondering whether that is distorting my results.

 

I've sent my camera off to be serviced as I'm wondering whether the cooling is not as reported, since I noticed that one of the fans stopped working, plus I needed to have the desicant recharged anyway. When it comes back I'm going to shoot a new darks library for longer exposures at both bin1 and bin2, probably 10 min and 15 min initially as I'm not ready to go longer. I will also shoot some new flats, flat darks and bias, so that I can revert to also including bias in my processing. Once that is done I can try imaging again with longer exposures to see what difference I get.

 

Once I have a bunch of data to process, I'll probably download the trial for APP to see how calibration, stacking, etc. compares with my existing ImagesPlus software. I'm also tempted by PI, but from what I read that software eats computer resources for breakfast and I'm not ready to splash out multi £k on a new computer.

 

I feel that I've learnt a huge amount from this discussion, not least finding some tools in my existing astro software that I'd not previously used, so many thanks for your detailed analysis and explanations.

Cheers,

 

Geof

Edited October 1, 2019 by geoflewis

[vlaiv]

1 hour ago, geoflewis said:

If the master dark for a given pixel has a value of 300, with no photons reaching the sensor, then the "dark subtraction" process reduces the value of that pixel in the light frame by 300, assuming that 300 ADU of the light value is due to a thermal noise contribution.

 

1 hour ago, geoflewis said:

5 minutes for Ha is not very long, so it is not surprising to me that you have a lot of 0 data after calibration. I usually shoot twice that for luminance data 1x1 and 8 minutes for 2x2 color.

Well, that is wrong reasoning as it does not include effects of bias (not all of ADU value in raw dark are from dark current). Read noise is Gaussian type of noise that can both "negative and positive" values around some DC offset (mean value). Poisson type noise / associated signal is always "positive" - meaning you can't detect "minus" photons - you are always going to get positive (including zero) detections on the pixel. Due to bias, once removed - even zero detections on the sensor will produce in some cases negative results.

But don't be bothered by all of that as you will include other software like APP in your workflow. Once you start imaging again, and using APP - you can examine your calibrated sub to see if there are in fact negative values and what is the number of 0-valued pixels. You will also see if you can see the difference in your stacks once you have properly calibrated subs.

[dph1nm]

22 hours ago, geoflewis said:

I’m interested in your observation, so what would you recommend please?

I think vlaiv's picture answered that! I cannot see the difference between the two noise fields, so I personally would be quite happy with a sky 5x the square of the read noise (when measured in electrons of course). After all, what matters for amateur imaging is what the final image looks like - if you were doing precision photometry you might make a different choice. But it is all a matter of personal taste I suppose - I prefer my exposures to be as short as possible.

NigelM