Flat Issues - Broken Flat Field Generator?

[vlaiv]

14 minutes ago, AndyThilo said:

I don’t know how you’re doing that, what software? 

I'm using ImageJ - it is free software for scientific image analysis / manipulation, but that is not really important here - it just helps me calculate some things about light levels. You can't really use the method that I've used to get proper image - it is just estimate and all things that darks otherwise remove won't be removed like this (see amp glow for example - proper dark will remove it).

[AndyThilo]

That's the thing my darks do remove amp glow. I showed that in the image on my first post. Adding in the flats caused loads of issues. Maybe I'm not understanding it. The best processing method for me to get the cleanest images is manually in PI with no flats as below:- 

No gradients, no amp glow. I wonder if I need flats at all. The only darks issue I can see is the slight smudge above and right of the DSO. Maybe I'm getting it all wrong though? I'm just going on my limited knowledge and what I see on the screen. The image below has had a DBE, SNCR and manual stretch in STF. Am I missing the point? I appreciate all your help, just trying to learn :). 

[AndyThilo]

One other thing i don't understand. Left is PI manual processing without flats. Right is exactly the same but with flats. The right is also the same as I get using BPP. Both are stretched using STF Autostretch. Nothing else.

[vlaiv]

1 hour ago, AndyThilo said:

That's the thing my darks do remove amp glow. I showed that in the image on my first post. Adding in the flats caused loads of issues. Maybe I'm not understanding it. The best processing method for me to get the cleanest images is manually in PI with no flats as below

Darks polluted with light leak will remove amp glow - that is to be expected. In most cases light leak will act as light pollution. Same way you record your target although light pollution has been added - dark will record amp glow. So will light, once you subtract the two - amp glow will cancel out.

Light leak will not cancel out. That is the issue.

I don't know what your master dark looks like, but one single dark that you posted has gradient. If I do simple removal of dark from that light without doing anything else (no fiddling around with background subtraction, no flat calibration - nothing, just dark subtraction), I get this:

Now you have gradient to the opposite side than on dark. It is clear that dark subtraction caused gradient from dark to be transferred to flat. Amp glow is gone, and that is ok - both have same amp glow, but dark has this gradient that is not present in light.

Few things could be happening here that make a difference between my example and one that you gave from PI.

a)  I'm doing calibration with single dark. Maybe this particular dark is different from other darks for some reason, or maybe each dark is different because of different amount of light leak. You've used average so any differences averaged out, while I used only one that has distinct gradient

b) You used background wipe in PI and PI removed this gradient in same way it would handle LP gradient.

In any case dark is what is causing your flat calibration to fail. I can show you with a bit of math what is going on.

Imagine you have only two pixels rather than whole image (or maybe left and right side of image what ever is easier for you to imagine). One received 70% of light due to vignetting / dust shadow, while other received 90% of light again due to vignetting/shadow/whatever.

Now imagine that both of these pixels are background pixels that recorded just sky background. I say this because this means they ought to be uniform in intensity - there should not be variation (let's for moment leave LP gradients aside - this is just to understand how dark calibration impacts flat calibration).

Let's say that background ADU value is 100ADU. This means that first pixel would record 70e and second pixel would record 90e. We need them to have same value in the end if our calibration is good (because we have even sky background).

These values are just light signal, but light frame also contains bias and dark signal (dark subs contain both as well). Let's say that dark signal is 20e.

So what we recorded in our light frame would be 90e and 110e (70e+20e, 90e+20e). Now let's do calibration to get even background. Our flat will be 0.7 and 0.9 (because 70% and 90% of light reach sensor).

Perfect case:

((90e, 110e) - (20e, 20e)) / (0.7, 0.9) = (70e, 90e) / (0.7, 0.9) = 70e / 0.7, 90e / 0.9 = 100, 100 - we have equal pixels, or uniform sky brightness. Calibration is good because we had proper dark value.

Under correction case - dark has larger value than it should (because of light leak, some additional electrons were accumulated):

((90e, 110e) - (30e, 30e)) / (0.7, 0.9) = (60e, 80e) / (0.7, 0.9) = 60e/ 0.7, 80e / 0.9 = 85.7143, 88.88888

We no longer have uniform sky background, calibration failed, and first pixel still has lower value then second although we used proper flat (0.7, 0.9). Because vignetting / dust shadow is still present - we call that under correction - flat did not manage to fully correct image - but not because flat is wrong - it was because dark was wrong - larger than it should be.

Over correction case - dark has larger value than it should (in reality this rarely happens like that - it happens when lights have light leak and darks have lower value in comparison to lights, but let's do math anyway to show over correction happening):

((90e, 110e) - (10e, 10e)) / (0.7, 0.9) = (80e, 100e) / (0.7, 0.9) = 80e/ 0.7, 100e / 0.9 = 114.2857, 111.1111

Again - no uniform sky background but this time first pixel is brighter than second pixel - "inversion" happened and what was darker in uncalibrated image now is brighter as if flats corrected too much - we call that over correction.

-----------------------------

Above was to show that perfect flats can still fail to do flat calibration if there are issues with either lights or darks.

I believe that both your light and dark subs are polluted with light leak because of:

1) Dark has gradient. It is very unlikely that dark sub can have such gradient, and also such gradient is missing from light sub (but everything that is in dark should also exist in light - amp glow for example - if it's in the dark sub, it will certainly be in light sub as it is feature of dark signal). This shows that gradient is not feature of dark signal and is in fact "external" (only external signal can really be due to some sort of light / radiation)

2) You subs when calibrated show Over correction - that can happen if darks are "less strong" then they should be (see above). Since it is highly unlikely that dark current in darks is less strong than in lights (it can be if we have situation where cooling was not set to same temperature - but from what I can tell - it is not case here) then it must be the case where lights are somehow stronger than they should be. This points to light leak again. Lights have some sort of external signal that did not come thru objective of telescope lens. Otherwise it would be corrected by flats because flats describe how such signal behaves (how much it is attenuated).

Hope this all makes sense.

1 hour ago, AndyThilo said:

One other thing i don't understand. Left is PI manual processing without flats. Right is exactly the same but with flats. The right is also the same as I get using BPP. Both are stretched using STF Autostretch. Nothing else.

You have very red result in your right image because your flat panel is giving off very blue light. It has already been mentioned that red component of flat is very low in value. This can happen due two things. Either camera has very low QE in red part of spectrum (not the case) - or flat source used produces light that has much less of red in it than other two (green and blue). Cool light has this "feature" (with warm light it is opposite - less blue and more red and green).

Since you have low red signal compared to other two - flat fielding with such flat will produce very strong color dis balance.  Nothing that white balancing can't fix, or flat normalization (process where each of color peaks in your flats is normalized to 1 - that removes any color dis balance that flat panel produces).

[Physopto]

4 hours ago, vlaiv said:

Here flats are quite ok as far as saturation is concerned. Not sure where 1/3 rule came from - I've heard it before, but I don't think it is a good rule (unless someone can explain exactly why it is used). Aim at 80% or so histogram is better option. In fact any histogram value is good as long as you don't have low or high clipping. Higher histogram peak value only ensures that you have good SNR for your flats and that you won't be polluting your lights with noise much.

This is part of a page from the Maxim DL manual

Sky Flats Assistant MaximDL Application

This app for use with MaximDL allows the user to take a series of sky flats while automatically compensating for the changing sky brightness. It is also useful for users with light boxes since the exposure duration will need to be adjusted 

There is some common setup between the two modes - you need to enter the initial, min and max exposure durations, the target ADU level and error range

The initial exposure duration should be chosen to be reasonable for the desired target ADU count – this reduces the iterations required to get to the starting level. It doesn’t need to be accurate though, the app will iterate to get to the correct value.

The target ADU selected should be about 33%  of the saturation level of your camera. This will give the most accurate and noise free raw flats which will result in the best master flat once stacked. Going too high can result in pixels outside the linear range of the CCD and too low can result in poor signal-to-noise in the flat. The allowable error percentage allows you to control how close to the target ADU count the flats must be - exposures outside this range will not be saved, rather a new exposure time will be calculated and the exposure retaken. Making this too restrictive may result in a lot of exposures being thrown away.

The ideal target number of flats somewhat depends on the noise characteristics of your CCD, but after stacking about 10 flats the signal-to-noise ratio in the flat drops to a level which will generally be insignificant once applied to the light image.

[vlaiv]

13 hours ago, Physopto said:

The target ADU selected should be about 33%  of the saturation level of your camera. This will give the most accurate and noise free raw flats which will result in the best master flat once stacked. Going too high can result in pixels outside the linear range of the CCD and too low can result in poor signal-to-noise in the flat.

Ok, let's discuss this for a moment.

It says that 33% of saturation level will result in most noise free raw flats. Is this statement true?

Most noise free raw flat will be one with best SNR. With flats most dominant type of noise is shot noise. Read noise in comparison is very small (even with old CCD cameras) and due to usual exposure lengths involved, dark current noise is also very small. Therefore we can approximate noise by shot noise associated with light signal.

Let's now compare two SNR values of usual CCD camera that has 15k FW capacity. One flat at 33% and one flat at 50%. Signal value in 33% flat will have on average value of 5000e. Associated shot noise is sqrt(5000) = ~70.71e and overall SNR is therefore 5000 / sqrt(5000) = 5000 / 70.71e = ~70.71e. On the other hand 50% signal level will be 7500e and associated noise will be ~86.60e and therefore SNR will be ~86.60e.

We clearly see that there is example where above statement is not true. In fact, if shot noise is dominant noise source - higher signal value always "wins" in SNR and therefore histogram that has peak higher than 33% will result in better SNR and be more noise free than sub with 33% SNR.

Now let address second part of that statement:

13 hours ago, Physopto said:

Going too high can result in pixels outside the linear range of the CCD

This statement can be interpreted in two ways and we need to address both. In first case it could happen that CCD sensor is not linear in higher range. I can't really argue this case except to say that in recent time that I've been doing astronomy, I have not heard of imaging sensor that is not linear enough over its useful range.

If you do search on linearity for your particular camera model, I'm pretty sure you will find a graph like this:

(atik383L+)

or this:

or this for CMOS camera:

or maybe this:

In second case - it is meant that there could be some sort of saturation and clipping and therefore pixels can register non linear response. This can happen due to shot noise for example. Shot noise can cause signal value to be higher or lower than actual signal level (that is why it is noise). In fact we know what magnitude that noise is, and we can calculate probability that any particular pixel is above saturation point of the sensor.

Let's take value that I recommend often - 80% and see how likely that any one pixel is above saturation point. We know that one standard deviation is square root of that number or 8.95%. This means that 66.7% of pixels will be within 80% +/- 8.95%, and 95.5% all pixel values will be within 80% +/- 17.9%. This is still within 100% saturation point as 80+17.9 = 97.9%. In fact, if we calculate it precisely 1.27% of pixels has chance to saturate in single pixel. In stack of 20-30 flat subs that will produce negligible error. In fact this will hold true only if flat is perfectly flat - and it never is - there is vignetting and only central part of flat need to be taken into account as other parts of flat usually have lower values and are less likely to saturate  (disclaimer: I used gaussian approximation for poisson distribution when large numbers are involved).

Edit: I've made mistake of using percentages where quadratic relationship is involved (things are not linear) - fact is that this percentage is even lower - look at my following post for details

So this statement is certainly true - going too high will saturate, but as we see, even going as high as 80% will make very few pixels saturate (in reality less than 1%) in single subs and resulting error from stacking 20-30 subs will be minimal.

Edited January 26, 2020 by vlaiv
Error

[vlaiv]

In fact, I was wrong in my previous post - I did the math incorrectly.

I used percentages instead of actual numbers (figured it will be easier to understand) - but I should not have done so. SNR involves quadratic relationship (noise is square root of signal) and linearity is not preserved - hence percentages will not match actual numbers. I'll just quickly redo calculation with real numbers to show that this issue is even less than percentages would suggest.

Let's again take 15k FW. 80% of that is 12000e. Noise / one sigma is square root of that - ~109.55 and we have 3000e until saturation (15000-12000 = 3000e). This means that we in fact have 3000/109.55 = 27.38 sigma. Probability that any single pixel will saturate is not 1% but closer to following statement: "Universe is simply not old enough for this event to ever have happened so far given it's probability and all cameras in the world"

In fact, even if you go for 95% histogram peak, you are unlikely to saturate with real sensor - it is still more than 6 sigma event.

[Physopto]

Hi Vlaiv

Yes I can see your argument. I always use around 50% FWD for my CCD (QSI 683), ( what QSI recommended I believe). So I aim for around the 25,000e ish. It is a long time since I did any maths/statistics on shot and thermal noise. So I just aim for what seems to work. I would guess it may be different for the various makes of CCD depending on how they work. Things have changed greatly in the last 25 years or so since I did my first degree.

All I was pointing out is where carastro may have seen or gotten the 33% figure from. I use Maxim DL but have never tried their auto routine so far. I get too little time to waste messing about with experimenting. I beleive in" if it ain't broke, then don't fix it!"

Derek

[Physopto]

Hi Vlaiv

Yes I can see your argument. I always use around 50% FWD for my CCD (QSI 683), ( what QSI recommended I believe). So I aim for around the 25,000e ish. It is a long time since I did any maths/statistics on shot and thermal noise. So I just aim for what seems to work. I would guess it may be different for the various makes of CCD depending on how they work. Things have changed greatly in the last 25 years or so since I did my first degree.

All I was pointing out is where carastro may have seen or gotten the 33% figure from. I use Maxim DL but have never tried their auto routine so far. I get too little time to waste messing about with experimenting. I beleive in" if it ain't broke, then don't fix it!"

Derek

[Physopto]

Hi Vlaiv

Yes I can see your argument. I always use around 50% FWD for my CCD (QSI 683), ( what QSI recommended I believe). So I aim for around the 25,000e ish. It is a long time since I did any maths/statistics on shot and thermal noise. So I just aim for what seems to work. I would guess it may be different for the various makes of CCD depending on how they work. Things have changed greatly in the last 25 years or so since I did my first degree.

All I was pointing out is where carastro may have seen or gotten the 33% figure from. I use Maxim DL but have never tried their auto routine so far. I get too little time to waste messing about with experimenting. I beleive in" if it ain't broke, then don't fix it!"

Derek

[Physopto]

OOPs

Something went wrong there

Just noticed that I should have said 25,000/2 for the flats and that is the minimum I use.

Edited January 26, 2020 by Physopto

[vlaiv]

4 minutes ago, Physopto said:

Hi Vlaiv

Yes I can see your argument. I always use around 50% FWD for my CCD (QSI 683), ( what QSI recommended I believe). So I aim for around the 25,000e ish. It is a long time since I did any maths/statistics on shot and thermal noise. So I just aim for what seems to work. I would guess it may be different for the various makes of CCD depending on how they work. Things have changed greatly in the last 25 years or so since I did my first degree.

All I was pointing out is where carastro may have seen or gotten the 33% figure from. I use Maxim DL but have never tried their auto routine so far. I get too little time to waste messing about with experimenting. I beleive in" if it ain't broke, then don't fix it!"

Derek

I figured that you just quoted potential source of 33% statement. On the other hand, I just wanted to point out that not all things should be taken as accurate / set in stone without running logic checks first.

This does not mean that there is no something else that will indeed make 33% recommendation better option for some CCDs out there that I've not taken into account. From what I've seen - figure of 80% works and it works well (I use it) - and logic behind taking flats supports it.

[Physopto]

I might try a figure closer to that next time out. I am at The Galloway Star Camp next month. So a few more hours spent trying out different levels would not come amiss.

 

[vlaiv]

9 minutes ago, Physopto said:

I might try a figure closer to that next time out. I am at The Galloway Star Camp next month. So a few more hours spent trying out different levels would not come amiss.

 

Simplest way to test if flats of different histogram peak level are working as they should is to do flat / flat calibration.

You take set at some peak values like 33% (to be reference value), 50%, 75%, 80%, 90% - do couple of each, and do couple of flat darks for each exposure. Create master flats and then divide master flats with each other (33%/50%, 33%/80%, etc) - you should get uniform gray sub (with noise obviously but it should not have any vignetting nor dust shadows evident).

[Physopto]

Well at least I can do all that in daylight. 🙂

I need to look into it further.

[AndyThilo]

Well I’m truly lost 😂

[vlaiv]

Just now, AndyThilo said:

Well I’m truly lost 😂

Don't be. Here is quick guide:

1. You've probably taken another set of darks? If not - do it, but take every precaution not to have light leak. Take camera off the scope, use camera cover and aluminum foil - place camera "face down" on desk and do darks.

As far as I can see deltaT is about 35C so you need to do this somewhere where you don't have heating (shed perhaps or basement or ....) as you need ambient temperature to be below 15C.

Compare that with the darks you have already taken - see if you get same gradient / mean ADU level.

2. Try calibrating data that you now have with existing flats (these seem ok) and new darks - just to see if it will work (I sort of doubt it - if there were a light leak, it affected both darks and lights) - if it does work properly - problem solved, if not - it is sign of light leak. This means that you need to examine your setup and change some things.

Light leak will usually come thru extension tubes, or adapters, or filter wheels or OAG. It can be diagnosed by using strong flash light and shooting short darks - which torch shining at different pieces of equipment. Highest mean ADU will tell you where likely leak is. 

[AndyThilo]

Camera is in the fridge as we speak cooling to -20 😂

[carastro]

Quote

Camera is in the fridge as we speak cooling to -20 

Might need to put your brain in there too, it must be fried by now.

Carole 

[AndyThilo]

Just about Carole haha

[AndyThilo]

Well well well. So I've 30 new darks fresh from the fridge. 180sec subs, 120gain, -20C, same as my lights. 

Comparison below. Left side is original dark and processed light master (no flats), right side is new dark master and new light master. Both processed in WBPP using identical settings and both had the same DBE process run (exactly the same sample generation settings).

Wow the new image is so much nicer! Background is clean, colours are better and zero gradients. I guess my original darks had a load of light leakage. 

[AndyThilo]

Conclusion, thank you @vlaiv, I will be deleting my whole dark library and making new ones :). And I really can't see how flats would improve this?

[vlaiv]

1 minute ago, AndyThilo said:

Conclusion, thank you @vlaiv, I will be deleting my whole dark library and making new ones :). And I really can't see how flats would improve this?

Good to see that there was only light leak in the darks and that proper darks took care of everything.

You now know that your flats are working properly. Only real issue with that flat panel is that it does not produce much red part of spectrum and your red histogram peak is quite low. It will not cause too much issues - but take more to compensate for this. Also, your color balance will be really thrown off if you don't normalize your flats prior to doing calibration with them.

[AndyThilo]

5 minutes ago, vlaiv said:

Good to see that there was only light leak in the darks and that proper darks took care of everything.

You now know that your flats are working properly. Only real issue with that flat panel is that it does not produce much red part of spectrum and your red histogram peak is quite low. It will not cause too much issues - but take more to compensate for this. Also, your color balance will be really thrown off if you don't normalize your flats prior to doing calibration with them.

The above was without flats. I’ll try with my flats tomorrow. I’ll see how they look with flats, if ok then I’ll order the filters from Gerd.

[vlaiv]

1 minute ago, AndyThilo said:

The above was without flats. I’ll try with my flats tomorrow. I’ll see how they look with flats, if ok then I’ll order the filters from Gerd.

you can try the old flats with these new darks once you have time for it. It'll cost you nothing except a bit of processing time - you don't need to retake them.