Flat Issues - Broken Flat Field Generator?

[AndyThilo]

Ok well either I'm doing something very wrong, or there's a problem with my flat field generator. So I created some new flats, all at gain 120, camera cooled to -20 and focus in same position as when I took the lights.I also created a fresh batch of BIAS files to use with the tests.

I took 30 subs each with the generator at full brightness (approx 12V), dimmed to 8V and dimmed to 5V. All at ADU values of 20000, 22000, 25000, 27000 & 30000. 

I then did some stacking in DSS and PI.

All processing in PI for comparison purposes was identical, ABE and SNCR. This was just to be able to get an idea on image quality.

In DSS I ran the same sequence with all 3 voltages @ ADU3000. All 3 resulting images looked virtually identical. So I then processed in PI. I then ran the same without flats.

In PI, I did a full manual integration as per the book Inside Pixinsight, and also used the BPP script. Again with and without flats.

6 images below show the following:

Top - DSS. Left with flats, right without flats

Middle - PI BPP. Left with flats, right without flats

Bottom - PI Full Manual Integration. Left with flats, right without flats

Again all 6 processed in PI with the Automatic Background Extractor and SNCR (green).

Conclusion, my flats are broke lol. My images are 100% cleaner with no flats and manual integration without flats the clear winner with DSS a close 2nd. For some reason BPP isn't removing my amp glow. No idea why, it used to! So I'm going to send this to the flat field generator supplier and manufacturer to advise. This is the generator I use:

https://www.astroshop.eu/flatfield-masks/gerd-neumann-jr-aurora-flat-field-panel-160mm-12v/p,46378

PI_DSS_Flats_Noflats_Compare by Andy Thilo, on Flickr

[Physopto]

I have the same Gerd Neumann flat field generator. I have never tried altering the voltage as I suspect it is better to use neutral density filters with it. I use 2x filter sheets (2 off) and vary the timing of the exposure. It can also be done with a white t shirt but again I don't think it is the best method. Just my opinion! You could try this anyway.

# Best of luck

Derek

Edited January 25, 2020 by Physopto

[AndyThilo]

All of the photos shown above were with full brightness, i.e connected directly to 12V. But looking at the comparisons I did in DSS, there was no difference in the output with flats at 12V, 8V and 5V (all at ADU 30000). Maybe the ND filter they sell is mandatory to even out the light field? I did buy one with the generator, but I soon creased it as it's so flimsy. I've emailed Gerd Neumann for some advice. 

 

[Physopto]

Yes they are flimsy but sandwiched between the flat field panel and the white dispersion cover they are fine. Easy to do. My flat field generator is fitted into a cut down an old plastic plant pot fitted with cut out foam so it fits over the scope dew shield.

Derek

[AndyThilo]

Oh it sandwiches between the diffusion panel and flat field panel? I thought it went on the outside... Might have to order another one. No idea which one to get though. What camera do you use and what exposure time/ADU? Any chance you could upload one of your flats so I can compare to mine?

Cheers

Andy

[vlaiv]

What is your calibration procedure and could you post one of these in fits format without any alteration

1. one light

2. one matching dark

3. one flat

4. one matching flat dark

 

[AndyThilo]

I posted it in my original post but DSS is just basic load them in and let it go. PI BPP I used Linear Fit for all, again let it run. For manual, I created Master dark/Bias. Calibrated flats with them then created master flat. Then calibrated lights with all the masters, followed by Cosmetic correction, debayering, star aligning and finally integrated the lights to their final master.

For tests without flats, I just left them out...

Here's a single light - https://1drv.ms/u/s!Ari3AWpbmLZ0gvwzUVAUpJrBlcKLww?e=EF9ptN

Flat - https://1drv.ms/u/s!Ari3AWpbmLZ0gvwyqruJs6BIvrZD6Q?e=2BETne

Dark - https://1drv.ms/u/s!Ari3AWpbmLZ0gvw0Tel6GxLMhRq8sw?e=b7Ct6L

Flat Dark - https://1drv.ms/u/s!Ari3AWpbmLZ0gvw18bGKf7pF8y7PBQ?e=zmnkQ1

[AndyThilo]

The manual process I followed exactly the process in Warrens book 'Inside Pixinsight'. It's definitely a flats issues as the results are replicable between DSS, PI BPP and PI manual  processes. 

[vlaiv]

1 hour ago, AndyThilo said:

I posted it in my original post but DSS is just basic load them in and let it go. PI BPP I used Linear Fit for all, again let it run. For manual, I created Master dark/Bias. Calibrated flats with them then created master flat. Then calibrated lights with all the masters, followed by Cosmetic correction, debayering, star aligning and finally integrated the lights to their final master.

For tests without flats, I just left them out...

Here's a single light - https://1drv.ms/u/s!Ari3AWpbmLZ0gvwzUVAUpJrBlcKLww?e=EF9ptN

Flat - https://1drv.ms/u/s!Ari3AWpbmLZ0gvwyqruJs6BIvrZD6Q?e=2BETne

Dark - https://1drv.ms/u/s!Ari3AWpbmLZ0gvw0Tel6GxLMhRq8sw?e=b7Ct6L

Flat Dark - https://1drv.ms/u/s!Ari3AWpbmLZ0gvw18bGKf7pF8y7PBQ?e=zmnkQ1

Light seems to be already calibrated? It's 32bit format but still 14bit values for some reason???

Could you provide one light sub straight out of camera? Or in case you are 100% sure this is it - what capture software did you use?

[vlaiv]

There is also an issue with your darks, or at least it seems so:

left is very stretched dark, while right is light sub without alterations stretched (one that you uploaded).

Dark has gradient over it, while light does not, although light is probably stretched more because it shows amp glow more than dark.

It could be that there was some sort of light leak when you took your darks. Could this be possible?

[AndyThilo]

All files are straight out of APT. Darks do have a slight issue with a mark on them. Not sure about light leaks, I did them outside with a black bin bag over everything, and lens cap of course. 

[Physopto]

I don't know if it is somehow my down load but the Flat in Maxim under stretch shows 3 distinct peaks very strange!

Derek

[carastro]

I am not familiar with either COMS Cameras, or aurora panels.  But I notice your ADU is 30,000.  I know some people do use that, but my understanding is it has to be 1/3 full well depth, and my Atik camera that is 65,000.  1/3 65,000 = 21666.  I always try to keep my flats around 22,000 - 24,000.

Have you used this panel successfully previously with this ADU?

Carole 

[AndyThilo]

 

1 minute ago, Physopto said:

I don't know if it is somehow my down load but the Flat in Maxim under stretch shows 3 distinct peaks very strange!

Derek

 

Yep there is definitely problem with flats. It was discussed on another forum...

Hi Andy,

the issue is definitely caused by the flat frames. You need to capture new flat frames.

It is conspicious that the red channel is very weak compared to the other channels. These are the mean values of the CFA channels of the MasterFlat:

CFA0 (red):    3109
CFA1 (green): 37843
CFA2 (green): 37840
CFA3 (blue):  23713

I suppose that you took sky flats -- this would explain why the red channel is that weak. Speaking from my own experience with the ASI294, there is no need to take long exposed flat frames. I took my flat frames with a not dimmed flatfield box and an exposure time of 0.003 s. However, it is not clear to me whether the weakness of the red channel is the actual problem.

By the way, why did you prepare the Master Calibration files with DeepSkyStacker?

Bernd

[AndyThilo]

2 minutes ago, carastro said:

I am not familiar with either COMS Cameras, or aurora panels.  But I notice your ADU is 30,000.  I know some people do use that, but my understanding is it has to be 1/3 full well depth, and my Atik camera that is 65,000.  1/3 65,000 = 21666.  I always try to keep my flats around 22,000 - 24,000.

Have you used this panel successfully previously with this ADU?

Carole 

Hi Carole

I haven't used this panel successfully at all yet LOL. But I was told to have the histogram peek just left of middle? 30000 was just an experiment. My previous flats (just as bad) were at 25000.

[vlaiv]

10 minutes ago, AndyThilo said:

All files are straight out of APT.

I'm not familiar with APT so I don't know if subs are only recorded as 32bit fits for some reason (it just wastes space) or has something been done to them. I wonder why only lights is recorded in this format?

Ah, sorry, my bad - it looks like I converted file to 32bit without realizing (I always do that with subs and must have done so by impulse when I opened it and then forgot about it).

All is fine with light sub!

12 minutes ago, AndyThilo said:

Darks do have a slight issue with a mark on them. Not sure about light leaks, I did them outside with a black bin bag over everything, and lens cap of course. 

You should first redo your darks - I suspect they are causing issues rather than your flat panel. It will cost you nothing to do that (maybe a bit of time - but it can be done indoors on a cloudy night so not much is wasted).

However, you should maybe change your flat panel - it has very odd distribution of light - one component is 1/10 of value of highest component - third histogram peak is very low at about 4000ADU.

Does your flat panel have distinct color cast to it (maybe bluish light or very warm?)

5 minutes ago, Physopto said:

I don't know if it is somehow my down load but the Flat in Maxim under stretch shows 3 distinct peaks very strange!

Derek

That is quite normal for OSC sensor - R, G and B components of sensor have different sensitivity and one always ends up with 3 distinct peaks when using color camera

5 minutes ago, carastro said:

I am not familiar with either COMS Cameras, or aurora panels.  But I notice your ADU is 30,000.  I know some people do use that, but my understanding is it has to be 1/3 full well depth, and my Atik camera that is 65,000.  1/3 65,000 = 21666.  I always try to keep my flats around 22,000 - 24,000.

Have you used this panel successfully previously with this ADU?

Carole 

Here flats are quite ok as far as saturation is concerned. Not sure where 1/3 rule came from - I've heard it before, but I don't think it is a good rule (unless someone can explain exactly why it is used). Aim at 80% or so histogram is better option. In fact any histogram value is good as long as you don't have low or high clipping. Higher histogram peak value only ensures that you have good SNR for your flats and that you won't be polluting your lights with noise much.

[vlaiv]

6 minutes ago, AndyThilo said:

Yep there is definitely problem with flats. It was discussed on another forum...

It is not necessarily a problem that flat source is weak in one component - it just means that flat panel is not producing very white light - it has color cast.

If red is weak - it just means that flat panel has very cool light (probably leds that try to emulate light of very high temperature >7000K or similar).

In fact - your flats seem fine otherwise and only thing that I see wrong with your data are in fact darks - it costs nothing to redo your darks, but this time be careful there is no light leak. Maybe take camera off the scope, cap it off and use aluminum foil and place "face down" on a desk while taking your darks.

[AndyThilo]

I’ll have to do them outside to get my -20, but I’ll get it running tonight and stick a bin bag over it as well for good measure. 

[carastro]

I don't agree with 80% Vlaiv, if the flat is too bright it doesn't work as I have found out to my cost in the early years.  Also if you remember Marmo was struggling with his flats recently and the only ones that works were those around the 1/3 range.  

Edited January 25, 2020 by carastro

[carastro]

Quote

I’ll have to do them outside to get my -20,

I don't think you need to cool flats, I never have, and only know one person who does.  Unless it is different for CMOS cameras. 

Carole 

[AndyThilo]

1 minute ago, carastro said:

I don't think you need to cool flats, I never have, and only know one person who does.  Unless it is different for CMOS cameras. 

Carole 

Sorry I was taking about darks

[vlaiv]

11 minutes ago, AndyThilo said:

I’ll have to do them outside to get my -20, but I’ll get it running tonight and stick a bin bag over it as well for good measure. 

It is definitely due to darks, here is (light - some_value)/flat:

It shows almost perfect flat calibration. I used 1100ADU as a base and kept increasing it until I got good calibration. I needed to remove dark current and bias for flat fielding to work properly so I needed to guess right amount to be subtracted from lights. In fact I ended up subtracting around 1800ADU.

You can still see amp glow as I in fact did not use dark sub at all.

13 minutes ago, carastro said:

I don't agree with 80% Vlaiv, if the flat is too bright it doesn't work as I have found out to my cost in the early years.

Can you give any explanation why it might not be working?

[carastro]

Sorry Vlaiv, I am not a technical person like you, I just do what works, and definitely I find if the flat is too bright it doesn't do it's job.   I was told 1/3 full well when I started imaging and that definitely works for me.  Also I have heard that's what most other people use.  I have never heard of any-one doing 80%.  

Carole 

[AndyThilo]

24 minutes ago, vlaiv said:

It is definitely due to darks, here is (light - some_value)/flat:

It shows almost perfect flat calibration. I used 1100ADU as a base and kept increasing it until I got good calibration. I needed to remove dark current and bias for flat fielding to work properly so I needed to guess right amount to be subtracted from lights. In fact I ended up subtracting around 1800ADU.

You can still see amp glow as I in fact did not use dark sub at all.

Can you give any explanation why it might not be working?

I don’t know how you’re doing that, what software? 

[vlaiv]

3 minutes ago, carastro said:

Sorry Vlaiv, I am not a technical person like you, I just do what works, and definitely I find if the flat is too bright it doesn't do it's job.   I was told 1/3 full well when I started imaging and that definitely works for me.  Also I have heard that's what most other people use.  I have never heard of any-one doing 80%.  

Carole 

Fair enough, let's not get bogged down in technical discussion - if 1/3 works for you (and it certainly should) no need to change anything.

24 minutes ago, AndyThilo said:

I’ll have to do them outside to get my -20, but I’ll get it running tonight and stick a bin bag over it as well for good measure. 

It looks like you have light leak of some sorts and it is not only present in darks - it is also there in lights as well. Until you address that, you won't be able to properly calibrate your images. Here is what I've found out:

Here is light / flat:

It shows "inverse" vignetting - and that is fine, that is what you would expect from flat fielding when dark has not been removed. Since dark has gradient on it, I will not be using exact dark frame, but I'll try to guess average value of dark frame and subtract that.

Here is (light-900ADU)/flat - My first guess is that dark needs to be 900ADU:

Maybe a bit better, but still over correction of flats - we need to subtract more. How about another 900ADU - 1800ADU in total:

That is better, and like I said above about 1800ADU is some signal not related to light coming from aperture. If we continue, we will start going into under correction regime instead of over correction.

This is about 2000ADU removed:

And this is 2100 ADU removed:

And 2110ADU:

There last three start to show under correction (edges and corners are darker then the rest of image).

But here is important bit - ADU that needs to be subtracted is around 1800ADU.

Average value of dark that has light leak is ~1174ADU. This means

a) real dark mean value is lower than that, since we know there was light leak - and light leak is going to raise mean value.

b) there is a light leak in lights as well - because we need to subtract way more than 1174 to get flat fielding to work properly.

How do you connect your camera to your scope and is there any light source near by when you record your images?