Problem with flats mono CCD

[tooth_dr]

I processed a few subs of Leo on Wednesday night, using the frames listed below.  The camera is a QHY9 mono with the KAF-8300 mono sensor.  I used Astrophotography Tool (APT) for data collection and Astro Pixel Processor (APP) for processing.  I used the 'CCD aid tool' in APT to calculate a mean ADU of about 26k for my flats.  This resulted in an exposure of 8 seconds.  In my stacked image from APP, the outside of the image is bright and some dust bunnies are visible.  Any suggestions of what to try next?

LIGHTS: 24 x 300s

DARKS: 30 x 300s

FLATS: 30 x 8s

DARK FLATS: 30 x 8s

BIAS: 30 x 0.01s

[LightBucket]

1 hour ago, tooth_dr said:

I processed a few subs of Leo on Wednesday night, using the frames listed below.  The camera is a QHY9 mono with the KAF-8300 mono sensor.  I used Astrophotography Tool (APT) for data collection and Astro Pixel Processor (APP) for processing.  I used the 'CCD aid tool' in APT to calculate a mean ADU of about 26k for my flats.  This resulted in an exposure of 8 seconds.  In my stacked image from APP, the outside of the image is bright and some dust bunnies are visible.  Any suggestions of what to try next?

LIGHTS: 24 x 300s

DARKS: 30 x 300s

FLATS: 30 x 8s

DARK FLATS: 30 x 8s

BIAS: 30 x 0.01s

Hi,

I am just getting to grips with APP, so can’t really help at this point, but am looking for some CCD data  to test with as my own data does not have flats I have only used darks till now...so if you are willling to let me Have a go with your data, maybe I can help...

Have you not got your Atik 383 anymore..?

[x6gas]

It sounds like your flats are over-correcting for some reason.  I'm not familiar with APP but can you post one of your flats?

Have you tried processing the image without flats?

[vlaiv]

If both outer field and dust shadows are brighter - that is over correction.

calibrated = light / flat

If calibrated is larger in value, from above simple equation we have two possibilities - either light is larger than it should be or flat is lower than it should be.

Light can be higher under these conditions:

1. dark or bias (or both) is not removed, so there is residual signal besides light signal

2. there is source of light pollution which is not present when doing darks / bias files - such as scope having a light leak and you take camera off for dark / bias frames

3. Darks were "colder" than light subs.

Flat can be lower that it should be if:

- not properly calibrated (which I doubt since you took same duration flat darks under same conditions)

- flats were taken when sensor was cooled but matching darks were taken with hotter sensor.

- sensor is in non linear region when doing flats - again I doubt this since you went for ADU at about half way and in general non linearity would produce host of artifacts with calibration.

Here are some questions that could help to understand what is happening:

1. Was temperature regulation same in all corresponding subs?

2. How did you acquire subs (on scope / off scope / day / night - particular conditions for each)

3. Exact calibration method used. Maybe try without bias - it's not needed for fully matching darks.

[tooth_dr]

1 hour ago, LightBucket said:

Hi,

I am just getting to grips with APP, so can’t really help at this point, but am looking for some CCD data  to test with as my own data does not have flats I have only used darks till now...so if you are willling to let me Have a go with your data, maybe I can help...

Have you not got your Atik 383 anymore..?

Thanks William. I can upload data if you want no problem.  I still have the Atik.

[tooth_dr]

42 minutes ago, x6gas said:

It sounds like your flats are over-correcting for some reason.  I'm not familiar with APP but can you post one of your flats?

Have you tried processing the image without flats?

I have not tried this, I will try that later and also post images. Thank-you.

[Xiga]

2 hours ago, tooth_dr said:

I processed a few subs of Leo on Wednesday night, using the frames listed below.  The camera is a QHY9 mono with the KAF-8300 mono sensor.  I used Astrophotography Tool (APT) for data collection and Astro Pixel Processor (APP) for processing.  I used the 'CCD aid tool' in APT to calculate a mean ADU of about 26k for my flats.  This resulted in an exposure of 8 seconds.  In my stacked image from APP, the outside of the image is bright and some dust bunnies are visible.  Any suggestions of what to try next?

LIGHTS: 24 x 300s

DARKS: 30 x 300s

FLATS: 30 x 8s

DARK FLATS: 30 x 8s

BIAS: 30 x 0.01s

Hi Adam

I think @vlaiv is right about the Bias files. I don't use Darks myself with my DSLR (although I keep meaning to run some more tests on that) however, it is my understanding that if you are using Dark Calibration files, then you should not also use Bias files, as I think it will remove the dark current twice. So it's probably over-correcting at the moment.

Try without the Bias files and see how it goes.

[tooth_dr]

34 minutes ago, vlaiv said:

If both outer field and dust shadows are brighter - that is over correction.

calibrated = light / flat

If calibrated is larger in value, from above simple equation we have two possibilities - either light is larger than it should be or flat is lower than it should be.

Light can be higher under these conditions:

1. dark or bias (or both) is not removed, so there is residual signal besides light signal

2. there is source of light pollution which is not present when doing darks / bias files - such as scope having a light leak and you take camera off for dark / bias frames

3. Darks were "colder" than light subs.

Flat can be lower that it should be if:

- not properly calibrated (which I doubt since you took same duration flat darks under same conditions)

- flats were taken when sensor was cooled but matching darks were taken with hotter sensor.

- sensor is in non linear region when doing flats - again I doubt this since you went for ADU at about half way and in general non linearity would produce host of artifacts with calibration.

Here are some questions that could help to understand what is happening:

1. Was temperature regulation same in all corresponding subs?

2. How did you acquire subs (on scope / off scope / day / night - particular conditions for each)

3. Exact calibration method used. Maybe try without bias - it's not needed for fully matching darks.

Thanks vlaiv

What you are saying is very helpful and this will hopefully get me to the bottom of the problem.

1. All subs were taken at -15oC. The camera sometimes fluctuates to -16oC but 1/2 a degree should make much difference.

2. Lights - night time, luminance filter

Darks - on scope in shed at night time with end cap on

Bias - on scope in shed at night time with end cap on 

Flats - taken using a computer monitor set to white page, four sheets of white paper over end dew shield, held by a cardboard holder, scope about 40cm from screen, at right angles. Also used same luminance filter.

Flat Darks - on scope in shed at night with end cap on

Camera not moved between lights and flats.

 

[tooth_dr]

3 minutes ago, Xiga said:

Hi Adam

I think @vlaiv is right about the Bias files. I don't use Darks myself with my DSLR (although I keep meaning to run some more tests on that) however, it is my understanding that if you are using Dark Calibration files, then you should not also use Bias files, as I think it will remove the dark current twice. So it's probably over-correcting at the moment.

Try without the Bias files and see how it goes.

I will try this later Ciaran. Thanks ??

I just assumed APP would do what was needed automatically, and use them appropriately 

Edited February 8, 2019 by tooth_dr

[Xiga]

49 minutes ago, tooth_dr said:

I will try this later Ciaran. Thanks ??

I just assumed APP would do what was needed automatically, and use them appropriately 

That's a good point Adam. APP is now 'smart', so it is supposed to handle the files automatically for you. I still manually select mine one batch at a time (just because I've been doing it that way since the earlier versions, and, well, I'm on old dog, lol) so I don't know how well this functionality works. If you throw it files it doesn't actually need, will it ignore them? Or does it just know how to handle all the myriad calibration files, assuming you've selected all the right ones to begin with? Not sure tbh.

Another tip for you regarding APP - you don't need to finish a stack to know if calibration is working or not. As soon as you've finished Step 2 (Calibration) you can select any Light sub from the list at the bottom, and change the drop-down box at the top of the screen from 'Linear' to 'l-calibrated'. This will show you what the Calibrated Light sub looks like, and it should be obvious if it's working or not.

If the Bias files don't solve the problem for you - thinking outside the box here - you definitely haven't mixed up your Atik and QHY flats by any chance?

[tooth_dr]

28 minutes ago, Xiga said:

That's a good point Adam. APP is now 'smart', so it is supposed to handle the files automatically for you. I still manually select mine one batch at a time (just because I've been doing it that way since the earlier versions, and, well, I'm on old dog, lol) so I don't know how well this functionality works. If you throw it files it doesn't actually need, will it ignore them? Or does it just know how to handle all the myriad calibration files, assuming you've selected all the right ones to begin with? Not sure tbh.

Another tip for you regarding APP - you don't need to finish a stack to know if calibration is working or not. As soon as you've finished Step 2 (Calibration) you can select any Light sub from the list at the bottom, and change the drop-down box at the top of the screen from 'Linear' to 'l-calibrated'. This will show you what the Calibrated Light sub looks like, and it should be obvious if it's working or not.

If the Bias files don't solve the problem for you - thinking outside the box here - you definitely haven't mixed up your Atik and QHY flats by any chance?

Thanks Ciaran.  My calibrated subs look 'ok, certainly pretty flat to my eye, but if I look hard I can see the dust bunnies in some of them.  That was the first thing I checked and prompted me to post on here! However the stacked result has a very pronounced defect!  But this thread is useless without PICTURES!  When I mean APP would use them automatically, I just did one session and one camera, and added the above files.  No other cameras, or sessions were part of the files, I'll try that again without Bias and then without Flats to see what happens

[LightBucket]

1 hour ago, tooth_dr said:

Thanks William. I can upload data if you want no problem.  I still have the Atik.

Hi,

Yes that would be great, I have never bothered with flats, so have none for my images, and wanted some true data for testing this new software, can’t find anyon the Internet, unless you or anyone knows of any available on the net...?

[tooth_dr]

14 minutes ago, LightBucket said:

Hi,

Yes that would be great, I have never bothered with flats, so have none for my images, and wanted some true data for testing this new software, can’t find anyon the Internet, unless you or anyone knows of any available on the net...?

No bother, will upload it later.  It's quite nice data too ?

[Adam J]

5 hours ago, tooth_dr said:

I processed a few subs of Leo on Wednesday night, using the frames listed below.  The camera is a QHY9 mono with the KAF-8300 mono sensor.  I used Astrophotography Tool (APT) for data collection and Astro Pixel Processor (APP) for processing.  I used the 'CCD aid tool' in APT to calculate a mean ADU of about 26k for my flats.  This resulted in an exposure of 8 seconds.  In my stacked image from APP, the outside of the image is bright and some dust bunnies are visible.  Any suggestions of what to try next?

LIGHTS: 24 x 300s

DARKS: 30 x 300s

FLATS: 30 x 8s

DARK FLATS: 30 x 8s

BIAS: 30 x 0.01s

In general I don't think that you would use both dark flats and bias at the same time, in so far as I am aware you use one or the other and for CCD's the choice is BIAS. Using both might be tripping APP up a little. 

[Thalestris24]

Hi

It might be worth double checking that the adu of your flats/master flat is what you think it is. You can do that quickly and easily in Fits Liberator - just check the mean raw input value from the right hand side. 

Louise 

[tooth_dr]

34 minutes ago, Adam J said:

In general I don't think that you would use both dark flats and bias at the same time, in so far as I am aware you use one or the other and for CCD's the choice is BIAS. Using both might be tripping APP up a little. 

Hi Adam.  The final integrated image is actually quite bright, so this ties in with that, it's like the overall brightness has been boosted.

[tooth_dr]

13 minutes ago, Thalestris24 said:

Hi

It might be worth double checking that the adu of your flats/master flat is what you think it is. You can do that quickly and easily in Fits Liberator - just check the mean raw input value from the right hand side. 

Louise 

Hi Louise, thanks I checked and it's 26k.  I even retook new flats and the problem was still there.

[Thalestris24]

2 minutes ago, tooth_dr said:

Hi Louise, thanks I checked and it's 26k.  I even retook new flats and the problem was still there.

Ok. Um, you have a gain and offset setting for the qhy9? I take it your settings allow up to 65535 at saturation? I guess it would have to be way out to affect the normal dynamic range used in flats but maybe something odd is going on!

Louise

[tooth_dr]

I've attached a few integrations here.  The image looks smoother without the flat, not sure what to make of this all!  Any help appreciated.

ALL CALIBRATION FRAMES

 

 

NO BIAS

 

 

NO FLATS

 

 

ALL CALIBRATION FRAMES BUT ONLY THE BEST 50% LIGHTS

 

And here are the masters:

BIAS

 

 

DARK

 

 

DARK FLAT

 

 

FLAT

 

 

 

 

Edited February 8, 2019 by tooth_dr

[vlaiv]

Ah, that is opposite from what I assumed - this is under correction in flats.

If we go to original formula calibrated = lights / flats

and calibrated value is lower than it should be - we have two possible cases:

1. lights are lower in value than they should be - this can happen from improper calibration (like multiple darks / bias subtraction, but I don't think it is the case)

2. Flats are "stronger" than they should be (higher in value) - this can happen if one is for example calibrating flats with bias only and there is significant dark current, but I don't think it's the case here.

Another reason why flats might be stronger, and this would be my main suspicion in this case - flocking and baffling of telescope - some unfocused light is making it's way to the sensor when using flats, bypassing "regular" optical train. Depending on scope type, this can happen with:

Newtonian scopes if there is reflection from tube walls opposite focuser that end up going down the focuser tube, it can happen because reflection of secondary support that ends up in the tube. Focuser tube needs to be baffled as well, if it's not - it can "channel" light inside (multiple bounces coming directly from flat source).

For folded designs with central obstruction it would depend on central obstruction fully covering aperture - if you look at the back of your scope you should not be able to see the light coming straight from aperture - it needs to bounce of secondary.

Refractors should be pretty immune to this - all the light that enters telescope goes thru the lens, so there is simply no chance of it being unfocused and reaching sensor.

[tooth_dr]

Just now, vlaiv said:

Ah, that is opposite from what I assumed - this is under correction in flats.

If we go to original formula calibrated = lights / flats

and calibrated value is lower than it should be - we have two possible cases:

1. lights are lower in value than they should be - this can happen from improper calibration (like multiple darks / bias subtraction, but I don't think it is the case)

2. Flats are "stronger" than they should be (higher in value) - this can happen if one is for example calibrating flats with bias only and there is significant dark current, but I don't think it's the case here.

Another reason why flats might be stronger, and this would be my main suspicion in this case - flocking and baffling of telescope - some unfocused light is making it's way to the sensor when using flats, bypassing "regular" optical train. Depending on scope type, this can happen with:

Newtonian scopes if there is reflection from tube walls opposite focuser that end up going down the focuser tube, it can happen because reflection of secondary support that ends up in the tube. Focuser tube needs to be baffled as well, if it's not - it can "channel" light inside (multiple bounces coming directly from flat source).

For folded designs with central obstruction it would depend on central obstruction fully covering aperture - if you look at the back of your scope you should not be able to see the light coming straight from aperture - it needs to bounce of secondary.

Refractors should be pretty immune to this - all the light that enters telescope goes thru the lens, so there is simply no chance of it being unfocused and reaching sensor.

This was my fault for describing it wrong in my first post. Sorry about that vlaiv.

These were taken with an ED80 refractor.

Im retaking some of the calibration data now as I’ve spotted the fits header is the darks and bias was different ie it says QHY9-CCD whereas the flats and dark flats just say QHY9 (they were taken a couple of days ago).

[Thalestris24]

I also wonder if there's a light leak which is messing everything up.? The calibration frames which should be dark, aren't.

Louise

[vlaiv]

Do you have any software to check all fits files that you've taken - like inspect Fits headers and pixel values - that can be important clue if something was recorded wrong.

This can also be the case - like silly truncation to 8bit data or something like that?

Maybe if you post one of each - fits files I mean, just a single sub, one of each - no need to post masters or whole set.

If it is ED80 - then light leak is not likely suspect - refractors don't have these issues (unless light leak is on the other side - after focuser, but that is a long shot - one would need to have opening in optical train that would be lit up when taking flats).

[tooth_dr]

@vlaiv

Here are the single fits, the flats and dark flats were taken on a few days ago, the darks on Christmas night (not sure what I was doing taking darks then lol)

 

 

 

 

DF_2019-02-03_22-55-06_Bin1x1_8s__-15C.fit

F_2019-02-03_20-22-36_Bin1x1_8s__-16C.fit

D_2018-12-25_20-31-51_Bin1x1_300s__-15C.fit

[tooth_dr]

Here is a new dark taken tonight, do they look any different to the above dark?

I've attached a new bias too

B_2019-02-08_20-39-50_Bin1x1_0.001s__-16C.fit

D_2019-02-08_20-17-38_Bin1x1_300s__-16C.fit