DLSR flats causing confusion

[tooth_dr]

There are so many variables here it might be hard to work out what is going on.

 

2 DSLRs

2 sets of calibration files

 

Firstly - the flats don't seem to be doing what they are intended to do.  I've attached a 40d light and a 40d flat, and a 1000d light and a 1000d flat.  I've attached also the final stacked imaged from the calibrated subs using APT. (Each set of subs was calibrated for each camera).  Flats were taken the day after the lights, scope and camera not moved and focus wasn't touched.

Secondly - the flats are different colours.  Both DSLR cameras are modded, both 10MP, same sensor, I used the same settings for taking flats, both taken last night (ipad, paper, APT used to capture flats).  Any ideas?  This is more out of curiosity, but any ideas would be welcomed.   There seems to be more blue than red in the 40d which cant be right.  All images were shot in RAW, I just converted to JPEG to upload here.

 

Thanks in advance

Adam

 

 

[Oddsocks]

If it just the colour that is a problem Adam then that might be explained by differing white balances set in the two cameras. The white balance info gets recorded with the raws and some image processing programs read the embedded white balance data and use it to set the colour balance of the debayered raws. The solution is to set a custom white balance for each of your cameras in daylight by pointing the camera at a 50% grey card and tweaking the in-camera settings until you get a plain grey result with no tint.

However once you modify the camera by removing the built-in IR filter then your camera becomes overly sensitive to the red end of the spectrum and it might be that there is insufficient adjustment of the in-camera white balance settings to compensate, especially in daylight when there is a lot of IR about.

If you are using a clip-in LP filter in the cameras then these include blocking of the far IR and it would be easier to get a daylight/nightime custom balance adjusted.

You would have to try it and see.

If successful then after the custom white balance is enabled your night image shots and flats will be daylight balanced and only residual light pollution colour gradients should be visible.

Have a read of this web page, especially the part regarding spectral matching of flat illumination sources to the night time sky.

http://diffractionlimited.com/flat-fields-stray-light-amateur-telescopes/

 

[Oddsocks]

Forgot to mention too, as I have not kept up to date with the various permutations of astro modified cameras, that I seem to recall you could modify some of the Canon sensors in two stages. Remove the near IR filter only to allow the Ha band through and still be able to set a custom white balance in-camera to allow daytime conventional photography, or remove both near and far IR filters to allow Ha and SII through but at the expense of not being able to set a custom white balance for conventional daytime use because of insufficient range of the in-camera controls. I think there was a third option offered to replace both existing IR filters with a custom one that just squeezed through the Ha and SII bands but blocked enough IR to allow a custom white balance to be set in-camera, not really sure about that though, maybe just wishful thinking.

I wonder if this applies to your cameras and one of them has only the near IR filter removed giving the bluer image while the other camera has both near and far IR filters removed giving the redder image?

[tooth_dr]

28 minutes ago, Oddsocks said:

Forgot to mention too, as I have not kept up to date with the various permutations of astro modified cameras, that I seem to recall you could modify some of the Canon sensors in two stages. Remove the near IR filter only to allow the Ha band through and still be able to set a custom white balance in-camera to allow daytime conventional photography, or remove both near and far IR filters to allow Ha and SII through but at the expense of not being able to set a custom white balance for conventional daytime use because of insufficient range of the in-camera controls. I think there was a third option offered to replace both existing IR filters with a custom one that just squeezed through the Ha and SII bands but blocked enough IR to allow a custom white balance to be set in-camera, not really sure about that though, maybe just wishful thinking.

I wonder if this applies to your cameras and one of them has only the near IR filter removed giving the bluer image while the other camera has both near and far IR filters removed giving the redder image?

Thanks for the reply! I modded both myself, both have had same single piece of glass removed, only the front blue piece.

I’m more concern why dust bunnies aren’t removed?

 

Thanks!

[Oddsocks]

44 minutes ago, tooth_dr said:

I’m more concern why dust bunnies aren’t removed?

Which software are you using for image calibration, registration and integration?

Were the flats calibrated using bias-for-darks or bias and separate darks?

If you want to post a link to a Dropbox or Google Drive folder containing uncalibrated master bias, master flats and master dark (if applicable) and the registered, integrated but uncalibrated result of the lights, so four separate images in total, then l’m sure somebody will take a look at them over Easter. 

It’s really is almost impossible to make a diagnosis from compressed jpg’s in the forum post unless something obvious jumps out.

[carastro]

On the whole they don't look too bad to me, I can see dust in both flats, and the vignetting seems to have been dealt with.  But I can see a couple of faint dust motes on the last image, the one with the stacking corners, which I presume is both images stacked together.

Any chance of going back and labelling each image so we know which is which?

Its a while since I used a DSLR and models have "moved on" since, but I was recently helping a new Astro society member with DSLR imaging and his flats were not working at all.  He was using the flats plan on APT and it simply was taking too long an exposure.  We sat down together and experimented.  We ignored the flats plan and did single "shots", he switched to manual mode (I think I recall), and we experimented with different length exposures of less than a second until we got the histogram in the right place, i.e. 1/3 to less than half way across, this was completely different to the results that the APT flats plan was giving him, but now his flats work just fine.  As a newbie, he didn't really know what a flat should look like and was trusting the flats plan to get it right for him.

Hope this may perhaps ring some bells and might help.

Finally, when you stacked the subs from the different cameras together, what method did you use?  I am sure you know this, but just in case, the flats need to be applied to the relevant set, and the images only combined together afterwards.  If using DSS you can either use the "batch" mode or stack the images separately with their relevant calibration frames, and then stack the stacks. 

Carole 

 

[tooth_dr]

10 minutes ago, Oddsocks said:

Which software are you using for image calibration, registration and integration?

Were the flats calibrated using bias-for-darks or bias and separate darks?

If you want to post a link to a Dropbox or Google Drive folder containing uncalibrated master bias, master flats and master dark (if applicable) and the registered, integrated but uncalibrated result of the lights, so four separate images in total, then l’m sure somebody will take a look at them over Easter. 

It’s really is almost impossible to make a diagnosis from compressed jpg’s in the forum post unless something obvious jumps out.

Thanks for taking the time to reply

1) I am using APP for image calibration, reg and integration.

2) I am not calibrating my flats, I made a master flat from 30 flats, and a master bias from 30 bias, these were used as is in APP.  I am not using darks, or flat darks.  My flat exposure time was 1/30s for both cameras.

3) I'll try to post these images below.  They are labelled if you hover over them with the mouse ( I uploaded them in this order - 40d BIAS / 40d FLAT / 1000d BIAS / 1000d FLAT)

 

Thanks for your patience.

MB-ISO_gain_800.0-exp_1.25E-4s-29subs-CANON_EOS_40D-SC_1_3.0-med.fits

MF-ISO_gain_800.0-exp_0.005556s-30subs-CANON_EOS_40D-SC_1_3.0-1xbl-med.fits

MB-ISO_gain_800.0-exp_2.5E-4s-30subs-CANON_EOS_1000D-SC_1_3.0-med.fits

MF-ISO_gain_800.0-exp_0.008s-30subs-CANON_EOS_1000D-SC_1_3.0-1xbl-med.fits

[tooth_dr]

10 minutes ago, carastro said:

On the whole they don't look too bad to me, I can see dust in both flats, and the vignetting seems to have been dealt with.  But I can see a couple of faint dust motes on the last image, the one with the stacking corners, which I presume is both images stacked together.

Any chance of going back and labelling each image so we know which is which?

Its a while since I used a DSLR and models have "moved on" since, but I was recently helping a new Astro society member with imaging and his flats were not working at all.  He was using the flats plan on APT and is simply was taking too long an exposure.  We sat down together and experimented.  He switched to mamual mode (I think I recall), and we experimented with different length exposures of less than a second until we got the histogram in the right place, i.e. 1/3 to less than half way accross, this was completely different to the results that the APT flats plan was giving him, but now his flats work just fine.  As a newbie, he didn't really know what a flat should look like and was trusting the flats plan to get it right for him.

Hope this may perhaps ring some bells and might help.

Carole 

 

The images are as follows:

1) flat 40d

2) flat 1000d

3) single light sub 40d

4) single light sub 1000s

5) calibrated stack of 40d + 1000d

 

The flats are 1/30 seconds which seems ok?  On screen camera histogram was about 1/2 or just over it.  This is the first time my flats have left big dust marks, but it's also the first time I've used APP, and also the first time I've stacked separate camera images at the same time.

[carastro]

OK, well that all seems to be more or less OK, histogram just slightly too high but as dust can be seen on the flats I think they are probably OK. 

Have you stacked and calibrated the subs for each camera separately as this might help narrow things down.  Do the flats seem to work on the separate camera stacks as you've only posted single subs above?

Carole 

[tooth_dr]

13 minutes ago, carastro said:

OK, well that all seems to be more or less OK, histogram just slightly too high but as dust can be seen on the flats I think they are probably OK. 

Have you stacked and calibrated the subs for each camera separately as this might help narrow things down.  Do the flats seem to work on the separate camera stacks as you've only posted single subs above?

Carole 

I've not tried them separately, I will try this out next.  I'm currently stacking uncalibrated subs from both camera.  Despite an i7 laptop, it takes some time to stack over 100 subs 

[tooth_dr]

I've been going over the subs, and this seems odd, dust appearing after the meridian flip.  Perhaps with the camera at a different angle, the shutter has moved some dust?  This is the 1000d

Top image = before meridian flip

Bottom image = after meridian flip

All subsequent images for the rest of the session are like the bottom image.

[Oddsocks]

8 minutes ago, carastro said:

well that all seems to be more or less OK, histogram just slightly too high

Thats odd Carole, I have opened in PixInsight the bias and flats fits images that Adam posted but see very different results:

I have attached an image of the PI desktop, using standard linked channels in the STF tool and the first thing I notice is that there is very little difference in colour of the debayered frames between the 40d and 1000d.

The second strange thing is how low the flats exposure looks in P.I. Measuring at the same pixel location in the flats and bias frames of the 40d, 3783 and 2481, bottom right, P.I. gives the reading for pixel ADU as tabled:

                   Red               Green              Blue

40d  flat     0.0896             0.0791             0.0791

40d bias     0.0630             0.0629             0.0628

In P.I at least there seems very little difference between the bias background and the flats exposure, a score of 1.000 being saturation in PI for this image.

In the attached image of the desktop the histogram window is reading the flats exposure for the 1000d on the lower right of the desktop, the 1000d is very similar to the 40d, and you can see the 1000d's flat histogram is way too far to the left.

The last odd thing that stands out is that APP seems to have got the sensor size wrong, I really don't know APP at all but looking at the image for the bias frames, pixels are recorded across the full width and height of the image but in the flats exposures there is a black frame top and bottom of dark pixels. It seems that APP has included the normally unseen dark rows and columns at the edge of the sensor that are used for auto dark sensing and shift registers.

I know that including these dark pixels when calibrating lights with flats in some programs causes a bad calibration, but I dont know how APP handles these, it might be happy and it might not.

It could be that this disparity between the total pixel count in the APP flats and biases and the actual active pixel count of the lights is the reason that Adam can't achieve a good result?

William.

[tooth_dr]

William, I am very grateful for this.  I'm did notice a difference in the pixel sizes too, but I had no idea, your explanation is very good.

I'm currently stacking uncalibrated flats here.

[Oddsocks]

Adam,

The other half is working on call over Easter and we are turning in for the night, there are only restricted rail services tomorrow so I have to get up and drive her up to London at 5am! 

I will log off for tonight and try to catch up later tomorrow and see how things are progressing.

William.

[tooth_dr]

Thank you so much William.

[tooth_dr]

I found this - looks odd.  After the meridian flip with the 40d the M51 image has been recorded rotated but the dust bunnies are in the same place in both images.  This might be throwing off APP?

 

[carastro]

Thanks for that analysis Oddsocks, I am afraid I am not a very analytical person in statistics, I can only judge by what I see, so I will leave you to that side of things.

One thought that has struck me reading Tooth's more recent posts is the meridian flip, it can often catch people out and even after all my years imaging I still have to think about it.   Think very hard about the image and the dust bunnies being upside down and whether you are flipping the images to stack and calibrate and does the target land in the same place on both sides of the meridian. 

It still makes my brain hurt after many years imaging.  So to be on the safe side I tend to stack and calibrate separately and then stack the stacks. 

Carole 

[tooth_dr]

I hadnt turned off autorotate on the 40d, thought I had done it.  This could be an issue but how can the image be rotated but the dust not?

[carastro]

The dust should rotate with the camera, but I was just wondering whether the flip has anything to do with the dust not lining up, still trying to get my own head around it.

Say for instance your target position does not match on the flip, well the stacking software will line it up but the flats will only be one set I presume, so if one image is shifted over, it won't line up with using the one set of flats. 

Am I talking nonsense?  I always calibrate each side of the flip separately to be on the safe side. 

Carole 

[tooth_dr]

Carole I honestly don't know either 

[carastro]

Might be worth trying stacking them separately to see if it makes any difference.  Then I just stack the two Tiffs (already calibrated) into one image.

Carole 

[geoflewis]

53 minutes ago, carastro said:

Say for instance your target position does not match on the flip, well the stacking software will line it up but the flats will only be one set I presume, so if one image is shifted over, it won't line up with using the one set of flats.

Hi Carole, isn't calibration performed before alignment/stacking, it certainly is with my processing software, so it doesn't mater if the stars shifted after merdian flip or for any other reason. The dust/other artifacts will always be in the same location on the images unless it physically moved on the sensor/lens, even though the stars may not be, so calibration prior to alignment and stacking should correct the images whether the images are upside down, or not, or slightly rotated, etc. Of course if you rotate the light frames before calibration then the flats will not work as the orientation of the light frames camera will be different to the flats. Well that's how I understand it anyway...

Regards, Geof

[Stub Mandrel]

16 hours ago, tooth_dr said:

The images are as follows:

1) flat 40d

2) flat 1000d

3) single light sub 40d

4) single light sub 1000s

5) calibrated stack of 40d + 1000d

How are you doing the stack?

In DSS you need to keep the flats in the same groups as the lights they apply to.

If a 100D flat gets applied to a 40D light or vice versa it will not work properly.

 

[tooth_dr]

Hi Neil

 

I havent posted any results DSS yet, but I am currently stacking the 1000d subs to see how it compares.  I cant use these both together in DSS, as you have suggested, because for some reason when it opens the files from the two cameras in DSS there have a different pixel size by 2px in the length so it refuses to stack them.  Yet in APP it sees them both as the same pixel size.

 

[tooth_dr]

42 minutes ago, geoflewis said:

Hi Carole, isn't calibration performed before alignment/stacking, it certainly is with my processing software, so it doesn't mater if the stars shifted after merdian flip or for any other reason. The dust/other artifacts will always be in the same location on the images unless it physically moved on the sensor/lens, even though the stars may not be, so calibration prior to alignment and stacking should correct the images whether the images are upside down, or not, or slightly rotated, etc. Of course if you rotate the light frames before calibration then the flats will not work as the orientation of the light frames camera will be different to the flats. Well that's how I understand it anyway...

Regards, Geof

One of the cameras was set to autorotate, so I don't know what this means to my final images?  Is the orientation of the sensor known/labelled, such that the software will apply the flats and bias always in the right way, even if you rotate the images before stacking?   If that is the case then calibration will be applied correctly to all my frames, and the calibrated lights will simply be rotated to suit in the stacking/intregration process