Lightbox for Flats

[Marmo720]

10 minutes ago, RolandKol said:

if histogram was around 50%, that will do the job, - but simply make more of them... not 5 but 50.

Ok good. We have clear skies tonight so will try a few sets of 50 flats and see what they look like when combined with relevant lights.

Thanks

[vlaiv]

I'm afraid these flats are a mess on multiple levels

First let's start by the fact that again - all flats are underexposed by quite a bit.

Here are full range histograms of each of these flats (or rather master flats made from 5 flats of each group):

All are underexposed as you can see - they are not even at 10% or so and they should be at 80-85%. This means that you need to increase flat exposure by factor of about 8 times. I see that you had 1/8 second exposure for one set of flats - why don't you try 1s exposure next to see if you will properly expose flat. Use flat panel when doing flats.

Flat panel works better than iPad - here is what average value of flat exposure across flats looks like (flats 1-5):

Very small variation in mean value (good).

For comparison - same measurement for iPad (frames 21-25):

Much more variation between numbers - as if panel is flickering and producing different intensity light.

Histogram for first master flat (flats 1-5) looks as it should (except for being under exposed):

So does second master flat (flats 6-10), but third one (flats 11-15) looks different:

As if color balance changed or light source changed.

Even stranger looks iPad histogram (flats 16-20) as it shows only two peaks:

so does second iPad histogram (flats 21-25).

Out of all of these flats, flat-flat calibration is worth trying only on first two master flats and last two master flats (two of flat panel, and two of iPad):

Flat-Flat calibration (with missing bias, but let's do it anyway) of first two master flats looks like this:

That actually looks rather decent. Yes there are dust bunnies all over the place (I'll explain cause of that later) - but it is actually rather flat image (look at histogram, it is bell shaped and looking ok). Here is what it looks like filtered for noise:

There is very small gradient visible - if I stretch it it will look horrible like:

But these are variations less than 1% in intensity (from -0.5% to 0.17%). Due to fact that we did not remove bias - this is rather good.

Now second flat-flat calibration:

This one has obvious gradient and I would not call it success. Here it is smoothed out:

This gradient is now 3% wide compared to above less than 1%, so it is noticeable.

In the end, I would say that something is very wrong with the way you are taking your flats - look at this animation - it shows differences between flats of same settings (animation shows stretched flats 1-5):

Dust shadows are dancing around! Even large doughnut in the corner seems to be shifting between subsequent subs. What could be shaking your setup so hard that dust particles are dancing around? This happens on all the frames that you uploaded - some have less of shake and some more.

If you are manually triggering your camera - maybe set 2s delay (or even 10s delay) to let your scope and mount settle. Maybe look into mirror lock so it does not move out of the way each time and cause vibration. How firm is your setup? Is scope/camera connection tight and firm? How is flat panel attached to the scope? Do you hold it against the scope yourself? It would be best if you had means to mount everything without holding anything yourself and not touching either scope or camera (remote shutter release).

[Marmo720]

11 minutes ago, vlaiv said:

I'm afraid these flats are a mess on multiple levels

First let's start by the fact that again - all flats are underexposed by quite a bit.

Here are full range histograms of each of these flats (or rather master flats made from 5 flats of each group):

All are underexposed as you can see - they are not even at 10% or so and they should be at 80-85%. This means that you need to increase flat exposure by factor of about 8 times. I see that you had 1/8 second exposure for one set of flats - why don't you try 1s exposure next to see if you will properly expose flat. Use flat panel when doing flats.

Flat panel works better than iPad - here is what average value of flat exposure across flats looks like (flats 1-5):

Very small variation in mean value (good).

For comparison - same measurement for iPad (frames 21-25):

Much more variation between numbers - as if panel is flickering and producing different intensity light.

Histogram for first master flat (flats 1-5) looks as it should (except for being under exposed):

So does second master flat (flats 6-10), but third one (flats 11-15) looks different:

As if color balance changed or light source changed.

Even stranger looks iPad histogram (flats 16-20) as it shows only two peaks:

so does second iPad histogram (flats 21-25).

Out of all of these flats, flat-flat calibration is worth trying only on first two master flats and last two master flats (two of flat panel, and two of iPad):

Flat-Flat calibration (with missing bias, but let's do it anyway) of first two master flats looks like this:

That actually looks rather decent. Yes there are dust bunnies all over the place (I'll explain cause of that later) - but it is actually rather flat image (look at histogram, it is bell shaped and looking ok). Here is what it looks like filtered for noise:

There is very small gradient visible - if I stretch it it will look horrible like:

But these are variations less than 1% in intensity (from -0.5% to 0.17%). Due to fact that we did not remove bias - this is rather good.

Now second flat-flat calibration:

This one has obvious gradient and I would not call it success. Here it is smoothed out:

This gradient is now 3% wide compared to above less than 1%, so it is noticeable.

In the end, I would say that something is very wrong with the way you are taking your flats - look at this animation - it shows differences between flats of same settings (animation shows stretched flats 1-5):

Dust shadows are dancing around! Even large doughnut in the corner seems to be shifting between subsequent subs. What could be shaking your setup so hard that dust particles are dancing around? This happens on all the frames that you uploaded - some have less of shake and some more.

If you are manually triggering your camera - maybe set 2s delay (or even 10s delay) to let your scope and mount settle. Maybe look into mirror lock so it does not move out of the way each time and cause vibration. How firm is your setup? Is scope/camera connection tight and firm? How is flat panel attached to the scope? Do you hold it against the scope yourself? It would be best if you had means to mount everything without holding anything yourself and not touching either scope or camera (remote shutter release).

Thank you for the analysis. Yes I noticed the bunnies dancing too. 

My camera to telescope is pretty firm but I only have 1 second wait between flat exposure so will change to be longer and see if that fixes the issue. My flat panel is inside the original box with circle cut to allow the dew covers inside and block outside light. It rests on the dew cap when I take the flats so I don't touch any part of the imaging chain. 

The two flat-flat calibration, is the first one the flat panels (the good one) and the second set is the iPad? If so, I will stop using the iPad and stick with the lightbox.

Carole showed me how to do the flats manually so I can change the exposure time etc so will try 1sec instead of the current default APT plan. 

Thanks again.

[vlaiv]

2 minutes ago, Marmo720 said:

Thank you for the analysis. Yes I noticed the bunnies dancing too. 

My camera to telescope is pretty firm but I only have 1 second wait between flat exposure so will change to be longer and see if that fixes the issue. My flat panel is inside the original box with circle cut to allow the dew covers inside and block outside light. It rests on the dew cap when I take the flats so I don't touch any part of the imaging chain. 

The two flat-flat calibration, is the first one the flat panels (the good one) and the second set is the iPad? If so, I will stop using the iPad and stick with the lightbox.

Carole showed me how to do the flats manually so I can change the exposure time etc so will try 1sec instead of the current default APT plan. 

Thanks again.

Yes, first one is with light box and I would recommend that you use that one.

There is however concern about third set of flats - color balance issue (look how different histogram shape is for some reason). When doing AP it is best if you leave everything on "manual", don't use any auto features, don't color balance your data on camera and such - just shoot raw data. All that stuff can be done in processing and it should be done in processing after calibration is finished.

I have no idea why should things jump around so much after each sub, but they do. That is going to cause you problems later on in imaging as same things will happen with lights as well.

[carastro]

Quote

Carole showed me how to do the flats manually so I can change the exposure time etc so will try 1sec instead of the current default APT plan. 

I suggested this because you can have more control over the length of flat (I found this out helping some-one else using APT).  (In this person's case he ended up doing something like 1/86 sec if I recall rightly), 1sec might even be too long, it all depends on how much light you have in the light source.  The main thing when doing this manually is not to guess how long the sub is  likely to need to be, but to try different exposure lengths out checking the histogram at the same time.  1/3 to just under halfway seems to be the optimal point for DSLRs.  Once you think the histogram is in the right place, only then take a bunch of them.  

Carole 

[Marmo720]

1 hour ago, carastro said:

I suggested this because you can have more control over the length of flat (I found this out helping some-one else using APT).  (In this person's case he ended up doing something like 1/86 sec if I recall rightly), 1sec might even be too long, it all depends on how much light you have in the light source.  The main thing when doing this manually is not to guess how long the sub is  likely to need to be, but to try different exposure lengths out checking the histogram at the same time.  1/3 to just under halfway seems to be the optimal point for DSLRs.  Once you think the histogram is in the right place, only then take a bunch of them.  

Carole 

Thanks Carole. Interestingly, I had a look at my old M27 flats and each was around 1/100 sec and they worked much better (minus the strong dust donut). 

I also didn't have advanced bias/flats selected so couldn't change exposure length for flats until now so will play around and see what I can get tonight. Will share once I am done with tonight session.

In the meantime, I wish everyone happy holidays and enjoy this time with family and friends.

[Marmo720]

 

Happy new year all. Hope you all have good health and clear skies.

I took some flats last night which are linked to below. I don't have any matching lights, darks or bias so I have attached selection from different night. The camera was moved several times in between and so o wouldn't expect a nice image. Just checking if the flats are done correctly.

I started with automatic plan that I have used in the past where the exposure is calculated by the camera/APT. The dust bunnies jump around a lot as previously observed by @vlaiv. I then did 2 sets of manual exposure settings as suggested by @carastro and the dust bunnies don't move around as much/at all which was good to see. I did one manual with same exposure as the automatic so they can be checked side by side and also one at 1/100s so I can reach 75% on the histogram.

Any advise and help is much appreciated.

Lights, darks and bias from same night but different to flats date:

https://we.tl/t-Fkdf2TYAId?src=dnl

 

Automatic flats produced by APT plan:

https://we.tl/t-xr6A1hoQIQ?src=dnl

 

Manual flat plan at 1/160, which is same exposure length as the automatic plan:

https://we.tl/t-0AF8llWMZd?src=dnl

 

Manual flat plan at 1/100, which puts histogram at close to 75%:

https://we.tl/t-SQYlCTBRSe?src=dnl

 

Thanks,

Mo

[carastro]

I just tried out 5 of the 1/100sec manually done flats, and 5 subs of the original images, and am pleased to report there is no vignetting.  As I suspected the APT flats plan does not seem to work.

Of course the dust does not line up  because the camera has been moved, but the main problem here was the vignetting and we are all aware of dust not lining up if you move the camera. 

I tried the 1/160 of a second sub flats but they did not seem to work.  

Attached is the stretched image using 1/100 sec flats showing no vignetting.  (remember I only did 5 subs of lights and flats just to save time and no darks, so it is noisier than it needs to be.

Also below is the same image after gradient exterminator and a further stretch.

Assuming we have now sorted your flats, further improvements can be gained from:

a) Longer exposures (are you guiding yet? If so take 300secs).  

b) Gradient Exterminator will be a very useful plugin for Photoshop since like me you live in a LP location.

Meanwhile I am stacking the 1/160secs all light and full calibration.  = RESULT - THE FLATS DID NOT WORK ON THESE. 

Carole 

 

 

Edited January 1, 2020 by carastro

[Marmo720]

4 hours ago, carastro said:

I just tried out 5 of the 1/100sec manually done flats, and 5 subs of the original images, and am pleased to report there is no vignetting.  As I suspected the APT flats plan does not seem to work.

Of course the dust does not line up  because the camera has been moved, but the main problem here was the vignetting and we are all aware of dust not lining up if you move the camera. 

I tried the 1/160 of a second sub flats but they did not seem to work.  

Attached is the stretched image using 1/100 sec flats showing no vignetting.  (remember I only did 5 subs of lights and flats just to save time and no darks, so it is noisier than it needs to be.

Also below is the same image after gradient exterminator and a further stretch.

Assuming we have now sorted your flats, further improvements can be gained from:

a) Longer exposures (are you guiding yet? If so take 300secs).  

b) Gradient Exterminator will be a very useful plugin for Photoshop since like me you live in a LP location.

Meanwhile I am stacking the 1/160secs all light and full calibration.  = RESULT - THE FLATS DID NOT WORK ON THESE. 

Carole 

 

 

Thank you Carole - very happy to see the results. I could tell straight away that doing the flats manually was better as the dust bunnies stay in same place.

I am guiding but I assumed anything longer than 60 secs at our locations would result in higher noise and not much signal gain? You recommended going up to 300 sec at iso 800 even at our locations and on a dslr? I guess I could do trial run on next clear night with 180sec and 300 sec. 

 

[carastro]

Yes 300secs but use a LP filter.  I've got a CLS LP filter I hardly ever use now if you want to give it a try.  It's a clip in filter for DSLR cameras.  However I am not sure with the Advent of LED lighting whether the CLS filters are still any good and there may be better ones around, see what others think but as I say you are welcome to borrow it. 

These are images I have done from home some years ago with a Modified DSLR camera, all 300secs with CLS filter from Bromley (scroll down for the specs and click on image for larger version without the platesolving):

https://www.astrobin.com/2113/B/?image_list_page=6&nc=&nce=

https://www.astrobin.com/2150/?image_list_page=6&nc=&nce=

https://www.astrobin.com/2160/?image_list_page=6&nc=&nce=

https://www.astrobin.com/2164/?image_list_page=6&nc=&nce=

https://www.astrobin.com/15802/B/?image_list_page=6&nc=&nce=

Carole 

 

 

[Marmo720]

32 minutes ago, carastro said:

Yes 300secs but use a LP filter.  I've got a CLS LP filter I hardly ever use now if you want to give it a try.  It's a clip in filter for DSLR cameras.  However I am not sure with the Advent of LED lighting whether the CLS filters are still any good and there may be better ones around, see what others think but as I say you are welcome to borrow it. 

These are images I have done from home some years ago with a Modified DSLR camera, all 300secs with CLS filter from Bromley (scroll down for the specs and click on image for larger version without the platesolving):

https://www.astrobin.com/2113/B/?image_list_page=6&nc=&nce=

https://www.astrobin.com/2150/?image_list_page=6&nc=&nce=

https://www.astrobin.com/2160/?image_list_page=6&nc=&nce=

https://www.astrobin.com/2164/?image_list_page=6&nc=&nce=

https://www.astrobin.com/15802/B/?image_list_page=6&nc=&nce=

Carole 

 

 

Lovely images. I would love to borrow your CLS filter as I have seen a few on sale here and on ABS and would be good to know if it works or if I need to look elsewhere. 

So glad the flats look better and thank you for the manual advise. Excited for the next clear night

[carastro]

I suggest you do another image (twice), once with the CLS filter and once without and do a comparison.  My images were much more noisy without the CLS filter and rather poor data. 

I'll leave it to you to organise when you want to collect it, and in the meantime if you get a chance do a new image with correct flats and see if you can repeat the flats correctly again.  If your light source is not always the same brightness the length of flat will vary. 

Carole 

Edited January 1, 2020 by carastro

[carastro]

Mo came round to my house last night to collect the CLS filter I am lending him, and also I got him to bring his camera, scope, lightbox and laptop, and sat next to him while we tried out flats.  I was happy to see the manual method in APT seems to work at 1/100 sec with his lightbox at maximum brightness with the histogram approx 1/3 across, so I sincerely hope we have got to the bottom of his flats problems.  This was done using the CLS filter so he is set up for future imaging from S London. 

But he now needs to do an image using this flats plan, to double check as all the recent tests have been done after the camera was moved.

Fingers crossed.

Carole 

 

[Marmo720]

21 hours ago, carastro said:

Mo came round to my house last night to collect the CLS filter I am lending him, and also I got him to bring his camera, scope, lightbox and laptop, and sat next to him while we tried out flats.  I was happy to see the manual method in APT seems to work at 1/100 sec with his lightbox at maximum brightness with the histogram approx 1/3 across, so I sincerely hope we have got to the bottom of his flats problems.  This was done using the CLS filter so he is set up for future imaging from S London. 

But he now needs to do an image using this flats plan, to double check as all the recent tests have been done after the camera was moved.

Fingers crossed.

Carole 

 

Thanks Carole! Lovely home and nice seeing your setup.

I got all the gear setup now and just waiting for that clear night to test it all

[carastro]

Fingers crossed, with good flats, guiding, longer subs and the CLS filter you will be getting some nice images.

Carole

[Marmo720]

Seems to have worked! I only had just over an hour last night to get everything setup, polar aligned and image so went with M31. There is a deep dust bunny which I could have cleared but didn't have time and wanted to check if flats would work on such a dark bunny. 

I took these images with your cls filter Carole and didn't have time to do without filter but it is setup for next time. I took the chance to take 60sec and 180sec subs. I think I can go for 300 seconds next time with your cls filter!

I also had issue that my neighbor forgot to turn off the garden security light so it kept coming on hence why top left corner looks brighter once stacked. I didn't have any of the bottom right vignette that I kept seeing in all my previous images.

Here are the calibration files:

https://we.tl/t-IIhgwbAvqm?src=dnl

Lights:

https://we.tl/t-mpw3PPaTSA?src=dnl

Thanks,

Mo

[carastro]

Attached are the stacked and processed subs.

There seems to be a stubborn bit of dust that the flats are not calibrating out which probably needs cleaning off the sensor or whether it is but otherwise the flats have worked.

Doing 180 secs is a good step in the right direction, but now needs 300secs and lots of them.  At least 2 hours worth, preferably 3 - 5 hours.  The more the better and this will reduce noise and make the image smoother. 

180 sec subs only stacked showing stubborn bit of dust.  There was still a gradient (most probably from neighbours light, so I used Gradient exterminator. 

I also did a version where I cloned out the dust speck.

This one is the result of all the subs stacked together.

 

Edited January 7, 2020 by carastro
added more images and text

[Marmo720]

Thank you very much @carastro! So glad to see the manual flats working and I also don't have that darkening of the bottom right that I have seen in previous images. I knew that dust bunny wasn't going to be removed by the flats as it was very dark even in single lights. I will clean it off the lens at the next opportunity but now everything seems in order for me to start imaging properly  

I noticed that with your CLS filter on, the 60sec was about 1/6 of the way to the left in the histograms so I know I can do longer exposures. Without your filter, the 60sec histogram was pretty much in the middle or slightly to the right. Looking forward to seeing what 5min exposures with total >2h integration looks like!

Quick question - so I know the longer single subs are very useful for targets without cores but what would be the suggestion for bright targets with cores such as M31/M42? 5min subs for the fainter outer parts and shorter subs for the core? Using same ISO or do people change that depending on target brightness? Just checking as I am practising on M31 and don't want to waste any nights by not imaging correctly.

Thanks,

Mo

[carastro]

M42 300secs and a batch of 30secs for the core.  It is complicated to combine and I always have to use a Masking tutorial for this.  Give us a shout when you are at that stage and hopefully I will have found the tutorial again, not used it for years and I recall it got moved inside something else, so not easy to find.

What are you using for post-processing?  This uses Photoshop.

The more subs you can get the better the final result would be I recall taking 2 nights worth on the Rosette and getting about 7 hours on it once.   By the way that is another good target for you as a beginner probably better suited than M42 as it doesn't have the bright core problem. 

Carole 

 

Edited January 7, 2020 by carastro

[Marmo720]

I am using Gimp as I can quickly check the images. I also have Photoshop CS2 to see how I get on with Photoshop but not used it before.

Rosette is a good idea and I considered it last night but as I only had a short time to test, I went for M31. Now that things are working again, I will commit to one target and try get as long exposure as possible. Rosette is also in a good spot for me in the sky right now.

Thanks

Edit: Now that everything is working well - the clouds have decided to take over. No clear skies for over a week and half according to Yr   

Edited January 7, 2020 by Marmo720
clouds

[carastro]

Mo, you might benefit from looking at some video tutorials I have done in Photoshop.  Some will be too advanced for you at this stage and some are only relevant to mono imaging.

Try this one for starters:

 

[RolandKol]

3 hours ago, Marmo720 said:

Edit: Now that everything is working well - the clouds have decided to take over. No clear skies for over a week and half according to Yr   

Plenty of time to clean the glass!
Once cleaned, do some flats to check if that major one has gone away....
I do recall you had it previously and it was on the sensor....

P.S.

Happy to see, things have much improved and you actually are very close to create the nice and clean image

[Marmo720]

15 hours ago, RolandKol said:

Plenty of time to clean the glass!
Once cleaned, do some flats to check if that major one has gone away....
I do recall you had it previously and it was on the sensor....

P.S.

Happy to see, things have much improved and you actually are very close to create the nice and clean image

Thanks Roland! Getting closer with every clear sky

It is similar dust bunny to last one and not sure if on sensor or CLS filter but either should be easy to remove now as done it before. 

@carastro, thanks for the video. Will practice on my old images while waiting for clear skies.

[tooth_dr]

I’m looking some practical advice -

I bought a light box like the one linked on page 1. I also bought a sheet of 3mm opaque acrylic to diffuse the light. It passes 42% of light.

On taking my subs tonight I had the light box on the lowest setting and I still had to use 15 sheets of paper to get just above 3.5 seconds for my 2x2 subs.  This is on top of the acrylic sheet!

1x1 flats also require several sheets of paper.  Any suggest another thickness or material rather than having to use this much paper.

Also I bought A4 thinking it would be ok for my 180mm scope. I didn’t think to measure the actual aperture and on the epsilon it’s 250mm. I had then light bleed from the edge of the panel so i had to cover with cardboard.  Temp job done see below.

Any suggestions to reduce light output evenly?
 

 

 

[carastro]

Quote

On taking my subs tonight I had the light box on the lowest setting and I still had to use 15 sheets of paper to get just above 3.5 seconds for my 2x2 subs.  This is on top of the acrylic sheet!

Adam I believe you are using the Atik383, which has a mechanical shutter so your flats need to be longer to allow the shutter to get out of the way.  I used to own one of these and remember this being a right PITA.  Then of course binning 2 x 2 you really need shorter subs but you can't because of the shutter, so you need to use more paper.  With a light box of this design I can't see a better option, I used to do sky flats with thick layers of white fabric, but you can't squeeze them into a box.   Can you get thicker paper?

Carole 

 

Edited January 9, 2020 by carastro