Marmo720

1 hour ago, carastro said:

I suggested this because you can have more control over the length of flat (I found this out helping some-one else using APT).  (In this person's case he ended up doing something like 1/86 sec if I recall rightly), 1sec might even be too long, it all depends on how much light you have in the light source.  The main thing when doing this manually is not to guess how long the sub is  likely to need to be, but to try different exposure lengths out checking the histogram at the same time.  1/3 to just under halfway seems to be the optimal point for DSLRs.  Once you think the histogram is in the right place, only then take a bunch of them.  

Carole 

Thanks Carole. Interestingly, I had a look at my old M27 flats and each was around 1/100 sec and they worked much better (minus the strong dust donut). 

I also didn't have advanced bias/flats selected so couldn't change exposure length for flats until now so will play around and see what I can get tonight. Will share once I am done with tonight session.

In the meantime, I wish everyone happy holidays and enjoy this time with family and friends.

11 minutes ago, vlaiv said:

I'm afraid these flats are a mess on multiple levels

First let's start by the fact that again - all flats are underexposed by quite a bit.

Here are full range histograms of each of these flats (or rather master flats made from 5 flats of each group):

All are underexposed as you can see - they are not even at 10% or so and they should be at 80-85%. This means that you need to increase flat exposure by factor of about 8 times. I see that you had 1/8 second exposure for one set of flats - why don't you try 1s exposure next to see if you will properly expose flat. Use flat panel when doing flats.

Flat panel works better than iPad - here is what average value of flat exposure across flats looks like (flats 1-5):

Very small variation in mean value (good).

For comparison - same measurement for iPad (frames 21-25):

Much more variation between numbers - as if panel is flickering and producing different intensity light.

Histogram for first master flat (flats 1-5) looks as it should (except for being under exposed):

So does second master flat (flats 6-10), but third one (flats 11-15) looks different:

As if color balance changed or light source changed.

Even stranger looks iPad histogram (flats 16-20) as it shows only two peaks:

so does second iPad histogram (flats 21-25).

Out of all of these flats, flat-flat calibration is worth trying only on first two master flats and last two master flats (two of flat panel, and two of iPad):

Flat-Flat calibration (with missing bias, but let's do it anyway) of first two master flats looks like this:

That actually looks rather decent. Yes there are dust bunnies all over the place (I'll explain cause of that later) - but it is actually rather flat image (look at histogram, it is bell shaped and looking ok). Here is what it looks like filtered for noise:

There is very small gradient visible - if I stretch it it will look horrible like:

But these are variations less than 1% in intensity (from -0.5% to 0.17%). Due to fact that we did not remove bias - this is rather good.

Now second flat-flat calibration:

This one has obvious gradient and I would not call it success. Here it is smoothed out:

This gradient is now 3% wide compared to above less than 1%, so it is noticeable.

In the end, I would say that something is very wrong with the way you are taking your flats - look at this animation - it shows differences between flats of same settings (animation shows stretched flats 1-5):

Dust shadows are dancing around! Even large doughnut in the corner seems to be shifting between subsequent subs. What could be shaking your setup so hard that dust particles are dancing around? This happens on all the frames that you uploaded - some have less of shake and some more.

If you are manually triggering your camera - maybe set 2s delay (or even 10s delay) to let your scope and mount settle. Maybe look into mirror lock so it does not move out of the way each time and cause vibration. How firm is your setup? Is scope/camera connection tight and firm? How is flat panel attached to the scope? Do you hold it against the scope yourself? It would be best if you had means to mount everything without holding anything yourself and not touching either scope or camera (remote shutter release).

Thank you for the analysis. Yes I noticed the bunnies dancing too. 

My camera to telescope is pretty firm but I only have 1 second wait between flat exposure so will change to be longer and see if that fixes the issue. My flat panel is inside the original box with circle cut to allow the dew covers inside and block outside light. It rests on the dew cap when I take the flats so I don't touch any part of the imaging chain. 

The two flat-flat calibration, is the first one the flat panels (the good one) and the second set is the iPad? If so, I will stop using the iPad and stick with the lightbox.

Carole showed me how to do the flats manually so I can change the exposure time etc so will try 1sec instead of the current default APT plan. 

Thanks again.

10 minutes ago, RolandKol said:

if histogram was around 50%, that will do the job, - but simply make more of them... not 5 but 50.

Ok good. We have clear skies tonight so will try a few sets of 50 flats and see what they look like when combined with relevant lights.

Thanks

Is it still under exposed? Even with the iPad and lightbox at maximum brightness and with just 1 white paper, I could only get the combined histogram at about 50%. I will try to do the flats manually and aim for 75% peak. 

At least if the new histogram are less under exposed then we are heading in right direction...

26 minutes ago, vlaiv said:

Could you upload a few bias subs with ISO 800?

I did not keep old ones and old upload was deleted by WeTransfer (it expired)

Will do later tonight when I get home. 

Thanks,

Mo

5 minutes ago, RolandKol said:

result is even more insteresting...
have you moved your camera since the last sesion?

P.S. stacked iPad+LightBox with 1 sheet only.

Flat still looks underexposed...
Next time use Full ligh on the lightbox and 2 or Max 3 sheets.
if it was already on full power, stay with one sheet and increase exposure a bit.

Light box RGB histogram channels were concentrated in one  peak nicely, - iPad was spread. I would stick to the lightbox.

Thanks @RolandKol. Yes the camera was originally removed for storage and attached it now for testing and tonight. Probably out of focus too. 

I had both ipad and lightbox at max brightness so interesting there is under exposure. Can you see any gradient for the light box due to led on one side or is it even? 

I had a quick go in between rain showers to take some flats with the lightbox and iPad using white papers. I aimed for about 75% on the histogram and used ISO800. Each flat was taken 5 times so you will see 5x5 flat attempts. 

Hopefully this is useful in diagnosing a potential solution. Clear ish skies tonight so will try with lights too 

https://we.tl/t-0s7e1vQpx3?src=dnl

16 minutes ago, vlaiv said:

I don't think that clouds have anything to do with it. Clouds can make target dimmer and background brighter - but all of that is light reaching sensor and subject to any vignetting / dust shadows. Flat calibration should work regardless if one is shooting clouds or clear skies and there should be no vignetting and dust shadows visible in the calibrated sub. Perhaps uneven background but not vignetted.

That is fair point and would explain why I could see the same issue on different target several hours later. 

Interested to see if Roland had luck with some of the subs and why that might be.

Just now, RolandKol said:

I cannot explain that, - have no relevant knowledge...

Will stack half of the session later  on and see that we have. PI now generating the Bling movie and I am cooking....  a bit too much of multitasking

That is interesting and I really hope it was just a cloud and I can salvage/learn from this. But I think I also had the same issue with M42 which was much later in the evening. I will check once I get home from work in a few hours. 

@vlaiv If almost 50% have a cloud then it is possible you chose single sub with a cloud? Also, this might be a combination issue where single subs show the issue with my under exposed flats while full stacked image shows the impact of the clouds more? 

12 minutes ago, RolandKol said:

Ok, so I took some time and stacked them all.

I have autoremoved background from Light without any cropping and autostreached them using delinger script on Pi...

Single subs did look nice and even master Flat looks even, but after correction it kills the right side of the images...

AutoStreached Dark

 

Flat

 

Single Light

 

Light (stacked)

 

 

 

 

Thanks @RolandKol. Interesting that the master flat looks fine but the stacked image has one side looking so bad. I would think the issue might have been due to lightbox having LED on one side but why doesn't that show in the master flat? Is it possible the issue is with my lights instead of the flats? 

I think I will do the flats that @vlaiv recommended with both the lightbox and the iPad methods just to rule that out as a potential issue. 

@carastro - a little bit frustrating to stay up all night imaging and to find out that it can't be used due to flats issue. But I am learning loads and looking forward to solving this and hopefully producing some good images. The last good images I took were the Witches Broom and M27 and both had stubborn dust bunnies which we finally resolved. So I am sure this will be too and excited for when it finally is all working  

1 hour ago, vlaiv said:

Maybe it would be best to first just take flats and then we can see how to best test them. I would personally use ImageJ to do all processing and testing. You can use that also - it is free software used for scientific image manipulation and it's written in Java, which means that it will work on almost any computer / operating system. Once you have your flats and bias, I'll walk you thru the procedure how to do various things with them - like check histogram, do statistical analysis (mean value, min/max and such sort of things) and how to do image math (calibrate them, stack them, etc) ....

You will need one more piece of software (also free) - FitsWork. That one is used to convert DSLR raw files into fits that can be then used in ImageJ.

As for lights and histogram position - it is down to two things:

1. linearity of sensor

Here you want to ensure that there is no clipping left / right (low or high values - all three peaks present and looking nice in histogram). There is also concern about linearity of sensor response - it can happen that sensitivity depends on signal level (or rather gathered signal so far). Which means that doubling of amount of light, or doubling exposure length does not produce twice as high recorded signal (ADU level). This is bad thing for calibration and needs to be addressed in certain way. I think that nowadays pretty much no sensors that we use in AP behave that way, but "most linear" region was around 2/3-3/4 back then when it happened. In fact saturation and anti blooming gates cause non linearity in modern sensors - but we consider them to be "clipping" to the right.

2. Minimizing noise impact.

Once you are nice linear region and that part is covered - well, you want to have the least noise in your flats as possible - that is why we take more than one flat as stacking them reduces their noise. This noise will end up being injected back in the image in a certain way so you want to minimize it. There is two ways of minimizing the noise:

1. stack more subs

2. make sure SNR of subs is high to begin with - this part relates to histogram position. SNR is signal to noise ratio or signal/noise. If we want it to be high - we want our signal to be strong. Stronger signal is right on histogram or higher value of pixels (so closer to 1 but not too close as it will star saturating - that is why 80-85% is often mentioned).

bottom line - you can use histogram at even 5% but make sure you have enough flat subs to stack to minimize noise coming from subs.

 

 

Thank you - helpful advise and appreciate the support.

I will do the flats/bias and share so we can work through them.

Based on the last part then, is it ideal then for the histogram for lights and flats to be in similar position? Or at least not too far apart to result in over/under correction without the use of large number of flats? I guess going further, I am imaging from bortle 8 location so at some point I would need to maximise my sub duration/iso settings so will be interesting to know where the light histograms should be? Around 80% to maximise SNR without clipping? I have guiding so I could technically do 5 min subs but without LP filter, I am easily pushing the histogram too far to the right even at ISO800 and sub lengths of 3mins. 

For reference, the images I attached have the histogram for the lights at about 50-60% while the flats are around 30% although I am not so sure on the flats reading as they took 10-30 seconds to capture. Both taken at ISO800 and lights duration is 60 seconds.

19 minutes ago, vlaiv said:

I think there is something that you can try without a clear night as it just involves flats.

Try flat-flat calibration to rule out flat settings as culprit for the problem.

This would mean that you take bias subs (you can do just one or couple - number does not matter here except for reducing noise), at each ISO setting that you will test. Test with at least two different ISO settings.

For each ISO setting - shoot 2-3 different "type" of flats (where type here only means exposure length / strength of flat panel). Aim to have proper histogram each time (no clipping either to left or right) - as you did with above flats, but have different position on histogram each time - around 20-30% around 50-60% and proper one at around 80-85% for highest peak.

Each of these flats should calibrate others (once bias is removed) - meaning that you take any two flats - flat1 and flat2 and divide them, you should end up with perfectly uniform illuminated sub (no vignetting / no dust shadows to be seen).

If this happens and each flat calibrates others - you can be 100% sure that your flats are working as they should - then we need to look elsewhere for the source of the issue.

Thanks for the advise. I will try this. Just to check:

- The idea is to stack flat1 and flat2 in DSS where flat1 is input as a light sub and flat2 is used as a flat sub? So the DSS process will subtract the two and result in uniform sub? Of course using Bias as well as suggested. 

- How do I view the histograms you shared? I have been using the histogram in APT and you said the split between the peaks is correct but the location of the entire histogram should be to the right about 3/4? I have noticed this is the case if I take brighter flats so just want to check that increasing the brightness of the lightbox so that the histogram in APT is about 3/4 of the way from the left is going to be correct? Or have I misunderstood? I read a few places the histogram in APT should be 1/3 from the left so just checking.

Many thanks again.

On 21/12/2019 at 12:40, vlaiv said:

I'm late to the party and did not read most of what has been written so far (sorry about that), so I might mention something that has already been addressed, but here are my finding so far:

- Flats are severely under exposed - both ISO800 and ISO100. Not sure if that is the cause of the problem - might not be, but it is something that you want to correct.

Flats have proper looking histogram, but I believe they are well under exposed. They should be at about 3/4 to the right of histogram, but on scale 0-16384 (14bit DSLR) it looks like this:

Otherwise, when zoomed in on relevant part, it looks fine:

Three peaks as it should have, nothing clipping, etc ...

- Second observation for all those who will attempt to process above data - don't even try to use ISO100 flats. Master flat should be calibrated with flat darks, or in this case, master bias will be enough. Both set of files (flats and bias) need to be shot at same ISO settings. Above included files contain ISO800 bias and both ISO800 flats and ISO100 flats. ISO800 flats will be fine to use with bias, but not ISO100 flats because of mismatch in iso setting.

Third observation is that something is seriously wrong with flats and I'm not still sure what it is, here is what flat fielding looks like (single sub after calibration with single dark, flat and bias):

This clearly shows that there was under correction of flat calibration in corners, but there was over correction in the central part - dust doughnut is brighter than it should be.

In most cases one has over or under correction over whole field and that is consequence of either improper calibration or some issues with darks or bias, or maybe some light leak, but this is quite a bit different.

My guess is that it has something to do with the way flats were taken, or the flat panel itself (although I'm not sure what it could be).

Can I ask what scope is this and what the setup looks like and under what conditions where flats taken?

In particular - was there change in anything in setup between lights and flats - for example - focus position?

Thank you so much for taking the time to provide feedback. 

Flats are probably underexposed due to me setting the lightbox on lowest setting and using 6 white papers. Each flat took significantly longer than I thought it would take but as the histogram looked correct, I assumed it was fine. In the past, I used an iPad at about 75% brightness and a t-shirt. So I guess change the brightness of the lightbox to being mid-high might be better and will try that next. I will also try with the iPad just in case as that used to work fine in the past.

I didn't realise a single flat fielding would show the impact so much so I could spend the next clear night just checking my sub/lights/darks/bias using single files before wasting another evening of lights that I can't make use of.

There wasn't any change between lights/flats besides moving the scope to better position for the flats. Don't think there was any focus shift as I followed up the M31 Flats with M42 lights and they also have the same issue.

My setup train is currently Altair Wave 80 ED APO -> Altair 1.0x field flattener -> Canon astromod 600D. My mount is the is the Skywatcher HEQ5 Pro and I am using the ASI 224 MC for guiding. Software is APT for capture, PHD2 for guiding and Stellarium for target selection.

I don't currently have any dew heaters and my original thought was maybe dew had covered centre part of the scope lens but I would think that would be obvious from lights and flats? Is this something could explain my images? 

Forecast is clear skies after midnight on Christmas Eve so maybe I will get a gift from Santa and I can have clear images too  

Thanks,

Mo

3 hours ago, carastro said:

Only just twigged Marmo is the same person I have been in E mail discussion with where I know you as Mohamed.  Lol

Haha now I understand why you were asking about my gear setup. Confused me considering you pretty much helped me choose and setup my entire gear

Thank you so much for spending some of your time trying to get this resolved and I appreciate the help. Absolutely confused at the moment and hoping it is an easy fix.

@RolandKol thank you for the help too. I did think the flats took a lot longer this time than before and the lightbox is set to lowest with 6 sheets of white paper. I have in the past used ipad at much higher brightness and a t-shirt, and the flats would be much quicker but the histogram was about 2/3 towards the right side instead of where it is now. 

Hi all,

Below are links to my files from last week. Not having luck with stacking and cleaning the image so hope the community wisdom will help spot what I might be doing wrong and how to improve it. 

My camera is astromod Canon 600D, I have a field flattener on that and using an Altair Wave 80. I am using APT for capture, PHD2 for guidance with ASI 224 and I am applying dithering. Finally, I use DSS for stacking and a quick check on GIMP or Photoshop shows bright white halo in the middle and darker outsides. 

Many thanks in advance for any help.

Lights: https://we.tl/t-9O4TS659iR?src=dnl

Flats @ ISO100: https://we.tl/t-7UAh1fkHKr?src=dnl

Flats @ ISO800: https://we.tl/t-GtRCNFfw5X?src=dnl

Darks: https://we.tl/t-WTV5rQXlVg?src=dnl

Bias: https://we.tl/t-8RNquuP4ZE?src=dnl

10 hours ago, carastro said:

Eyes seem OK now.  Didn't go out as the "visitors" didn't leave until about 10.30 and was too tired by then. 

Carole 

I am glad to hear your eyes are OK now. 

I spent the entire night out capturing last night in the freezing cold. Went to bed at 3am and woke up at 6am for work. So very tired at the moment but hopefully worth it. 

I captured 90x60sec subs of M31 at ISO800. Took new bias and darks too. I also took 53x60sec subs at ISO800 of the Rosette Nebula before the clouds stormed in.

For Flats, I reduced the panel brightness to its dimmest and took:

- 10x ISO 100 with panel in normal position.

- 10x ISO 100 with panel in inverted position.

- Same as above 2 but with ISO 200.

- 20x ISO 400 and ISO 800 in the normal panel position.

I inverted the panel to check for any gradient from the LED lights being on one side. Visually, I couldn't see any at all and the flats looked the same regardless of panel orientation. Hopefully you guys can spot any gradient once I upload the masters for each setup.

I did a quick stack on the first set of IS100x10 Flats with my M31 this morning before rushing to work and I could still see the brighter centre and darker outsides after stacking. I will do stacks with all the different flats to see if any work and/or if you guys can spot where my issues are.

Thanks,

Mo

21 minutes ago, carastro said:

Hopefully my eyes will have recovered by this evening after going to the eye clinic today.  Had eye drops to dilate the pupils 4.5 hours ago and still can't see properly.

Carole 

Didn't know - I hope you recover fast and well. Are you planning to image tonight if your eyes are better? 

Thanks both.

I will upload the master flats for different ISO as well as the lights/darks/bias after the session tonight. Hopefully can also tell me if there is gradient from the lightbox or if that is OK.

Can't believe your filters haven't arrived yet Roland. Any hope of them arriving this side of Christmas before all the holiday posts?

1 hour ago, Laurin Dave said:

Do your flats work? Ie do they correct vignetting and dust bunnies..  if so all is well..  they generally  look worse than they are (ie bright centre dark corners) because they are auto stretched so much.  Measure the difference in intensities in the middle and the corners of an Unstretched flat I suspect it’ll be less than you’d think

The flats alone look fine to me in that I can see the slight vignetting and and dust bunnies that would be subtracted from the lights. Unfortunately, once I have stacked them in DSS with the lights, the final image looks bright at the centre and darker on the outsides. To the point where there is a massive bright halo around my target. 

In saying that, I sent the final tiff to @carastro and @RolandKol, and both produced fantastic looking images after some processing so maybe that is where my issue is? I just don't remember seeing such a huge halo back in the summer when I was using ISO100 for flats and found I could process easier. Hopefully skies stay clear tonight and I can produce a bunch of different flats to see what works.

10 hours ago, bottletopburly said:

Flat frames should be taken at same  iso setting as lights , I use an EL panel with a couple of sheets of A4 paper sandwiched between frosted acrylic absolutely no problems with flats now , turn dial on camera to AV mode before taking flats it will work out correct exposure and I take 25 flats each session .

Thanks. Not sure why I am getting bright centre and dark edges then as I currently take flats at same settings as lights. Maybe using too many flats? I currently use 50 flats while I see most people using closer to half that. I would think DSS should be able to take care of too many flats if that was the case?

Forecast is clear tonight so will try as many different things as possible. Now that I have a working light box, I can just take a bunch of flats at each ISO and see what works.

On 08/12/2019 at 14:46, Victor Boesen said:

Thank you very much! And yes, I live in the middle of Denmark's largest city so when I'm out imaging I always stay outside with my kit the entire time which means I'm sometimes limited by the temperatures outside;)

Amazing images and really love that M42 as it is one of  my favourite targets.

I have to say - your dedication to the hobby against backdrop of such light polluted city and being out in the street etc is amazing. I am looking forward to seeing more of your images!

Really nice image! 

It was one of my first targets and seeing those colours just floating in space is lovely  

25 minutes ago, Laurin Dave said:

It’s a Huion tracing panel..  

Thanks - not much more than mine and says even light. Will get it if the current one shows light gradient in my imaging.

The bright centre/dark outer + dust bunnies is what I saw after using the new light box to produce Flats so that is good news at least compared to iPad method I was using before which resulted in illumination on one side. I just need to get the correct Flat setting in APT to remove those effects. I hope!

6 minutes ago, Laurin Dave said:

I wonder if the half illumination is  because your shutter speed for flats is too fast and the panel is flickering too fast for us to see but slow enough to mess with the flats.. what shutter speed are you using? I’ll check my panel type and let you know pretty sure its evenly illuminated though ..  also it’s very blue ..  I’ve used it with my EOS600D and it was fine

Thanks. I don't remember the shutter speed but will check. However, I did the flats again last night using the lightbox instead of the iPad method and I didn't have the issue with the one side illumination which suggests to me that it might have been ipad slipping slightly or similar instead of camera issue/settings. 

However, using the lightbox last night resulted in bright centre and darker outer image so hoping the change the ISO down to 100 will help fix that.

Thanks for all the replies - apologies on my slow response as we had clear skies last night and I spent my time freezing trying to get the flats to work. 

@Davey-T - that is the exact same setup I did last night. I placed 6 sheets of white paper and set the light to the lowest level. Looking at the flats, I don't see any gradient caused by the LED being to one side so looks like it works for me even without rotating. 

@fireballxl5 - I thought it would also be evenly lit as well until I received it hence the post here. At a cost of £12, I was willing to try it with some white papers to diffuse the light. Seems to be working OK from last night test but will update once I have another clear night to test more. Not seen any that are evenly lit and assume might cost a bit. 

I did notice the light in the box looked "blue" but maybe that is due to everything else being in night vision. The histogram on APT looked fine as far as the colours go. 

@carastro - I forgot to add my setup to my signature, which would be helpful! I am using astromod Canon 600D with Altair Starwave 80 ED. No filter wheel etc. My current Lights setup is 60sec at 800 ISO. For Flats, I was using ISO 800 too which I now believe is the cause of my issues. My combined histogram for the Flats is pretty much dead centre or slightly to the right. I think it is meant to be about 1/3 from the left from reading online and discussion with you.

I was using ISO100 during the summer for Flats, which resulted in better flats but had a few stubborn dust bunnies that I was able to clear a few weeks ago. For some reason, I forgot to change the ISO down from 800 so hopefully that will resolve some of my headaches with Flats. 

@Laurin Dave - Is your tracing pad also lit from one side? No issues with that using a few papers to diffuse the light? The gradient to me seems less obvious when the dimness is set to the lowest but considering whether I need to rotate or not as per @Stub Mandrel suggestion? 

Many thanks all for the advise as always.