Remaining dust bunnies, processing them out

[kirkster501]

I've made the imaging cardinal sin of not doing flats in a timely manner after the imaging run - a few days later. I did them such due to various day to day upheavals.......  I have a thread running on Pixinsight forum as a result.

https://pixinsight.com/forum/index.php?topic=12416.0

The OTA, camera, focuser, FW have had not moved.  Consequently, the flats I did take have moved most of the bunnies and vignetting, but a few larger bunnies (the ones on the front objective) are faintly visible in a couple of places, probably due to a few motes of dust (I made the second cardinal sin of not covering up the objective since the run finished 4 in the morning and I was rushing as needed to get back to bed after locking up the obs).

I was thinking of using Photoshop blur to get rid of it.  Any thoughts on that please or any alternate method?  I suppose I wil give it a try.  The bunnies are not that bad and are large and faint.

Would you correct the final resultant image or correct the individual offending channel(s) immediately after pre-processing?

Thanks, Steve

[JohnSadlerAstro]

Hi,

Does photoshop have the ability to mask an area and then apply a brightness change to that mask (like Star Tools)? If so, that's probably the best way. You might need to make several attempts at getting the mask the right size/blurriness, though.

John

P.S. Nice M101!  

Edited May 23, 2018 by JohnSadlerAstro

[kirkster501]

Thanks.  I am not familiar with that technique.

It is 18 x 10 mins of luminance from two nights imaging and I am loathe to throw all that away so looking at how to remove that wide arc bunny.

Edited May 23, 2018 by kirkster501

[Oddsocks]

Looking at the side-by-side image you posted, all the donuts in the master flat have concentric bright rings around them. I thought at first that this might be just a bit of jpeg-ery edge detection enhancement going on with the screen capture but the fact you see these artefacts in the calibrated light suggests it is a real artefact in the flats.

Posted screen captures are really much too low quality here on the SGL forum to make an accurate diagnosis and it is always a good idea to post the fits images as well, or a link to a Dropbox or Google Drive folder containing them when asking for help with image artefact or calibration issues.

Taking your screen capture and applying curves in PS shows that the darker donuts in the flat master are also present in the calibrated lights, it is not just the bright arc that remains after calibration. 

There are only two ways to produce bright concentric ring artefacts in the flat masters (using PI) that I am familiar with.

First is a change in focus during the time it took to take the flats, i.e, the telescope focus was not consistent across the full sequence of flats used to create the master.

Second, either bias frames were not taken and included in the batch processing script (or if using the manual calibration tool in PI, a bias master was included but the tick box 'calibrate' was not ticked in flats selection window), or there is a temperature mismatch between the lights, darks, flats and bias frames, and/or a matching dark frame for the flats was not included in the batch processing script.

Because both the bright ring and the dark donuts are present in the calibrated image then it confirms the flat is not calibrated correctly.

In PI when calibrating images from a 8300 sensor it is essential that the full set of matching temperature calibration frames are included, if the flats exposure is longer than a few seconds, then matching time and temperature darks are also recommended.

If it is simple case that you did not take bias frames, or matching time dark frames for the flats, set the camera to the same temperature as for the lights and take a series of bias frames and matching time flat-dark frames now and rerun the batch processing script with the bias and flat-darks included, that should fix the bright ring artefacts around the dark donuts in the flats and calibrate the lights correctly, if it doesn't, try the manual calibration tool instead as I find it often produces better results than the batch processing tool as you have more control over the fine-tuning characteristics of the calibration procedure.

If you did take bias frames and flat-darks and include them in the batch processing script or manual calibration tool check the fits headers for the bias, darks, flats and lights, were they all taken at the same temperature? If these images were taken with the G2-8300 the 8300 chip shows a great variation in dark current with temperature so it is important the temperature is consistent across all the frames used (as I found out with my own QSI 683 that uses the same sensor).

I can't offer a post processing solution to fix your current image but maybe going back over the calibration procedure with the full set of matching calibration frames will solve the problem and you won't need to resort to "air-brushing" the final image.

HTH.

[kirkster501]

Wow odd socks.  I will check into that.  This was using my Atik 460/EFW2 on TEC140 + EFW2.

The BIAS and Dark are from my dark library - admittedly three months old - but at the same temperature as the lights.  Would not have thought that this mattered that much but since I am at the end of my own ideas then that is something I will check.  Focus between lights and flats is spot on - or as much as it can be with autoficus that changes through the night by a few points.

[Davey-T]

Hi Steve, are you using PI ?

I'm still using Maxim and it writes all the processing info into the Fits header so you can easily check if a step has been missed, presumably PI does something similar.

Those big bunnies look to be a long way from the sensor, it might be worth taking it apart and cleaning the filters and sensor cover glass before starting a new target.

Dave

[kirkster501]

Hi Dave, many thanks.  Yes indeed, I am going to give the filters and camera glass a good clean.

[ollypenrice]

Firstly any anomalies are unconnected with dust on your objective, which is far too far out of focus to have any effect. (Look at that big central obstruction in the middle of the lightpath of a Ritchey Chrétien!) Whatever is stopping your flats from working perfectly, it isn't that. Dust bunnies are created from contaminants much closer to the chip.

And, although I follow Oddsocks' logic, I have never ever detected any issues arising from changes in focus between flats and capture. We refocus regularly but nobody shoots a set of flats prior to doing a refocus - do they? I've never heard of anyone doing so and they'd be pretty unpopular at star parties. ? If, inadvertently, you'd shot your flats at a very different point of focus then, yes, they would probably have an issue but the tiny adjustments we make to focus on an imaging run simply do not require replicating when shooting flats.  Is there any chance of a big accidental change in focus?

Are you using the TEC flattener? I do find this needs to be kept clean because it's a huge piece of glass a good way from the sensor so it can create very large, faint bunnies.

PI always strikes me as hard work for stacking and calibrating and, on the two occasions when guests have tried to persuade me to use it, they've demonstrated it and found they'd done something wrong so it didn't work.... In AstroArt I just make a master flat by average combining the individual flats and putting in a master bias as a dark flat. Lots of people use this short cut. It works perfectly for me. The idea is that, for short flats exposures, there is no significant difference between a dark and a bias.

 I never assemble flats in the same pass as stacking lights, though. With experience you get a feeling for when a flat is 'right' and when it's suspect. I always want to look at a master flat before applying it. Why they are sometimes wrong I don't know, but occasionally they are, and I just reshoot them.

Your 'cresent moon' artefact would be an easy cosmetic fix in Photoshop (he claims! Famous last words.) Copy layer, top layer invisible and inactive. Open curves. Put a fixing point on the curve at the value of the background sky next to the crescent. (Put the cursor on a bit of this background sky and alt click.) Put a second fixing point on the curve below that. Pull down the curve above the top fixing point till the crescent is no longer brighter than the background. Top layer active and visible, use the feathered eraser to remove just the crescent. Dodge and Burn brushes at 1% set to shadows will provide a final tweak.

I tried on a screen grab:

Any cosmetic fix is a last resort, of course. I never use them myself. Ahem. ?

Olly

Edited May 23, 2018 by ollypenrice
Typo

[ollypenrice]

5 hours ago, JohnSadlerAstro said:

Hi,

Does photoshop have the ability to mask an area and then apply a brightness change to that mask (like Star Tools)? If so, that's probably the best way. You might need to make several attempts at getting the mask the right size/blurriness, though.

John

P.S. Nice M101!  

John, PS does have this ability but it also has layers and the eraser tool. For many, many processing adjustments I find this a far simpler method than masking for precisely the reasons you imply when you say,  'You might need to make several attempts at getting the mask the right size/blurriness, though.' If, instead of worrying about shaping a mask, you carry out a modification globally to a bottom layer, you can then use a soft edged eraser to remove the top layer where you prefer the bottom, modified, one. And you can see the result in real time as you do so. (As Sara once said to me, 'All you ever do is use the bloody eraser!'  Guilty as charged, m'lud.)

Olly

Edited May 23, 2018 by ollypenrice

[kirkster501]

Wow, thanks guys/Olly. What a wealth of genius this forum is.

 

Run that past me again @ollypenrice please?  So you use a master BIAS as your dark for the flats?  Do you not use a BIAS as a "bias" for the flats as well?  You develop this flat before you start fiddling with lights?

First, the focus between lights and flats is pretty darned close. The range of Lakeside focuser movement due to temperature change on my autofocus rig in the FITS headers is 11330 > 11310 - so the flats were taken  at 11320 - the median value.

Lights, darks, bias all done at at -15C CCD temperature.  The BIAS and Darks are three months old but surely the BIAS and DARKS - even old ones - could not cause this effect, they'd cause more general sensor noise, not stop flats working?    I read the temperature does not matter so much for the flats.  I could not get the CCD on the Atik to -15C and only to -9.  I really don't think this should matter so much? Should it? 

The FF of the TEC is brand new and immaculate.  It has been inside the focuser as soon as I got it and the ATik 460 has not been off of it.  I concede probably time to give the filters a clean and Atik 460 window perhaps.  But the flats for this run at least should take care of it...

I have done several sets of flats and varied the ADU from 7000 up to 49000 in increments of 7000.  All of them show this "arc".  So the lights are not being properly calibrated by the flats.

Anyway, for now, I will use Olly's cosmetic correction fix since I am loathe to bin and otherwise good luminance data set.  I will reshoot my BIAS and Dark calibration frames.  And I will ALWAYS from now on do the flats at the latest the following morning.

So that leaves what?  The only thing I can see is something to do with my calibration software?  I may give the AstroArt trial a go.  Or Deep Sky Stacker.

 

 

[kirkster501]

Used the manual way of doing this in Pixinsight as opposed to using the Batch Pre-processing script.  Same results.  I think I need to completely redo my calibration frames and start again and cosmetically fix this luminance.

I don't even think the vignetting is being completely removed - it is there in the corners still.

 

[ollypenrice]

2 hours ago, kirkster501 said:

Wow, thanks guys/Olly. What a wealth of genius this forum is.

 

Run that past me again @ollypenrice please?  So you use a master BIAS as your dark for the flats?  Do you not use a BIAS as a "bias" for the flats as well?  You develop this flat before you start fiddling with lights?

 

 

Yes, I use a master bias as my 'dark for flats.' The flats need this and only this by way of calibration. Don't subtract the bias a second time, once is just right!

And yes, I always make my master flat before using it on a set of lights. I need to believe in it. To believe in it I need to look at it - though I can sometimes be wrong in thinking a flat is OK when it isn't or thinking it isn't when it is. This doesn't happen often but it does happen occasionally. Even with a full frame camera the TEC with flattener gives a seriously flat field, almost free from vignetting. Any vignetting that I see comes from my tight fisted use of 2 inch mounted filters. When I use the Atik 460 in the flattened TEC I would describe the vignetting as zero.

You won't like this because you've already said previously that you don't like the 'one flat fits all filters' model, but when I find I have a dodgy flat for, say, green I just dump it and run the luminance flat for green. I do a lot of multi-panel mosaics and these need flat background skies. I've never had a problem doing it this way...

Olly

Edit: my calibration files may not be immortal but they are certainly geriatric! Flats for every night in an observatory based mount? Nooooooooo. One month's lifespan is unlucky, six months is lucky. A year is not unkown, not that I'd ever admit it on a forum... ?

Edited May 23, 2018 by ollypenrice
As stated.

[Whirlwind]

5 hours ago, ollypenrice said:

And, although I follow Oddsocks' logic, I have never ever detected any issues arising from changes in focus between flats and capture. We refocus regularly but nobody shoots a set of flats prior to doing a refocus - do they? I've never heard of anyone doing so and they'd be pretty unpopular at star parties. ? If, inadvertently, you'd shot your flats at a very different point of focus then, yes, they would probably have an issue but the tiny adjustments we make to focus on an imaging run simply do not require replicating when shooting flats.  Is there any chance of a big accidental change in focus?

It probably depends on which side of the focuser the flattener is.  If you have the ability to put the flattener on the telescope side of there is a set of lenses much nearer the ccd (e.g. An Edge) then if you have any dust on the flattener then you may need separate flats if you change the focus position because you are so much closer to the dust.  However I imagine that in most cases it probably isn't noticeable if you change focus position several times over one night as the stacking will likely reject the variations. It only becomes obvious when the flats are done at one position and the lights at another.

My general suspicion is that these effects arise when there is some slight motion because something isn't quite locked down as tightly as you realise resulting in a slight shift and a residual 'ring'.

[JohnSadlerAstro]

12 minutes ago, Whirlwind said:

My general suspicion is that these effects arise when there is some slight motion because something isn't quite locked down as tightly as you realise resulting in a slight shift and a residual 'ring'. 

I'm afraid I'm just plain superstitious when on the issue of flats taken the next morning, after 'nothing has changed'. I have never got next-day callibs to work satisfactorily. Possibly the setup is angered when the by-3am-half-dead astronomer puts sleep before photos and staggers to bed.  

You mention there is vignette, to be honest I cant see any?   Perhaps the effects of the website compression and screenshotting have masked it though.

John

[ollypenrice]

30 minutes ago, Whirlwind said:

It probably depends on which side of the focuser the flattener is.

I use both Petzval scopes (whose flatteners are on the objective side of the focuser) and flattened triplets (whose flatteners are on the camera side of the focuser) and have never found that flats were sensitive to fine focus. And think about all those robotic scope users. They can't shoot flats 'per focus.' No, I simply don't believe that flats are sensitive to fine focus at the level carried out during a night's imaging. It simply flies in the face of everyones' experience.

Olly

Edited May 23, 2018 by ollypenrice

[ollypenrice]

4 minutes ago, JohnSadlerAstro said:

I'm afraid I'm just plain superstitious when on the issue of flats taken the next morning, after 'nothing has changed'. I have never got next-day callibs to work satisfactorily. Possibly the setup is angered when the by-3am-half-dead astronomer puts sleep before photos and staggers to bed.  

You mention there is vignette, to be honest I cant see any?   Perhaps the effects of the website compression and screenshotting have masked it though.

John

? Well, I'll admit it, wild horses wouldn't drag me into doing flats at the end of ten hours out under the winter stars! The darned things can wait, and always have done, other than in summer when it is impossible, here, to cool the cameras by day. If, in summer, I need new flats then I do have to shoot them in the wee hours because shooting them in an observatory at 55 degrees C is not really an option!

But as for being superstitious - sure. Oh yes. Like most imagers I have a shedload of personal superstitions. Eg, Never use software to do anything you can do manually, Never add a gadget to 'simplify' your rig, Never use a hub when you can add another cable, Never listen to theory when you can use experiment... I could go on!!! But I'm a dinosaur and know it.

Olly

[JohnSadlerAstro]

I also keep a "remove before flight" tag on one of the tripod bolts, (to stop the setup blowing away in a wind--this one's been quite successful), and a pocket compass attached to the PA slo-mo controls (for good PA, this one's not quite so effective sadly). 

I have heard of keeping pet spiders or even rats nests in out-of-use OTAs is good for keeping up collimation, apparently their droppings prevent mirror corrosion are also better than most re-aluminizing methods.

And of course, the 130P-DS superstition about never saying "zero" near the setup while imaging--take the 0 of 130 and you get 13!  

John

[Whirlwind]

4 hours ago, ollypenrice said:

I use both Petzval scopes (whose flatteners are on the objective side of the focuser) and flattened triplets (whose flatteners are on the camera side of the focuser) and have never found that flats were sensitive to fine focus. And think about all those robotic scope users. They can't shoot flats 'per focus.' No, I simply don't believe that flats are sensitive to fine focus at the level carried out during a night's imaging. It simply flies in the face of everyones' experience.

Olly

However all of these are going to be dependent on the system being used.  The FSQs are well corrected sealed units so there is less likelyhood that these will show some form of aberration.  There's likely a number of factors that come into play.  What we do know is that the size of the shadowing is dependent on the distance to the sensor.  The further away the larger it becomes, but the more diffuse as the effect s spread out over a larger area of the ccd. The reason we don't see dust from an objective is because it is so far away that effectively it has been 'diffused' to being not visible. 

On on that basis it then becomes a question of percentage change where changing the location of the sensor to the dust makes a noticeable difference.  If the back focus distance between ccd and dust was 50mm and by focusing you changed that by 1mm then the difference in the dust shadow will be more significant compared to one where the back focus distance is much larger (let's say 150mm).  It will also be dependent on the size of the pixels as you will be more sensitive to such changes as they get smaller.  Hence it may not be a case that they aren't there in remote setups but they aren't noticeable as the setup as designed.  However more flexible setups may find they are more susceptible to this kind of fluctuation.

 

edit  - apologies for the typos, I hate typing on an iPad.

Edited May 24, 2018 by Whirlwind

[ollypenrice]

7 hours ago, Whirlwind said:

However all of these are going to be dependent on the system being used.  The FSQs are well corrected sealed units so there is less likelyhood that these will show some form of aberration.  There's likely a number of factors that come into play.  What we do know is that the size of the shadowing is dependent on the distance to the sensor.  The further away the larger it becomes, but the more diffuse as the effect s spread out over a larger area of the ccd. The reason we don't see dust from an objective is because it is so far away that effectively it has been 'diffused' to being not visible. 

On on that basis it then becomes a question of percentage change where changing the location of the sensor to the dust makes a noticeable difference.  If the back focus distance between ccd and dust was 50mm and by focusing you changed that by 1mm then the difference in the dust shadow will be more significant compared to one where the back focus distance is much larger (let's say 150mm).  It will also be dependent on the size of the pixels as you will be more sensitive to such changes as they get smaller.  Hence it may not be a case that they aren't there in remote setups but they aren't noticeable as the setup as designed.  However more flexible setups may find they are more susceptible to this kind of fluctuation.

 

edit  - apologies for the typos, I hate typing on an iPad.

The theory is perfectly sound, I agree, but does this actually ever happen in practice? I use pretty much the same setup as the OP - sometimes exactly the same depending on which camera he's using - and I've never seen such an effect.

I've also hosted two robotic TEC140s and, again, never encountered any refocusing effects on flats.

Olly

[kirkster501]

I am reshooting all my calibration frames and will just have to live with the fact that the M101 and M51 will have to be cosmetically fixed in Photoshop (aka "cheat" ?   ).  Photoshop is a wonderful thing, you ought to see my six-pack abs..... ?

The FF from TEC is spot on immaculate optically.  I am wondering, though, if I might need to matt blacken the two adapters I needed to adapt the scope to the FF 85mm backfocus and maybe even blacken the FF metalwork???  I wonder.....  It's all that shiny black metal and with extra optical surfaces and bright light to create flats, I wonder if I've had some reflections in the tube.  I never had this problem before the FF.  

 

[kirkster501]

I've stretched the flat to within an inch of its life.  Those large, faint circular bunnies look pretty darned consistent in size to me and as Olly says, they must be quite close to the sensor.  And they look almost like they are in a line.  I reckon I have internal reflections going on here caused by the FF......

EDIT:  I just checked this again.  Before I installed the the FF the inside of the TEC tube is as black as your hat - pitch black. The introduction of the adapters needed for the FF to mate to the camera/EFW2 and 2 nose-piece at the required distance has introduced some semi-glossy surfaces with the two Astro-Physics adaptors that I think is shining back onto the inner FF glass and causing a reflection with the brighter light needed for the flats.  I am convinced this is what is causing the problem.  I need to get some flocking material and blacken everything aft of the FF.  Will report back.

Edited May 24, 2018 by kirkster501

[ollypenrice]

I also use the AP adapter and it would astonish me if AP didn't do a good job of blackening. They're a pretty serious organization.

If using paint, be aware that you shouldn't use anything made with dyes, which are reflective in other wavelengths. I forget which. You need to use a paint made with pigments, such as high temperature paints for barbecues and stoves. There's a paper about this somewhere on the net.

Olly

[kirkster501]

There is a "sheen" to them Olly that I think could be slightly reflective.  I am clutching at straws to be honest since I cannot fathom why I am getting those large donuts that are not processing out.  I will flock the adapters with flocking material that i have ordered off of ebay.

[Miguel1983]

Hi, i'm reviving this topic because i have te same problem currently.

My flats don't seem to be working properly and dust smudges ar left in the image, i've already took new flats but that did nothing.

Any suggestions ?

Thanks

[Merlin66]

Flats, flats...

why not just clean the filters, front glass cover and sensor of the camera?????

I find I need to verify the cleaness of the camera (I do solar imaging) using a telescope covered by a clear plastic bag viewing a bright sky at f15 or greater.

Any other dust bunnies which arrive must then show where the filters etc need cleaning.