Aircraft Tracks, Covid-19 and Processing

[groberts]

I live north of Gatwick Airport, though not actually under the flightpath and naturally have to deal with aircraft that still turn over our location.  As if that isn't bad enough, we also get our fair share of planes from Heathrow + long-haul oveflights and low flying emergency service helicopters at all times of the night from Redhill aerodrome - I know, we need to move.  Like others we're all noticing how much quieter the covid-19 world is + darker the skies, making astronomy in these otherwise weird times all the more pleasant.  However, despite the signifcant reduction of flights in order to stop spreading the virus, it's ironic that I'm noticing more flights than ususal - most of which seem to find their way through my images!

Not for the first time in recent weeks, when I've been imaging high in the sky around Ursa Major - last night being M63 - using 5-minute exposures no less than 30% of my subs were spoilt by high tracking aircraft between from 10.00pm until about midnight. i.e. almost certainly not orginating from Gatwick or Heathrow but maybe navigating via there.  Thereafter things thankfully improved with only a couple of tracks over the next couple of hours.  So, in this world in which flying has suposedly been severely restricted, what's happening?  Freight perhaps but it still seems a lot of flights and certainly noticeably more than usual.

I stack & calibrate using Deep Sky Stacker, usually after removing subs with the aforementioned plane tracks but was wondering if DSS can actually deal with such problems or is it best practice to remove them from stacking?

Last year my wife and I went on an ammazing geological road trip of Wyoming-Montana-Utah-Colorado (we're both geologists) - you wouldn't have to be a geologist to enjoy it so here's my write-up if anyone's interested  https://roundthebendpart1.wordpress.com/2019/10/17/dinosaurs-geysers-mountains-an-american-odyssey/.  One of the highlights was the Dinosaur National Monument Park  https://www.nps.gov/dino/index.htm which straddles Colorado and Utah and apart from having the most amazing rocks and literally scores of dinosaur finds, is a dark sky park.  As a result one evening we went back into the park to enjoy it at night + get some pictures of the wonderful Milky Way sky.  Mostly taking +/-20 second DSLR exposures, less than 1-in-10 of the resulting images were clear of aircraft tracks and this in an otherwise pristine night sky miles from anywhere = very sad!

Graham

       

 

 

   

[JamesF]

Look at the sigma stacking options.  They should remove the trails and allow the affected subs to be used.

James

[groberts]

Thanks James, that sounds promising - will take a look.

Graham 

[ollypenrice]

Yes, a good sigma routine will identify the pixels with trails as outliers and ignore them, giving them instead the pixel value derived from the average of the rest of the stack. The more subs you have, the better it will work. More than a dozen is best.

Not all sigma routines are equal. The one in AstroArt from V5 onwards is excellent. AstroArt also has a 'remove line' feature. For severe trails you click on both ends and apply the filter. The trail will be diminished or removed. If you clean up your worst affected subs first, before stacking, you can often keep the lot.

Olly

[JamesF]

1 hour ago, ollypenrice said:

Not all sigma routines are equal.

I don't understand why this would be, I have to say.  It's a very well-defined process to calculate the standard deviation of all the values of the pixel at a given position and for those outside a given multiple of the standard deviation either set them to black or replace them with the median value for the pixel.  I can't immediately see how it's possible to do that correctly and obtain a worse result than anyone else.  Is there something else that AA does at the same time I wonder?

James

[groberts]

OK I've had a look at DSS and am struggling to see what's meant by the Sigma Stacking Options. 

I currently use Kappa-Sigma in the Lights stacking settings, which does not remove the plane tracks - do you mean this or something else?

Graham

[JamesF]

How many frames are you stacking, and what value are you using for kappa?

James

[ollypenrice]

1 hour ago, JamesF said:

I don't understand why this would be, I have to say.  It's a very well-defined process to calculate the standard deviation of all the values of the pixel at a given position and for those outside a given multiple of the standard deviation either set them to black or replace them with the median value for the pixel.  I can't immediately see how it's possible to do that correctly and obtain a worse result than anyone else.  Is there something else that AA does at the same time I wonder?

James

I can't give you an authoritative answer but I presume that the level of deviation from the norm before the outliers are identified as outliers is a decision made by the software authors. In AA the sigma routine is adjustable though I haven't found an explanation of what the 1.8 default setting actually means. (It's a long manual!)

I do remember that Tom and I were both imaging the Witch Head, a region plagued by geostationaries, some years ago. I'd just moved from AA4 to AA5 and the trails were disappearing for me. Tom was struggling with them in AA4 and upgraded to solve the problem.

Olly

 

[groberts]

52 minutes ago, JamesF said:

How many frames are you stacking, and what value are you using for kappa?

James

Obviously this varies but usually at least say +5 hours of 3min or 5min = 60 to 100 subs; at the moment with good weather and time this has increased to +150.

Kappa is set at 2.0

Graham

 

 

[JamesF]

3 minutes ago, ollypenrice said:

I can't give you an authoritative answer but I presume that the level of deviation from the norm before the outliers are identified as outliers is a decision made by the software authors. In AA the sigma routine is adjustable though I haven't found an explanation of what the 1.8 default setting actually means. (It's a long manual!)

Apologies if I'm teaching granny to suck eggs here, but you can think of the standard deviation (sigma) as a measure of how far away (on average) the pixel values are from the average pixel value.  In a normal "bell curve" distribution, just over 68% of pixel values will be within one standard deviation of the mean value, just over 95% will be within two standard deviations and a smidge under 99.75% will be within three standard deviations of the mean.

In Kappa-Sigma stacking, the "kappa" value is the multiplier for the standard devation that sets the limit for pixel values you consider acceptable, so in your case you'd accept any pixels having values within 1.8 times the standard deviation from the mean (just under 93% I think) and ditch anything else.  Making that value smaller makes your acceptable range smaller and larger means you'll accept a wider range, so changing it allows you to control what percentage of pixels will be disregarded (though in a way that isn't really directly intuitive).

The other wrinkle I've seen is how the out-of-range pixels are then handled.  Sometimes they are set to black before stacking.  Sometimes they can be set to the median (the value in the middle if you arrange them all in order).  I guess they could just be ignored for the purposes of stacking too.

James

[JamesF]

1 minute ago, groberts said:

Obviously this varies but usually at least say +5 hours of 3min or 5min = 60 to 100 subs; at the moment with good weather and time this has increased to +150.

Kappa is set at 2.0

Graham

 

 

That seems like plenty of subs.  Perhaps it's worth experimenting with the kappa value a bit.  You could try Olly's 1.8 value, or maybe just try a few times with progressively lower values.

It does work very nicely for me at getting rid of satellite trails.  Aircraft are rather less common around here at the moment.

James

[groberts]

Thanks James, I will be doing some experimentation this afternoon using some of your suggestions; your reply to Olly on the subject is also interesting - always wondered what some of those terms in the menu meant!  In view of this, by decreasing Kappa does this have any other impact on other aspects of the stacked image?

Yes, we know Somerset well as our daughter lives in West Harptree by the Mendips + previousy in Ilminster.  We'd move there in a flash were it not for another daughter and grandchildren in Brighton! 

Graham

Edited April 21, 2020 by groberts

[Laurin Dave]

Can't add anything to your issues with trail rejection other than as James and Olly suggest reducing Kappa, I'd have expected that number of subs to come out clean but clearly not..   but I did want to say how much I enjoyed reading your Blog..  fabulous geology including a 33000' throw fault!!  and dinosaurs, brilliant!   Somewhere to visit if we're ever allowed out again,

Dave

Edited April 21, 2020 by Laurin Dave

[ollypenrice]

1 hour ago, JamesF said:

Apologies if I'm teaching granny to suck eggs here, but you can think of the standard deviation (sigma) as a measure of how far away (on average) the pixel values are from the average pixel value.  In a normal "bell curve" distribution, just over 68% of pixel values will be within one standard deviation of the mean value, just over 95% will be within two standard deviations and a smidge under 99.75% will be within three standard deviations of the mean.

In Kappa-Sigma stacking, the "kappa" value is the multiplier for the standard devation that sets the limit for pixel values you consider acceptable, so in your case you'd accept any pixels having values within 1.8 times the standard deviation from the mean (just under 93% I think) and ditch anything else.  Making that value smaller makes your acceptable range smaller and larger means you'll accept a wider range, so changing it allows you to control what percentage of pixels will be disregarded (though in a way that isn't really directly intuitive).

The other wrinkle I've seen is how the out-of-range pixels are then handled.  Sometimes they are set to black before stacking.  Sometimes they can be set to the median (the value in the middle if you arrange them all in order).  I guess they could just be ignored for the purposes of stacking too.

James

Thanks. I didn't know the underlying maths but guessed at the principle. 

Olly

[groberts]

48 minutes ago, Laurin Dave said:

Can't add anything to your issues with trail rejection other than as James and Olly suggest reducing Kappa, I'd have expected that number of subs to come out clean but clearly not..   but I did want to say how much I enjoyed reading your Blog..  fabulous geology including a 33000' throw fault!!  and dinosaurs, brilliant!   Somewhere to visit if we're ever allowed out again,

Dave

Thanks Dave - yes in 50-years of international travel and geology it's right up there as one of my best trips ever (the other would be to the montains of Iryan Jaya - western Papua New Guinea - including cannibals too, at a distance!), which I would heartedly recommend.  The mid-west of the USA is really beautiful + fantastic rocks/scenery and great night skies, much of it away from the more traditional tourist spots.  

Graham

[vlaiv]

Kappa of 1.8 is too low, so is kappa of 2

Set it to somewhere like 3.

There is known "rule", named: 68–95–99.7 rule

Best explanation is:

On average, only 0.3% of subs will be above 3 sigma distance from the mean. It also means that 2 sigma values - there is 5% that actual sub value will be rejected. That is roughly 5 out of 100 subs will be rejected not because they are outliers but because they have good statistic.

If value is within 3 sigma - then it probably can't be distinguished from rest of population as outlier.

Also keep iterations to 3 or there about (second parameter).

3 hours ago, JamesF said:

I don't understand why this would be, I have to say.  It's a very well-defined process to calculate the standard deviation of all the values of the pixel at a given position and for those outside a given multiple of the standard deviation either set them to black or replace them with the median value for the pixel.  I can't immediately see how it's possible to do that correctly and obtain a worse result than anyone else.  Is there something else that AA does at the same time I wonder?

James

Because it is not well defined.

Actual algorithm goes like this - calculate standard deviation of samples, exclude samples that are bigger or smaller than mean +/- kappa*sigma, repeat "num_iteration" number of times.

There are several "errors" that can be easily made - one is not to reject values but to replace them with average or median of values in actual sub. Other - for example that you suggested - is to replace them with black for some reason? (0 value). Third option that could produce different result would be to replace value with calculated mean and keep them included.

So you see - there are numerous variations that one can come up with.

[JamesF]

20 minutes ago, vlaiv said:

Actual algorithm goes like this - calculate standard deviation of samples, exclude samples that are bigger or smaller than mean +/- kappa*sigma, repeat "num_iteration" number of times.

I need to think about this whole iteration thing.  At first glance, iterating multiple times excluding samples outside mean +/- kappa * sigma seems as though it would be the same as a single iteration eliminating samples outside mean +/- ( kappa - k ) * sigma.

James

[vlaiv]

1 minute ago, JamesF said:

I need to think about this whole iteration thing.  At first glance, iterating multiple times excluding samples outside mean +/- kappa * sigma seems as though it would be the same as a single iteration eliminating samples outside mean +/- ( kappa - k ) * sigma.

James

No it would not - because in each iteration both sigma shrinks and mean changes "position". In single "broader" culling - mean stays the same.

[JamesF]

Ah, yes, I'd neglected to notice that the mean value would also change.

James

[groberts]

For the record - I've now completed a test of stacking three identical sets of L subs of M63 including plane tracks (approx 30%) and, at least based on this, Kappa (K)1.50 and K2.0 show no plane traces after stacking and stretching but K2.0 is noticably less noisy. Whereas K3.0 still shows vestiges of the aforesaid plane tracks and is the least noisy.  From which I conclude K2.0 is the best outcome, which was where I started but had been removing subs with plane tracks from the processing and stacking. 

I'm pleased to see that afterall I can now include those subs spolit with plane tracks, though would be even more happy if they weren't there - oh well. Notwithstanding, thanks for the input and comments - I can now increase my integration times by up to 30% by just returning the subs with planes back into the pile = result!

Graham

Edited April 21, 2020 by groberts

[Laurin Dave]

If the K3.0 is least noisy then in Photoshop you could either erase the trail using the spot healer or paste the K3.0 over the K2.0 and erase the trail in the K3.0 revealing the trail less K2.0 underneath then flatten..  I do the latter when I don't have many subs 

[Xplode]

Seeing the hysteria from a lot of people about satellite trails supposedly being the end of astrophotography i wonder if it would be a great idea to write a guide on how to remove them with different softwares and in different ways?
I'm thinking a combined thread for different softwares so it would be easy to find and easy to link to.
I'm certainly not the person do to it, but i can think of at least a couple guys that could help like @ollypenrice  and @vlaiv

[ollypenrice]

1 hour ago, groberts said:

For the record - I've now completed a test of stacking three identical sets of L subs of M63 including plane tracks (approx 30%) and, at least based on this, Kappa (K)1.50 and K2.0 show no plane traces after stacking and stretching but K2.0 is noticably less noisy. Whereas K3.0 still shows vestiges of the aforesaid plane tracks and is the least noisy.  From which I conclude K2.0 is the best outcome, which was where I started but had been removing subs with plane tracks from the processing and stacking. 

I'm pleased to see that afterall I can now include those subs spolit with plane tracks, though would be even more happy if they weren't there - oh well. Notwithstanding, thanks for the input and comments - I can now increase my integration times by up to 30% by just returning the subs with planes back into the pile = result!

Graham

You do have an additional option if you use a layers-based program like Photoshop.

- Make two stacks, Version 1 for low noise and V2 for low trails. Give both an initial and identical stretch in levels (Just take the grey point slider to the same value in both, enough to make residual trails visible in V1)

- Paste V1 low noise onto V2 low trails.

- Run a small, feathered eraser over the visible trails on the top layer. Flatten and save.

This will mean you have the low noise version everywhere except where the trails used to be. I've used this basic idea several times to save damaged subs from the dustbin.

Olly

Edit: I missed Dave's post above saying the same thing!!! Sorry Dave. 🤣

Edited April 21, 2020 by ollypenrice
My mistake.

[Laurin Dave]

25 minutes ago, ollypenrice said:

Edit: I missed Dave's post above saying the same thing!!! Sorry Dave. 🤣

No worries Olly..  its what you taught me!