Data Quality

[Rodd]

The strangest thing just happened.  I collected 5 hours of 60 sec luminance during unusually good conditions for me.  I knew that I would have to cull 15-20 minutes due to passing clouds, but seeing was good.  I removed the clouded subs and then I was upset to see that I had to remove another 20 minutes of data due to jumps in guiding...every so often a sub would appear where stars would be like bar bells, with a line between ends.  I don't understand why this happens.  Between the jumps in guiding, total rms errors with <.4", so it wasn't due to seeing or bad calibration.  If anyone has an idea about this, I would love to hear it.

But that is not what led me to make this post.  I removed all poor subs (guiding and clouds) and integrated 261 60 sec subs.  The result has a mean FWHM of about 2.1", which for my sky is quite decent.  However, I then integrated all 299 subs, including the 40 or so minutes of guide jostled and clouded subs and was amazed to see that the resulting stack was better than the one without the poor subs!.  The FWHM of the 299 sub stack is a bit lower (2.14 vs 2.16), which is not much, but one would expect it to be worse, not better.  Even the sky background is better in the stack with the clouded subs.  

Question....why bother reviewing subs when the software seems to do a fine job of it.  

[Elp]

I used to but now generally don't review them now as you can tell from a FWHM graph plot after registration a rough percent to cull from the stacking. Review helps save filling your disk space with junk subs however.

[Rodd]

7 minutes ago, Elp said:

I used to but now generally don't review them now as you can tell from a FWHM graph plot after registration a rough percent to cull from the stacking. Review helps save filling your disk space with junk subs however.

So, the software eliminates the subs?  How then can one know how many subs were used.  I removed 41.  Did the software remove 47, or doesn't it work that way

[Elp]

In Siril after registration for example it tells you in the console window what it's done, if say you have 100 lights and it finds 10 with non round stars the console will state it's registered 90 subs and 10 failed, and the registration file it generates will then only contain the 90 good subs. When you then go onto stacking you can state as one of the stacking rejections a weighted FWHM and set a percentage, say you set it at 80 percent, any successfully registered subs which then have stars which fall out of FWHM plus or minus will also be rejected at that stage. Kappa Sigma stacking with a narrow top and bottom figure usually minimises/averages/removes satellite trails from the stack.

[Rodd]

7 minutes ago, Elp said:

In Siril after registration for example it tells you in the console window what it's done, if say you have 100 lights and it finds 10 with non round stars the console will state it's registered 90 subs and 10 failed, and the registration file it generates will then only contain the 90 good subs. When you then go onto stacking you can state as one of the stacking rejections a weighted FWHM and set a percentage, say you set it at 80 percent, any successfully registered subs which then have stars which fall out of FWHM plus or minus will also be rejected at that stage. Kappa Sigma stacking with a narrow top and bottom figure usually minimises/averages/removes satellite trails from the stack.

Yes, but I registered 299 subs and none failed.  The software removed the subs during the integration phase, not the registration phase

[Elp]

That is odd. Did you run the process twice to see if it did the same thing each time?

[vlaiv]

I don't think subs were removed.

Stacking probably removed anomalous pixels from bar bells (second less bright star and bar) with sigma clipping. These pixel values were anomalous and were rejected in stack.

Similar thing happens with satellite and airplane trails and cosmic hits. Even hot pixels when dithered can be removed like that.

 

[Xilman]

1 hour ago, Rodd said:

 I removed the clouded subs and then I was upset to see that I had to remove another 20 minutes of data due to jumps in guiding...every so often a sub would appear where stars would be like bar bells, with a line between ends.  I don't understand why this happens.

I have been suffering from this effect for years and I still don't understand it. Scrupulous cleaning and re-greasing the RA worm & wheel seemed to help, leading me to believe that grit may have been causing some of the problems, but definitely not all.

When I return next month I will try other ideas, including replacing the ST4 cable with a new high-quality one and trying to tweak the autoguider settings yet again.

I'm also interested in reading other's advice.

Incidentally, sigma clipping is not recommended for photometry (my major interest) as it distorts estimates of magnitudes and their errors. It can work well for astrometry and pretty pictures.

 

[tomato]

I sort the subs on quality in APP then display them graphically. There are usually a few outliers at either end, most of the subs (>85%) reside on a flat line. I check the outliers visually (APP will often put a nudged multiple star image at the top of the pile) and remove the obvious bad ones and stack the rest. This will usually include a number of subs affected by passing cloud. I try to include as many subs as possible given how good the software is these days at managing inferior data.

[Rodd]

2 hours ago, vlaiv said:

I don't think subs were removed.

Stacking probably removed anomalous pixels from bar bells (second less bright star and bar) with sigma clipping. These pixel values were anomalous and were rejected in stack.

Similar thing happens with satellite and airplane trails and cosmic hits. Even hot pixels when dithered can be removed like that.

 

Does this mean that subs are not rejected copmpletely by integration algorithym, that some data is kept and some is rejected?  I see, you alreafy answered this in teh first line.  I guess it poays to use even bad subs.

Edited Friday at 19:54 by Rodd

[Rodd]

2 hours ago, tomato said:

I sort the subs on quality in APP then display them graphically. There are usually a few outliers at either end, most of the subs (>85%) reside on a flat line. I check the outliers visually (APP will often put a nudged multiple star image at the top of the pile) and remove the obvious bad ones and stack the rest. This will usually include a number of subs affected by passing cloud. I try to include as many subs as possible given how good the software is these days at managing inferior data.

Thats what I do as well.  I was just surprised to see that the software removed poor stars and clouded subs better than I could.

[Rodd]

2 hours ago, Elp said:

That is odd. Did you run the process twice to see if it did the same thing each time?

I imtegrated 2 sets, 1) the best subs after I removed about 15%, and 2) all subs, including all the bad ones.  

[tomato]

APP does a weighted stack based on the quality, but with an accidental nudged “double star image” classed as the reference sub I don’t know what would happen if I left that one in.

[Rodd]

28 minutes ago, tomato said:

APP does a weighted stack based on the quality, but with an accidental nudged “double star image” classed as the reference sub I don’t know what would happen if I left that one in.

I was very careful NOT to use a compromised sub as reference.  Though I think "reference" means SNR, noise, that kind of thing as oppose to FWHM or elogated stars

[ollypenrice]

Regarding the barbells...

This is consistent with a correct tracking spell, a rapid movement in one axis to a new position, and then another spell of sustained accurate tracking in that new position.  It would be unlikely for an error affecting both axes to produce a consistently straight bar or a bar aligned the same way in multiple images. This can be confirmed easily by seeing whether the bars are aligned with RA or Dec. If they are not, this would be very odd. If they are, you know which axis to investigate.

A common explanation is backlash. The round barbell weights are created when the scope is resting on one end of backlash and then on the other, the bar arising from the passage across the backlash. I think your AP mount has spring loaded worms driving the wheels, precisely in order to eliminate backlash. I also think I've read about this spring loading becoming jammed and ineffective, needing a loosening off, a tap or two to free it and a re-locking in position.

I suppose a guiding software bug could also give a correct position, then bug for a while into a new position, then debug back to the correct one. If your bar between the weights doesn't align with RA and Dec, that's where I'd look.

Regarding bad subs, I've always felt that being too pernickety is counter productive.

Olly

[Rodd]

5 hours ago, ollypenrice said:

Regarding the barbells...

This is consistent with a correct tracking spell, a rapid movement in one axis to a new position, and then another spell of sustained accurate tracking in that new position.  It would be unlikely for an error affecting both axes to produce a consistently straight bar or a bar aligned the same way in multiple images. This can be confirmed easily by seeing whether the bars are aligned with RA or Dec. If they are not, this would be very odd. If they are, you know which axis to investigate.

A common explanation is backlash. The round barbell weights are created when the scope is resting on one end of backlash and then on the other, the bar arising from the passage across the backlash. I think your AP mount has spring loaded worms driving the wheels, precisely in order to eliminate backlash. I also think I've read about this spring loading becoming jammed and ineffective, needing a loosening off, a tap or two to free it and a re-locking in position.

I suppose a guiding software bug could also give a correct position, then bug for a while into a new position, then debug back to the correct one. If your bar between the weights doesn't align with RA and Dec, that's where I'd look.

Regarding bad subs, I've always felt that being too pernickety is counter productive.

Olly

Thanks Olly.  Here is a bad sub.  I see what you mean about aligning with DEC--which I think this does.  But if you look closely, there are little RA jogs in the line.  I will say all of my bad subs are horizontally challenged.  I have not seen one that had vertical trailing (sensor is set to landscape horizontally.  Commonly, I will get good guiding (.4") for 5 minutes of so, then the guide graph jumps into a 6-7 arcsec spike, right off the scale.  If I wait, it will stablie and have good guiding.  However, it settles down and for this data set I did not have a compromised sub after about 1 hour.  The funny thing is, this sub aligned just fine and when integrated, posed no issues.   My question is does the software use the good data in the sub and reject the rest?  If subs are not rejected wholly, then picking out bad subs doesn't make sense.  Is it different for clouded subs?  Several subs were dim--clouds had passed by.  They were not fully grayed out.  I could see stars but theywere obviously dimmer, and the nebulosity was washed out.  For those subs, does the software use what good data there is and feject the rest?  

Would recalibration help?, or  shoult I loosen the gears and then retighten as you (and the manual) reccommend.  It hard to believe there is a serious issue as the mount was gone over by AP aboyurt a year ago.  

[ollypenrice]

Interesting. There seem to be slight pauses as well as wiggles on the bar itself.

Either you have this backlash issue in Dec or your guiding in Dec is somehow bugging. You should be able to run the mount and try wiggling it in Dec to see if there's any difference in feel from RA, which is OK. After that I'd uninstall, reinstall and recalibrate the guiding (and change the guider cable.) If you use an OAG, is the turret solidly mounted without play? It's just possible that it might shift in only one plane.

I think Vlaiv's explanation on stacking has it right.

Olly

Edited Sunday at 14:42 by ollypenrice

[Rodd]

1 hour ago, ollypenrice said:

and change the guider cable.

I dont use a guide cable (ST4).  I use pulse guiding through the computer.  Makes sense to recalibrate guider.  I think the OAG is solid.  I'll check everything again.  Its hard to under stand how the mount can be guidingb great then have monentary wierdness, then guide great again.  Maybe its grit in the teeth?  who knows.

[ollypenrice]

1 hour ago, Rodd said:

I dont use a guide cable (ST4).  I use pulse guiding through the computer.  Makes sense to recalibrate guider.  I think the OAG is solid.  I'll check everything again.  Its hard to under stand how the mount can be guidingb great then have monentary wierdness, then guide great again.  Maybe its grit in the teeth?  who knows.

You could check the time needed for a full rotation of the worm wheel against the timing of the tracking bugs. Do they match? If so grit, etc, is a possibility. I'd certainly replace the guide cam USB if using pulse.

I think intermittent bugs are from unknown since a simple bad contact can confuse a piece of software very easily.

Olly

[Rodd]

1 hour ago, ollypenrice said:

You could check the time needed for a full rotation of the worm wheel against the timing of the tracking bugs. Do they match? If so grit, etc, is a possibility. I'd certainly replace the guide cam USB if using pulse.

I think intermittent bugs are from unknown since a simple bad contact can confuse a piece of software very easily.

Olly

I have a cable going from the guide cam to the main cam, then the main Cam connection to the computer (the opower cable), which plugs into na powered hub, then the oowered hub connects to the computer.  Which cable were you thinking of?

[pipnina]

6 hours ago, Rodd said:

Thanks Olly.  Here is a bad sub.  I see what you mean about aligning with DEC--which I think this does.  But if you look closely, there are little RA jogs in the line.  I will say all of my bad subs are horizontally challenged.  I have not seen one that had vertical trailing (sensor is set to landscape horizontally.  Commonly, I will get good guiding (.4") for 5 minutes of so, then the guide graph jumps into a 6-7 arcsec spike, right off the scale.  If I wait, it will stablie and have good guiding.  However, it settles down and for this data set I did not have a compromised sub after about 1 hour.  The funny thing is, this sub aligned just fine and when integrated, posed no issues.   My question is does the software use the good data in the sub and reject the rest?  If subs are not rejected wholly, then picking out bad subs doesn't make sense.  Is it different for clouded subs?  Several subs were dim--clouds had passed by.  They were not fully grayed out.  I could see stars but theywere obviously dimmer, and the nebulosity was washed out.  For those subs, does the software use what good data there is and feject the rest?  

Would recalibration help?, or  shoult I loosen the gears and then retighten as you (and the manual) reccommend.  It hard to believe there is a serious issue as the mount was gone over by AP aboyurt a year ago.  

 

Is there any evidence of non-stellar objects being ghosted or trailed like this, or are they too faint to show up?

I had behaviour where my RisingCam 571 (which doesn't blink before exposures) having star trails burned in when things like guiding calibration were being done, for me leading to a right angle pair of lines in my first sub post-calibration.

And for dithered subs too but of course not this long.

No idea if this is your issue but I'll throw it into the ring just in case, does your camera support "blink before exposure" in the settings?

[ollypenrice]

11 hours ago, Rodd said:

I have a cable going from the guide cam to the main cam, then the main Cam connection to the computer (the opower cable), which plugs into na powered hub, then the oowered hub connects to the computer.  Which cable were you thinking of?

My first suspect would be the guide cam to main cam but, unfortunately, communication between guide cam and PC passes through all the components in your list. This is why I take the rather steam-age view that the most reliable system uses direct USB cables from each appliance to the back of a desktop PC with lots of USB ports. I don't like 'multi tasking' appliances because their compactness and convenience comes at the risk of one component failure being inflicted on other components.

Given that USB cables are cheap I'd try replacing all of them and try another hub if you have one.

Olly

Edited Monday at 07:07 by ollypenrice
False click

[Rodd]

20 hours ago, pipnina said:

Is there any evidence of non-stellar objects being ghosted or trailed like this, or are they too faint to show up?

I had behaviour where my RisingCam 571 (which doesn't blink before exposures) having star trails burned in when things like guiding calibration were being done, for me leading to a right angle pair of lines in my first sub post-calibration.

And for dithered subs too but of course not this long.

No idea if this is your issue but I'll throw it into the ring just in case, does your camera support "blink before exposure" in the settings?

No idea--in subs that don't have issues, there are no trails.  In subs where the stars trail, everything does, really.  But an interesting idea.  Not sure if the 1600 has blink