I give up - I need to Guide. Pls advise.

[Spacehead]

Hi all, ive had a lot of trouble getting my images stacked with flats - I am convinced this is due to drift during the session.  Whilst the drift is slight enough to allow me to get some great single 1 min exposures, during the night the position of the stars on the area I am looking at changes substanially.  This is fine when stacking with no Flats - but I need flats as I get a lot of vignetting - with the middle being washed out.

I really also want to get longer exposures to take better advantage of my new modded 1300d.

So - I have a 200p on eq5 with RA and DEC motors.  I have the conversion kit to add an ST-4 port to the handset.

As I understand it - I can go two ways now.
1) A syguider by skywatcher which I can connect onto my existing standard Skywatcher finderscope, plug it into the new ST-4 port on the handset and off I go.
or
2) Get a guide camera which fits my standard skywatcher finderscope and plug that into my laptop, then run a cable from the laptop USB port into the ST-4 port on the hand set.
I get software on the laptop (PHD I think) and learn from there on.

What shall I do?  What is the difference between the Synguider 2 and the Synguider?  (if any).

Cheers all

 

[Ouroboros]

Drift should make no difference to how well flats work when stacking.  The software uses the flat data before the lights images are registered and stacked.  It sounds like there's something wrong with the flats you're taking, or how you're using the stacking software. What stacking software are you using. DSS?  A photo or two would help.

As for guiding, I use an ST4 cable from the camera to the mount, plus a cable from the camera to my laptop with PHD2 Software for guiding.  Sorry I don't know the EQ5 conversion kit and how best to use it. 

[steppenwolf]

I concur with the above, Flats are applied before stacking so this isn't your problem.

33 minutes ago, Spacehead said:

1) A syguider by skywatcher which I can connect onto my existing standard Skywatcher finderscope, plug it into the new ST-4 port on the handset and off I go.
or

Although the SynGuider does work I have tested and reviewed the original), a standard guide camera and a laptop running guiding software like PHD(2) is a better solution.

34 minutes ago, Spacehead said:

2) Get a guide camera which fits my standard skywatcher finderscope and plug that into my laptop, then run a cable from the laptop USB port into the ST-4 port on the hand set.
I get software on the laptop (PHD I think) and learn from there on.

This solution BUT an ST4 cable and USB guide control are mutually exclusive - you choose one or the other. ST4 is a simple 4 switch system, the USB option is called Pulse Guiding and is generally superior but ST4 is fine too.

[Spacehead]

Can we just consider the logic of this a moment - for my benefit.

"flats are applied before stacking."

Exactly the problem.  If they are applied before alignment, then the logical operations used to cancel out the bits of data are applied in the wrong position - IE not the final position of the aligned star image.

Lets say we had a picture, with a star in the dead center and to that image we applied the flat.
We then took our last picture, which had a star in it in the top right corner and we applied the flat.

Notice that the image with the star in the top right corner would not have had much data removed around its star - because the top right corner of the flat is dark.
Notice that the image with the star in the center will have had a lot of data removed from around it - as the middle of the flat is light.

We then align - but we have aligned an incorrectly "flat cancelled" image with the top right star - the top right star moves to be dead center, and drags along with it all the incorrectly removed shading - incorrect because it was cancelled against the wrong area of the flat - IE because it wasnt in its final resting place.

Do you see my quandry?  Now - do this with the stars positioned due to slight drift over 1 hour.  Say 60 frames - we have a line of stars if stacked with no alignment.  That line - is approximately as long as the streaks in my final build - and also that line is the same direction as the drift.

Hence my theory of what is causing these streaks.

 

[Spacehead]

Being a noob - I am not at all afraid of making a complete pillock of myself - this topic is HARD - so if I'm talking utter garbage - please feel free to insult me and let me know i'm an idiot as you see fit.
:)

 

[ollypenrice]

Yup, it's hard but Steve does know what he's talking about!

Forget stacking and consider just one image (one subframe.) It suffers from uneven illumination, like every astro image ever taken. All scopes give a brighter middle than edges and nobody is defect free. Your flats correct the defects of illumination on that image.

Now your stacking software takes a look at your second subframe which will not be in exactly the same position as your first one. (A good thing.) The scope will not be in exactly the same position relative to the sky (however well you are aligned and are guiding) but the uneven illumination is created by the scope, so it will have moved with the scope. It will be properly applied to the second image. And so on.

Finally your 'correctly corrected' individual subs will be combined based on star position and will be quite marvellous!

Olly

[Spacehead]

I don't think uneven illumination (vignetting) in this case moves with the scope at all.  The vignette remains constant in the center of all frames - (or in my case slightly off left center) - and the stars move across it - positioning themselves as per the drift.   Irrespective of the position of the scope.

This can clearly be demonstrated by the data set I have uploaded (from which the above image sample is derived).

For correct flat alignment in a drifted data set either the flat must be applied with the current frames x-y offsets calculated into the positioning of the flat against the frame, or the frame must be aligned first - before the vignette mask is applied.

This is so fundamentally correct in my head - that I cannot believe that the software developers have not thought of it - and the problems it would cause when trying to stack nebula (in particular).  Infact my disbelief is what is making me think I am completely wrong - yet in actuality - it is the only thing I can find wrong with my images pre-processing!  (beyond the expert level that is - I mean they are clearly coma'd, eggy, etc etc) - but fundamentally - this lot should stack with a flat.  They stack with out a flat - but not with - they end up smearing.

I know the mask isnt perfect - but it DOES work GREAT on 2 or 3 closely taken frames - seriously reducing the vignetting.  Just not when its used across lots of frames with drifting.

http://www.serlimited-uk.co.uk/space/IMG_0870.CR2 - thats one of the images.  I cant bring that streaking out in it - because it isn't there.
http://www.serlimited-uk.co.uk/space/flat.tif
(there may be a bit of discrepancy on the size (x) I think - for some reason my tif conversion created a slightly wider image - but I did manage to get the right size for the stack -but thats the basic flat anyway.
No streaking can be brought out in that flat (not as far as I can get anyway).

 

[DaveS]

Consider what flats are and how they're made.

A flat is just a map of the light transfer function of the optical train. When they're made either with a light panel or a white T-shirt over the front of the 'scope. The 'scope doesn't move at all, and there're no stars involved.

[Spacehead]

Yeah I am getting that (to an extent) - but am I wrong in saying that flat application to a frame can only be done AFTER the frame is aligned to its final position within the constructed final image?

 

[Spacehead]

From the DSS website.

"Application of darks, flats and offsets before registering
If dark, flat and/or offset frames are checked they are automatically applied before the registering process."

By doing the above, with a diagonal drift in a set of x images, the alignment process will be applying a given section (depending in direction and amount of drift) of flat filtered light image to a final stack with incorrect flat filtered data for the final area in which the frame is positioned.

I don't think this is debateable.  Think of a large circle on your images stamped across all of them in the same place (your vignette mask).  Then move the whole image left and down (for example) so that the stars line up - then flatten the layers in the final stack.  You will end up with the vignette mask in a different place for every single frame.

It wont matter if there is very little (or no) drift during the set.  It also wont matter "that" much on starfield images - but where massively variable shading is concerned such as within a nebula - you WILL get smearing.

Thoughts guys?

[Ouroboros]

39 minutes ago, Spacehead said:

From the DSS website.

"Application of darks, flats and offsets before registering
If dark, flat and/or offset frames are checked they are automatically applied before the registering process."

By doing the above, with a diagonal drift in a set of x images, the alignment process will be applying a given section (depending in direction and amount of drift) of flat filtered light image to a final stack with incorrect flat filtered data for the final area in which the frame is positioned.

I don't think this is debateable.  Think of a large circle on your images stamped across all of them in the same place (your vignette mask).  Then move the whole image left and down (for example) so that the stars line up - then flatten the layers in the final stack.  You will end up with the vignette mask in a different place for every single frame.

It wont matter if there is very little (or no) drift during the set.  It also wont matter "that" much on starfield images - but where massively variable shading is concerned such as within a nebula - you WILL get smearing.

Thoughts guys?

My thoughts are that the streaks are due to residual noise not being adequately compensated for.   Have you tried just stacking the lights to see if the noise is still there?  Your flats, darks and bias data could also introduce noise. How many of these are you taking? The more the better of course to take advantage of averaging. Did you take the darks at the same temperature as the lights. A mismatch there has been suggested as a possible cause of this sort of noise. 

[Spacehead]

Ourboros - the darks arent a part of the equation.  the streaking is there with or without them so long as the flat is present.
I have 1 flat only - I have linked to it above.
Without darks OR flats - the 70 frames stack fine - just washed out in the middle due to vignetting - hence my needing a flat.  So I introduce a flat - just to get me going - and that happens.

I am 100% certain that it is the application of the flat before alignment which is causing this problem.  It makes total sense - and I have seen this before - it isnt just me.

Both registax 6 and DSS give the same result.

[Spacehead]

How about this - two frames (bear in mind I have 70!!!) - stars aligned with mismatched flat due to drift.  The implications are enormous.

See how there is extra flat pattern in the wrong places due to the drift - and remember that the final stacked detail of these areas at bit level is dependent on the intensity brought about by the mismatched flat.

Replicate these two circles (but much closer together and over 70 times in the frame over a larger distance - and you get the "drift" ho ho.

[Spacehead]

Do it with stars - not a problem - they overlay each other anyway.  But with clouds, the varying intesity in the wrong positions would cause the smears - it will be to do with non-overlapping varying intensities though - stuff getting "left behind" because it wasn't blanked out the same way as were the pixels in earlier frames.

[vernmid]

Since you only have one flat, wouldn't any noise in it become amplified by the stacking process? After all the stack will see it in all frames after flat subtraction.

I assume this is why lots of flats is good as it averages out flat noise.

Have you tried stretching colours in your flat to see if the streaks are hiding there?

 

[Spacehead]

Hi vernmid - try it - its uploaded there is a link there, I would appreciate it if you CAN find any noise.

[Spacehead]

(im not on main machine now so cant try it again - although I did mess with it - nothing jumped out at me)

[Spacehead]

Either way though - the misaligned mask is not really debateable - its there - and confirmed on the DSS site (by implication).  Wouldnt matter if there is no drift anyway.

[wxsatuser]

This is from your other topic.

The streaks are not present in the lights OR flat - no matter how I stretch or fiddle with them to try and bring the streaks out.
Stack with no darks and no flat - streaks do not show.
Stack with darks and flat - streaks show.
Stack with no darks and only flat - streaks show.
Stack with only darks and no flat - streaks do not show.
Stack with only flat - streaks show.

The give away here is the flat but your still missing the point, it's one flat.

You will be adding noise with one flat, period.
This is why we stack several images of each individual calibration frames.
The streaking is noise and not part of scene and you have'nt found a fundimental flaw in the stacking procedure.

BTW
There is something fundimentally wrong with the files you posted, they will not open in PS CC or PI
 

[Ouroboros]

7 hours ago, Spacehead said:

Ourboros - the darks arent a part of the equation.  the streaking is there with or without them so long as the flat is present.
I have 1 flat only - I have linked to it above.
Without darks OR flats - the 70 frames stack fine - just washed out in the middle due to vignetting - hence my needing a flat.  So I introduce a flat - just to get me going - and that happens.

I am 100% certain that it is the application of the flat before alignment which is causing this problem.  It makes total sense - and I have seen this before - it isnt just me.

Both registax 6 and DSS give the same result.

wxsatuser is right, if I understand correctly that you have only used one flat, you need to take and use more. I usually take at least 20 or 30.  I suggest what's happening is that noise in the flat is being added to different places in the final image as the individual subs are realingned (to correct for drift) in the stacking processes.

[Spacehead]

Ok.  What ill do is create more flats.  Ill spend hours on it.  Is it true to say that whilst they may not be perfect for the lights in terms of producing the very best quality final output, the fact I introduce more should eliminate the streaks?  The reason I ask is because whilst my camera angle on the scope will be the same, my focus will not be in quite the same position now.
I have read that people seem to create flats which do a good job without the need for doing them 100% immediately after the main shoot, even using "libraries" of flats they have built up over time.  So is it worth a go?  TBH - I am not sure I will be able to take flats every night after every shoot, I have to take my scope inside and set it up on the dining table with a PC flat screen.  Wouldnt be practical in any sense at all at 4am in the morning!

[Ouroboros]

11 minutes ago, Spacehead said:

Ok.  What ill do is create more flats.  Ill spend hours on it.  Is it true to say that whilst they may not be perfect for the lights in terms of producing the very best quality final output, the fact I introduce more should eliminate the streaks?

I expect it will help. It may not get rid of them completely.  At some level or other we will always see some noise if we stretch an image enough. The aim is to get it to an acceptable level. I will be interested to see what affect it has. 

[Spacehead]

I have corrected the two images I uploaded now - I apologise - my FTP was set to ascii (been doing other stuff with it) so it has probably filled my files with cr/lfs or something.  I have reset the transfer to binary and re-uploaded the two images, one flat and one light.  Would you see if you could use your methods for me to see if there is some noise?
I only use PaintShop X8 atm - I couldnt seem to see any major patterns.

[DaveS]

I never do my flats at the same time as I image. You don't have to and it just wastes time. As Mike said, you need to do 20 or 30 for each filter to take out noise. I do mine in evening daylight, set the cooling somewhere near the operating temp and use about 4 layers of white T-shirt over the 'scope. You need to get your histogram about 1/3 max value.

[alexbb]

This is how your stretched and resized to 50% flat looks like:

And there's something weird about the file itself, when opening in GIMP, it lets me choose between 2 pages.

I also don't do my flats after an imaging session. I only do it once and then I keep using the same flats (for more filters) as long as I don't modify anything in the imaging setup.

The registration of the frames is made using the stars. After registering, the frames themselves are calibrated before stacking so they should be evenly illuminated after the image is divided by the flat pixel values. Therefore, the dust bunnies should not be visible anymore in the calibrated frames that are going to be stacked. The calibration with flats is not that an amount is subtracted from the lights, but the pixel values in the light frame are actually divided by the pixel values in the flat.

Also, throw all the flat .cr2 files to DSS and let it build a master flat - a stack of all flat frames - instead of converting the raw files to tiffs.

Ideally, you should have all the calibration files at the same ISO/temperature, darks at the same exposure length. If flats have a longer exposure (maybe more than 0.5-1s), they need also darks (darks for flats) in order to reduce the pattern noise of the camera from the flats.

The dark streaks are because either there's some cold pixel in the lights or there's a hot pixel in the flats. That's why you should have proper darks: to calibrate the lights and also the flats if they are exposed for long (in a short exposure and with a high amount of light, the SNR is very high so darks for flats are not needed).

You could also improve things much if you slewed the mount on the DEC axis a small amount of pixels (~10) at least 5 times during an imaging session. That would be manually dithering. Ie. if you shoot 60 lights of 1 minute each, after 5-10 frames, slightly slew the mount on the DEC. When stacking, choose the Kappa sigma algorithm in DSS.

HTH,

Alex