Need help with AutoStakkert! 2, Flat Frames and Solar Images

[IanL]

I could do with a bit of advice regarding using flats in AS!2 on solar images.  I'm using my Lunt LS60 (Ha) and an ASI174MM with a basic Meade 2x Barlow.  I have two issues; broad horizontal banding which is fairly obvious with an extreme stretch in the stacked images, plus fine circular Newton's rings (barely visible but they are there if I try hard to pull them out).  I can suppress both of these reasonably well in final images, but I know I could do better.  The obvious solution should be to use flats during stacking, but it doesn't seem to be at all effective in AS!2.  (I know I could get a tilt adaptor to deal with the rings, but that isn't the main problem, it's the bands which I assume are from the filters).

My flat capture technique is to aim at the centre of the solar disk and de-focus, capturing a 500 frame SER.  I then use AS!2 (Image Calibration -> Create Master Frame) to generate the master flat (no master dark used).  I've tried experiments with exposures to give about 1/3rd (left)  and 2/3rds (right) of max ADU (stretched in PixInsight to exaggerate the effect, obviously using the unaltered version for calibration in AS!2):

As you can see, the bands, rings and dust bunnies all seem to be present and correct leading me to think (at first sight - but see below for more) I have good flats.  I tend to process my images in AS!2 in batches, so:

- Open a batch of SER files.

- Image Calibration -> Load Master Flat (select one of the flats created above).

- Analyse, select about 50% on the Quality Graph, stack 25% of frames, Auto place AP grid (size 24, Max Bright 10).

-Stack, as per settings below.

The problem is that the resulting output files still show the banding, rings and dust bunnies.  I've tried some experiments as shown below (again stretched in PI to show effect):

- Column 1 - two different examples stacked in AS!2 with no flat.

- Column 2 - same examples stacked with the 30% master flat.

- Column 3 - same but flipped the master flat vertically before using (the stacked images are flipped vertically in PI compared to the original SER files - I think this is just a presentational thing but wanted to rule out some kind of orientation issue in AS!2).

- Column 4 - same but using 60% master flat.

- Column 5 - 60% flat flipped vertically.

As you can see, none of the results are satisfactory - banding remains in all of them (it's less visible in a few examples simply due to the way PI's screen stretch works).

I tried subtracting an image calibrated with the 30% flat from the same image created without a flat, and got the following:

This looks OK, it's pretty much the same as the 30% flat at the top of column 2 above (note that it is flipped vertically, but again I think this is just PI loading TIFF and FIT images in different top down / bottom up order). The geometry is a bit cropped as you'd expect as I used the "Cropped" option in AS!2 so the image should have a smaller area than the flat. The solar disk remnant seems likely to be due to minor differences in AS!2 stacking for calibrated and non-calibrated versions.

I thought I'd check out the flats by using PI's ImageCalibration process to flatten one flat with the other.  Below are the results I got:

On the left is the 30% flat calibrated with the 60% flat (plus its relevant statistics) and on the right is the 60% flat calibrated with the 30%.  Again a screen stretch is applied.  Now if I had good flats, I'd expect to end up with a (pretty much) uniform grey result image.  Instead what I am getting is images with the horizontal banding gone but the Newton's rings clearly visible.  The difference in ADU across the image is about 15% of the possible range.  The only reason I could think for this result is a change of focus or a change of spacing between sensor and the barlow, but both flats were taken consecutively with no such changes (just different exposure lengths to get different ADUs).  The banding and dust bunnies are gone, the Newton's rings remain.  I have no idea what the wrinkly vertical stuff is coming down from the top of the frame.  The horizontal lines across the bottom of the frame are likely to be readout artefacts, probably due to the different exposure lengths.

I'm at a loss as to what to try next.  I would like to try manually calibrating the stacked images (column 1 above) using PI and the master flat, but AS!2 only seems to have options to crop the stacked image to the common area of all frames used OR the maximum area covered by the all frames used.  What I need is an option to create a stack that is the area of the reference frame so that it has the same geometry as the master flat, but no such option exists.  That way I could determine whether I have good flats or not with respect to the SER files. - it would not be 100% effective of course due to the alignment process, but should show improvement on the large scale banding at least.

Any suggestions as to why this is failing gratefully received, or further experiments I could try.

[Owmuchonomy]

Hi Ian that's a very comprehensive study. As you haven't had any replies yet I will just pass on my settings differences to you as I have the same kit. I am not technical enough though to comment in detail. For flats I use the same technique by defocusing on the centre disk but I adjust the histogram to match the equivalent light. Other comments on your AS!2 screenshot: I use TIF not FIT; I don't know what the rotation reference is highlighted in yellow either. I do note that your quality graph is very variable. I never see that profile but something much more gradual. I usually stack 25% of frames like you but I have 1600+ frames to choose from. I always make sure gamma is set to neutral. Not a very technical answer I'm afraid. My Solar efforts are in my link below. Hope you sort it.

[IanL]

2 hours ago, Owmuchonomy said:

Hi Ian that's a very comprehensive study. As you haven't had any replies yet I will just pass on my settings differences to you as I have the same kit. I am not technical enough though to comment in detail. For flats I use the same technique by defocusing on the centre disk but I adjust the histogram to match the equivalent light. Other comments on your AS!2 screenshot: I use TIF not FIT; I don't know what the rotation reference is highlighted in yellow either. I do note that your quality graph is very variable. I never see that profile but something much more gradual. I usually stack 25% of frames like you but I have 1600+ frames to choose from. I always make sure gamma is set to neutral. Not a very technical answer I'm afraid. My Solar efforts are in my link below. Hope you sort it.

Thanks, seems like I am on the right track then and not doing something fundamentally wrong.  I've now tried using TIF as the output instead of FIT; the only difference I can see is the vertical flip when loading in to PI (so the master flat which is always a TIF now has the same on-screen orientation as the stacked image).  The result is exactly the same - obvious banding that matches the banding in the master flat, so it still looks like the flat is ineffective.

The rotation parameter is just one of the experimental settings - not sure why I had it enabled to be honest - probably from an earlier run, but have turned it off and doesn't seem to make any difference.  I've made sure all the other experimental settings are as per defaults (and thus off).

Next I tried exporting a single frame from the SER file and calibrating that in PI using the master flat (should have thought of that before; it deals with the image geometry problem of a stacked image).  Again the result is much the same - banding remains.

So it is pretty clear to me that the master flat generated by AutoStakkert is not a good match for these particular images.

I tried a few other tests like exporting the flat and images to TIFs and manually creating a master and calibrating with it, to little success.

Finally I tried some other captures with the original master flat created by AutoStakkert.  These seemed to work better, though I am at a loss as to why given that the non-working captures were taken just before the flats, and the working ones just after with no changes to the optical train.  (It is hard to be sure though - the banding in an uncalibrated stack of the later images is a lot less visible, probably due to high cloud drifting in, so I may be imagining an improvement that isn't there).

Overall I am left with the impression that AutoStakkert's process for applying flats is somehow very sensitive to the relative illumination of the flat and the image.  This does seem to be a recurring theme with a lot of software that applies flats to deep sky images as well.  Why?  Flat fielding should really care too much about the illumination of the flat frame (unless it is practically black or burned out).  All it should be doing is normalising the range of brightness between the darkest and lightest parts of the flat to map the relative illumination making it through the optical system and sensor and then applying that to the image to correct for it.

The maths is elementary so I am left wondering if the specific implementations are flawed somehow (e.g. inconsistently correcting for hot and cold pixels or something).

Still puzzled.

[Bizibilder]

It may be worth posting your problem on the Autostakkert Yahoo group page.  I've not seen anyone with the same problem that you have and there are quite a few who use Autostakkert regularly without  issue.  I have used solar flats, as described, very successfully in the past but cannot see from your very detailed description where the problem may be occurring.

[IanL]

Yes just signed up to tge group and will seek help there. Thanks again.

[keramos]

Hi Ian,

I use the similar/same kit, Lunt 60 (Single stack) and ASI174MM, I too have great difficulties with getting flats to work with Lunt recorded SER files.

I've use flats before with AS!2 and Quark recorded SER files, there I would slew to the centre of the solar disc and de-focus and then record about 5-10sec of SER file and then generate a flat in AS!2, worked perfect with great results.

Reason I think I could not get flats to work with Lunt is because I could not fill the entire camera sensor with solar disc which causes problems in the final images. I've given up on this method and have yet to try another method where you can put a translucent bag in front of the scope and create a flat in AS!2.

I know this does not answer your questions but thought i'd share my experience with flats and Lunt.

[IanL]

I have te same issue with full disks at native focal length.  I have a diffuser made from translucent acrylic which I have used for flats, but it needs a long exposure and it's hard to get even illumination across the frame.

These images though are with a 2x Barlow and a 1024px square ROI set, so no problem getting full illumination at cente of the disk / defocussed.

[Zakalwe]

I had limited success using flats to calibrate out Newton's Rings. Even trying different flat fielding methods (cling film, de-focused image, FireCapture flat-fielding) I was still getting banding. The best solution is to remove the NRs before processing and to do that a tilt-adapter is needed:

 

 

[IanL]

23 hours ago, Owmuchonomy said:

I do note that your quality graph is very variable. I never see that profile but something much more gradual.

As a quick diversion, I looked in to the quality graph I typically get to make sure I was happy with what's going on during capture:

I took a screenshot of the quality graph from AutoStakkert and divided the (vertical) quality axis in to octiles (8ths).  It's already divided in to quartiles (the blue/grey horizontal lines), so I added the extra horizontal black lines to make octiles.  I then added the vertical black lines where the green line crossed each octile.  Assuming both axes are on a linear scale (they seem to be), by measuring the horizontal distance in pixels between each pair of vertical black lines and converting it to a percentage of the total width of the graph, I determined the percentage of frames in each quality octile (note that the last two lines at the right are immediately adjacent).

I then plotted these in Excel to give the graph at the top of the diagram, showing percentage of frames in the best (1st) octile to the worst (8th) octile.  Fitting a line to it, you can see the shape of the curve matches a Normal distribution (https://en.wikipedia.org/wiki/Normal_distribution).  Assuming AutoStakkert's estimation of frame quality is valid, this is exactly what one would expect from a random process like an image (primarily) degraded by seeing.

The width and skew of the curve would of course vary with the variablity of the seeing conditions.  In this case 48.9% of frames are in the better half of the distribution, so slightly worse than some theoretical "average seeing variability".

This measurement doesn't say anything about the absolute quality of one set of captures vs. another, for that one would have to compare frames from multiple captures and AutoStakkert would have to display some value representing the quality of the frames on an absolute scale, whereas the graph just shows the relative quality of a frame vs. other frames in the same capture.

All things being equal, I would expect that longer exposure lengths would give a narrower / central curve as the seeing variability would average out, where as shorter exposures would give a wider curve as there would be less averaging in each frame and thus more frames towards both the good and bad ends of the quality scale (which is what we want in lucky imaging of course).  Possibly skewed to one end or the other as well depending on whether seeing is good or bad.  Maybe that is why we see different plots.

 

[Steve Ward]

Try running it through PIPP first , I find it's Quality assessment better and it reorders the frames from best to worst , when AS!2 subsequently shows that graph after analysing the set it's much easier to see what's where , and easy to see which frames should be used to stack .

[IanL]

I've done some more tests, and I the conclusion seems to be that I should take flats more frequently during the session.  I processed a couple of captures, one immediately before taking the flats and one immediately after and this is what I got:

The top images are without the flats and the bottom pair are with the 30% flat.  I've use the MLT process in PI to remove all details with a scale below 64 pixels to try and isolate the large scale banding better.  It's really obvious in the top left image and absent in the bottom left.  It's less obvious in the top right and bottom right pair, but just about visible as a dark band in the non-flat version.  (Ignore the posterisation effect - that's not Newtons' rings, just an artefact of the screen shot).

The problem is with earlier images in the sequence, where the flats are not effective.  I am guessing it is some combination of focus/camera  slippage and heating of the optics/etalon causing the banding to shift or change its relative intensity.  Should be able to cure this for single images by taking a set of flats each time, but more difficult where I am taking a run of images for an animation as I tend to set it up and let FireCapture blast away every 30 seconds for half an hour to an hour at a time.

Thanks for the help everyone, will let you know how I get on next time the sun appears and I'm not working (so May 2017 probably!)

[Steve Ward]

Just thinking out loud here , but if you have 'focused' NRs in your captures then defocusing to make your flats will automatically misalign those rings .

Surely leaving things as is focus-wise but using a diffuser of some kind will help ?

[Zakalwe]

9 minutes ago, Steve Ward said:

Just thinking out loud here , but if you have 'focused' NRs in your captures then defocusing to make your flats will automatically misalign those rings .

Surely leaving things as is focus-wise but using a diffuser of some kind will help ?

 

The NRs stay sharps as they are caused by reflections between the sensor and sensor window (AFAIK). Here's a de-focused flat image with the NRs still sharp.

 

Quark Flat

 

I still maintain that getting rid of them (and they are a blooming scourge!) first is better than calibrating them out with flats. Tilt adapters are a necessary evil when narrowband imaging with CMOS cameras.

[Davey-T]

As per Steve above, I use cling film and adjust settings to move the histogram back to match, I think you said you've tried cling film though ?

Dave

[IanL]

Regarding the NRs, you can see them in the flats in the first post in the thread.  They are due to interference effects between optical elements that don't change with focus which is why they are visible in the flats (and so should be cleared by a good flat).  It's typically most obvious due to reflections between the flat optical window above the sensor and the first curved surface (e.g. back of the Barlow).  Any sensor can be affected, but CMOS ones are much better at picking them out.  Not sure why that is, but maybe due to the fact that the photosites are more widely spaced on a CMOS sensor than they are on CCD?

As I understand it, tilting the camera changes the angle of the flat and thus the spacing, causing the rings to get wider until they are not visible, but of course at the risk of not having good focus across the sensor if the tilt is too much.  I would have to get the slide rule out, but a bit of geometry plus the width of the critical focus zone at Ha wavelength should tell one how much tilt is possible for a given setup before that's an issue. 

As I said previously, I don't really see the fine rings in my final images, its the wide banding that is much more objectionable.  I'm not sure of the source of the banding - I guess it could be another set of NRs from a different surface in the train, but they are not curved (so would have to be reflections between two flat surfaces).  I don't see them in full disks taken without a Barlow, so need to think about that one.

I have tried diffuser flats for full disks, but my diffuser is relatively opaque compared to cling film so I have to take exposures of more than a second per frame - painful!

[Zakalwe]

23 minutes ago, IanL said:

As I said previously, I don't really see the fine rings in my final images, its the wide banding that is much more objectionable.  I'm not sure of the source of the banding - I guess it could be another set of NRs from a different surface in the train, but they are not curved (so would have to be reflections between two flat surfaces).  I don't see them in full disks taken without a Barlow, so need to think about that one.

 

The banding was the biggest problem for me as I found that the bright bands were overexposed when the dark bands where at the right level, or the bright areas were exposed correctly and the dark areas under-exposed. Once a pixel is blown out the detail is lost and no flat can recover it.

Regarding the critical focus zone, a Lunt L60 (f8.3) with a x2 Barlow and a camera with 5.8 micron pixels will have a CFZ of about 0.8mm. That'd give a star size smaller than it's Airey disc. I am guessing that solar imaging has a much wider focal zone, so you'd easily get away with 1mm of tilt across the chip. Of course, putting the tilt axis along the long axis of the sensor would be important.

My ASI 1600 had horrendous NRs and reflections with or without a barlow. That camera seems to be way more badly affected than my 174 chipped camera, possible because the pixels are much smaller?

[Owmuchonomy]

Very interesting stuff that I only partially follow. One thing said though that during Solar capture with either my ASI120 and ASI174 I have never seen a sniff of any rings? This is one of the reason I moved to the 174 which according to the ZWO user group does not suffer this problem? Gina and others are trying the 1600 for DSO so it's interesting to follow their threads too.

[Zakalwe]

10 hours ago, Owmuchonomy said:

Very interesting stuff that I only partially follow. One thing said though that during Solar capture with either my ASI120 and ASI174 I have never seen a sniff of any rings? This is one of the reason I moved to the 174 which according to the ZWO user group does not suffer this problem? Gina and others are trying the 1600 for DSO so it's interesting to follow their threads too.

I've heard that some ZWO users don't get NRs (the lucky so-and-sos!). I get very bad NRs with my 174mm and Quark (see the flat that I posted above) and with my Lunt 60mm. As for the ASI1600...mine is unusable without a tilt adapter. The NRs are horrendous in my Lunt. ive tried it with no Barlow and with an x2, a Televue powermate 2.5 and an Antares x3. All are unusable until I put a fairly hefty tilt into the imaging train (after the Barlow, before the camera).

[Owmuchonomy]

On 6 September 2016 at 08:59, Zakalwe said:

I've heard that some ZWO users don't get NRs (the lucky so-and-sos!). I get very bad NRs with my 174mm and Quark (see the flat that I posted above) and with my Lunt 60mm. As for the ASI1600...mine is unusable without a tilt adapter. The NRs are horrendous in my Lunt. ive tried it with no Barlow and with an x2, a Televue powermate 2.5 and an Antares x3. All are unusable until I put a fairly hefty tilt into the imaging train (after the Barlow, before the camera).

Sorry to hear that Stephen.  I must be lucky (for once)!  I too use a Powermate 2.5x.

[Merlin66]

Reviving this thread.....

Is there any update on this issue?

I do get NR with my ASI 174 but these are removed by the tilter. The Moire fringes I believe are an interaction of the CMOS and the cover plate.

 

[IanL]

Haven't really had much time recently to do solar imaging and not much to see most days. I have managed to get fairly decent images without flats or a tilt adaptor. The NRs aren't usually visible unless I try to pull them out. The wider bands are a bit hit and miss depending on exposure time and tuning of the Lunt. Usually OK but sometimes not.

Frequent flats is the only cure but can be hard to tell if needed until tge image is stacked and processed, by which time it is too late