ASTAP v0.9.9 with new internal alignment routine for FITS file stacking

[han59]

ASTAP, the free Astrometric STAcking Program version 0.9.9 can now stack deep sky FITS images without any plate solving program. It has a new internal  "star alignment" routine. It will compensate for drift in RA,DEC, rotation and flipped images.

http://www.hnsky.org/astap.htm

feedback is welcome.

Han

-------------------------------------------------------------------------------------------

Star alignment routine

This option allows stacking without any external program. It is not suitable for mosaics. No settings, fully automatics alignment for shift in x, y, flipped or any rotation using the stars in the image.

The program combines four close stars into a pyramid and compares the six pyramid dimensions with pyramids of the first/reference image. It selects at least the six best matches and uses the center position of the pyramids in a least square fitting routine for alignment. 

[pipnina]

This is a very interesting project. I have never heard of it before.

I notice it has support for Linux, and has a source code download. Is it free software or a form of freeware? (GPL/MIT license or proprietary?) I'm definitely intrigued to learn more about the project. I'm presuming you are the (or one of) the project's developer(s).

[han59]

It is distributed with a GNU General Public License 3+ as published by the Free Software Foundation. It is programmed in Object Pascal (Lazarus, FPC)  It is up to now a single developer project by me. At the moment I'm most interested in user testing and experience and suggestions for improvement.  It works for me, but maybe not for everybody. I don't expect anybody to support the code.

You can use it either as FITS viewer & editor with some tools as a basic CCD inspector and main use is now stacking of images. So it has a kind of dual personality.  The initial basic viewer HNS_FV was developed about 15 years ago and only the last 5 months ago the development was restarted and took some major steps forward.

Han

 

 

[han59]

The latest version 0.9.23 can stack L+R+G+B or RGB+L images. Secondly you can select images from several objects in different directories and run the stacking in one run. The stack results are reported in a results tab for easy access.

The program is actively in development and could be updated on daily basis. Bugs could be there. You could help with the development by trying the new version with your raw images and send reports or send your raw images for testing by me. Or you could suggest new features and possibilities which are missing.

For images of the same dimensions/camera the fast internal alignment routine could be used. For different dimensions the astrometric solution could work.

Han

 

 

 

 

 

 

 

[han59]

Importing  raw images from digital cameras

ASTAP is still further developed. The latest ASTAP version 0.9.76 can import  raw images from almost any digital camera, For this ASTAP executes the  DCRAW program from Dave Coffin. DCRAW is included with the ASTAP Windows edition and can be installed in Linux by the "sudo apt-get install dcraw" command. 

The file convert option is in the pulldown menu under TOOLS. ASTAP will execute DCRAW with parameters -D -4 and this will produce 16 bit PGM raw files. ASTAP can read these PGM files and convert them to a 16 bit FITS. After conversion the intermediate PGM are no longer required and can be deleted manually. The FITS can be used in the stacking. The Bayer matrix conversion to colour will be applied later in ASTAP. With this method the full depth of the original RAW files will be preserved.

Han

A raw file prior to applying the Bayer matrix looks like this:

 

 

 

[RolandKol]

Hi han59,

Thanks for the new software which, I hope, someday will make our life much easier.
I have tried it out yesterday and have some suggestions:

1) Master Darks, Master Flats, Master Bias? - is there any option to make them separately and use later? Or at least, save them separately during the first process? I used 250 Bias, 100 Flats - and it took ages to process 40 Lights...

2) Any change to make "optional option"  to choose Light reference image? It helps a lot while processing several different sessions.

3) I was not able to find descriptions for Stack Methods (average or sigma-clip-average - which one to choose and when?), wanted to try one and later another... but the initial stacking was very long, so I went to sleep.

4) Is there any option to generate the Stacking LOG so I would be able to share it with you? as the final result was not good enough... In each sub Nebula was always almost in the centre with some dithering. Staking has the line almost in the centre, like some subs would have Nebula on the right side of the frame... strange...

5) I wanted to make ASTAP as a default .Fits viewer on Win10 - after the right click on .fit file I was not able to locate ASTAP in the default software list, had to press on "Choose another software" and navigate to Programs Files > ASTAP.exe

The result I found in the morning was not very good... Deep Sky Stacker did a better job, but I like the idea of ASTAP -  dropping off everything in one place and wait for the results, plus it is very nice .Fits viewer. 

I used subs from different sessions made via ASI1600MM Pro (subs were done without a coma corrector, so some stars at the corners are really bad, maybe it had an impact?)

[han59]

Hello Ronald,

>>1) Master Darks, Master Flats, Master Bias? - is there any option to make them separately and use later? Or at least, save them separately during the first process? I used 250 Bias, 100 Flats - and it took ages to process 40 Lights...

After making the master, the idea is to have several masters selected. If you select classify by temperature and exposure length, the stacking process will select automatically the dark with the correct exposure length and temperature setting for the images stacked. So after having some masters ready, you don't have to look at them (unless you have uncontrolled temperature.).  I keep them in different directories. I agree now the it is better to give them also a different name so they can be stored in one directory. I will add the exposure time and temperature to the master file name.

The same applies for flats. But since only the flat filter is important you can classify by "flat filter" only. So in the first tab you can select several image sessions and the program will use the correct darks master and flat master to it.

Below my four masters. Note the classify by exposure and temperature. So if my images are made with 200 seconds exposure and temperature setting was -15C, it will use the fourth master only.

 

>>2) Any change to make "optional option"  to choose Light reference image? It helps a lot while processing several different sessions.

The program select automatically the light image with the best HFD (half flux diameter) value, so the sharpest image. I think this is the best option, so I didn't add a manual option. You can add several session of different objects and it should go all automatic. You can add in the first tab light images, darks, flats, bias and they will all end up in the correct tab. However having master dark and flat prepared in advance is a better way to work.

3) I was not able to find descriptions for Stack Methods (average or sigma-clip-average - which one to choose and when?), wanted to try one and later another... but the initial stacking was very long, so I went to sleep.

Average is fine for a few images. Sigma clip is the best but will take three times more time. This Sigma clip will remove outlier slike satellite tracks and cosmic rays. It will remove the so called outliers above e.g. three standard deviations as set.  For this it has to analyse the images times to calculate the mean, standard deviation and in the third step throw away the outlier pixels. What I always do is to stack a few images and if I'm happy, I select all and let it run. You can select to shutdown the computer after completion.

4) Is there any option to generate the Stacking LOG so I would be able to share it with you? as the final result was not good enough... In each sub Nebula was always almost in the centre with some dithering. Staking has the line almost in the centre, like some subs would have Nebula on the right side of the frame... strange...

There stacking log is in the bottom but is is not saved. The only way is to copy the text and send it by email or so. If you could post a part of the images, dark, flats bias/flatdarks that would be good test materials for me.  The program is in active development and every test material often gives new insights.

The best and fastest stack alignment is the internal star alignment . This will be selected automatic. Astrometric is the second option but looking to you images there are more then enough stars so stacking should be easy.

>> 5) I wanted to make ASTAP as a default .Fits viewer on Win10 - after the right click on .fit file I was not able to locate ASTAP in the default software list, had to press on "Choose another software" and navigate to Programs Files > ASTAP.exe

Adding ASTAP as the default FITS viewer during the installation is maybe too invasive. Maybe I should add the option in installation but default off.

>>The result I found in the morning was not very good... Deep Sky Stacker did a better job, but I like the idea of ASTAP -  dropping off everything in one place and wait for the results, plus it is very nice .Fits viewer. 

>>I used subs from different sessions made via ASI1600MM Pro (subs were done without a coma corrector, so some stars at the corners are really bad, maybe it had an impact?)

You could increase the pyramid tolerance from 0.003 to 0.005 but I doubt it is necessary. Optical imperfections will be the same for all the images. Looking to your image, I assume the program reports hunderds of pyramids detected and 20 or 30 matches. Only for mosaic building the optical imperfections could be a problem. Use "star alignment" as indicated below. There is also a test button to show the pyramids detected of the current image. If you use the little slower astrometric alignment select ignore FITS header, otherwise it will use the position in the FITS header rather then solving the image accurate.

Also it is often good to look to the analysed images list and click on image with a high HFD, low STAR COUNT or HIGH BACKGROUND value. These are often problematic images. You could rename them to *.bak with the right mouse button pop-menu. To delete these *.bak files you have to do outside the program. (I didn't wanted be responsible for deleting too many images)

Again if you could drop the images somewhere in the web ,  I could run a test with them and see where the process maybe is having some difficulties.

Han

 

 

[RolandKol]

Aaa, MANY thanks!

"Replace DARKS", means create it from the list of the darks! - I thought I have to have them already.

Very nice one!  I Like the option to create several darks based on Exposure or Temp and in one go!, - never seen such an option in any other software, - I will use ASTAP for this and for Flats as it has a similar option, - very handy.

However, it lacks the same option for Master BIAS - DarkFlats. 

I left to stack 27 Lights with  2 Master Darks and 1 Master Flat, but used 250 BIAS (as there is no option to generate MasterBias).... Used the Sigma Clipping method as my OIII images are very noisy.

00:23:07 Finished in 6899 sec = almost 2 hours.

ADAS Log.txt  Log file has 6750 entries of "Adding bias image*", - it repeats the same action with each BIAS file with each Light Frame (27*250=6750), I am not a specialist in this field,

but in my case, the option to use Master Bias, would make the process around 5 times Faster (keep in mind, that Bias files are very easily produced and I doubt if someone uses less than 50 subs) and other types of software usually do have MasterBias option, so I guess, this method is usable. 

As per my previous image, - it was my mistake.... My apologies...
I somehow managed to overlook and left two subs which been taken while the scope was slewing, they been stacked in... Deep Sky stacker probably rejected them automatically as the result was exactly like ASTAP's now.
This time ASTAP did the job as expected, only the lack of MasterBias prolonged the process.

 

 

 

 

[han59]

>>However, it lacks the same option for Master BIAS - DarkFlats. 

There is no need for master bias/flat dark.  The flat and bias are combined in the master flat. The math is as follow:

masterflat: = (1/n ∑ [flat] - 1/n ∑ [flat darks] )

So when you select flats and bias/flat-darks, then they are combined in the master flat.  You will see the flat being replaced by the master flat and the bias diassapear. The temperature is not so critical but as long the flats and bias are of the same temperature, the master flat is temperature independent. So you could have a master flat of -25 Celsius, and light images & dark of -15 Celsius. the only factor which is important is the filter used to make the flat. So a master flat with an UV/IR filter is different then one made with a H-alpha filter. That why the master flat is only selected on filter name. not on temperature.

 

So when you stacking you SHOULD see for each image file a message:  used 1 dark (master), 1 flat (flat-bias) and 0 bias.  That is the normal situtation. The remarks between brackets are my comments.

If you see " used 1 dark (master), 1 flat (flat-bias) and 50 bias",    it is wrong and it will take much longer.

 

I will modify in the next version the master name and will try to improve above message. Something like:

used 1 master dark (50x), 1 master flat&bias (50x, 200x)

That will make it more understandable. Will take one or two days two implement. So your feedback helps

Han

 

 

[RolandKol]

 

Also, please think of adding Totals somewhere, so you can see the summary before you press stack and probably after the process also. (For example:  Total: 250 BIas, 30 Darks, 30 Flats, 50 Lights, and maybe even, - Total Integration exposure: 3h 1m 5s).  Not necessary, but useful sometimes
 

P.S. ASTAP already crashed 2 times... I have both left hands probably...

 

[han59]

Looking to your log, there is something weird happening. The bias is too many times added. That should not happen. I can't trace back the problem.

Anyhow, I improved the counting of the routine in version 0.9.78. Now the number of flats, and flat-darks/bias is stored in the master flat. The same for the dark used for the master dark. So the reporting is always indicating the number of dark, flats and flat-dark used to create these files.

In the image below, you can see the master has a name and the name indicates 124xflat and 124xflat-dark.

In the reporting windows, these number s are repeated.

The final image name indicates 0xred, 0xgreen, 0xblue, 0*RGB, 3*luminance of 200 seconds.

Also the info is stored in the fits header under keyword history.

Can you test this version 0.9.78?  If you still have the problem of repeated bias files, maybe you should send me a part of your files to test.

Han

[RolandKol]

No problem, I  will do some tests on Sunday evening, will create a new dropbox or gdrive (mine are almost full).

Also, while we do some tests, maybe think of enabling log file saving automatically? 
I had 2 crashes and I am not able to recover the log and I am not able to recover what I was pressing and why, if each button press can be logged (on the app only )  - will give you more data. 

On Sunday, I will be a bit slower, or maybe even record my actions as I presume I will have another crash.

P.S.

Have a nice weekend and speak to you soon!

[han59]

I just have added in ASTAP version 0.9.79 a log file. After stacking it writes the memo text to a file with extension .txt at the same place as the stacked result. But if the program crashes halfway, it will not be there.  For this I would have to write text to a file for each step in the execution. That can be done but even then it will not tell me much more then where the problem starts. The best way to debug the program is to run it in the debugger and it will tell me where it goes wrong.

An other way is to release the program with the debugger info.  I haven't done that up to now but the program will then be 20 mbytes large.

I will leave it for the weekend. If it crashes, try to remember where went wrong and see if it reproducible. If so, and I can repeat the problem while running the program in the compiler & debugger and it will be quickly apparent where it goes wrong.

Memory should be not a problem. I develop and run it on an old desktop with 4 gbytes memory only.

But up to now it has been reliable so assume it will run well. A new data-set is always interesting since it often introduces new conditions, keywords, filter names or file variations. I have also an ASI1600 but up to now have used it exclusively in bin2x2 mode and not in bin1x1.

Have also a nice weekend! Clear skies are expected here, so it is time to make some more deep sky images.

Han

[RolandKol]

Hi again,

You can find some BIN 1x1 subs with calibration files using the link below, I uploaded them first, as you mentioned, you were interested in this type of subs.

https://www.dropbox.com/sh/yr16ovmwgjvb5pq/AAATZ6Acwpj9yuWuZeAs32DUa?dl=0

it took more than 10h to upload....  quite a bit.

I will create the FTP server and will give you access during this week, not sure how much to share, the easiest way would be ALL in one go, but it's a bit too messy and will be more than 100GB, will think on this topic for a while.

Yesterday Updated ASTAP and tried to stack my OIII bin 2x2, generated Master Dark/Flat, had no crashes, but Flats has not removed one "dust bunny" (Pixinsinght did, I have not checked DSS), wanted to upload files to DropBox, but ... lack of the space prevented.

Repeated the stack today in the morning from work and ASTAP performed perfectly   I pressed analyse only on Light Frames and pressed stack, without creating Master Dark/Flat/.

The result below: (29 Lights, 13 darks, 193 Flats, 479 Bias) took 2415 sec =  40min (not several hours as previously) , on Sunday I did generate Calibration Files before stacking by pressing that large button and the whole process did not take more than 30min in total,

not sure why it was faster.

The result is nice and quality is the same as Pixinsight (as to my eye) just Stack file is x10 smaller (PI stack file was around 250MB, - this one 19.36MB, quite a difference, - maybe some data missing? one Bin 2x2 Light sub was 8.0MB:) )

L_CrescentN_OIII_2018-07-07_02-13-50_Bin2x2_250s__-16C_ 0x0R 0x0G 0x0B 0x0RGB 29x250L _stacked.txt

L_CrescentN_OIII_2018-07-07_02-13-50_Bin2x2_250s__-16C_ 0x0R 0x0G 0x0B 0x0RGB 29x250L _stacked.fit

One more suggestion, - "Clear All" button, which would clear all file lists from all tabs, would be handy. Now each tab needs to be cleared separately.

And Tab order could be arranged as per procedure, Ligts>Darks>FlatDarks-Bias>Alignment>Stack>Results>Pixel Math - this would help for the AP new starters
Also noticed one small mistake in Stacked .Fits Header data

"EXPTIME =                 250. / The total exposure time in seconds "

EXPTIME should be 29*250

P.S. Still waiting for the T2>M48 adaptor to attach my Coma Corrector, - I hope I will manage to collect nice Veil Nebula data and stars for Full Hubble Pallete to be fed to ASTAP soon

[han59]

Hello Ronald.

I have downloaded the 3.7 gb files. The zip give an error but all files seems to be there. Enough material for the moment to experiment, so other files could wait. The bin1x1 make the files large. The center stars look sharp enough for this bin1x1.  For my scope I use bin 2x2.

 I already noted that the 500, 350 and 300 seconds darks are in one directory. First try it added all by accident in one master. Making a master using classification could be beneficial.

>>PI stack file was around 250MB, - this one 19.36MB, quite a difference

I don't know what PI is storing in the FITS file, but 32 mbytes is enough for a 16 bit files or 64 mbyte for  -32bit/floating point fits. Must be historical data or something else but not very efficient and requires a high end computer.

>>One more suggestion, - "Clear All" button, which would clear all file lists from all tabs, would be handy. Now each tab needs to be cleared separately.

I did that by purpose. The idea is to have a library with several darks and flats. "Clear all" could remove the usefull links.

 

>>"EXPTIME =                 250. / The total exposure time in seconds "

>>EXPTIME should be 29*250

I thought about it but this value should be a single number/float. Adding 7250 seconds is not the same. Up to now I write the history in the file with many keyword "history", but yes it is maybe good to add it also as dedicated keywords.  If you first build the red, green and blue stack and much later the LRB stack, the information will then not be lost. I will add more keywords similar as for the master dark and flats:

Master keywords are already implemented:

 dark_cnt

flat_cnt

bias_cnt

I have looked around for standard stacking keyword but they don't exist. Maxim DL has the following:

SNAPSHOT=                   30 /Number of images combined                       

Snapshot for me seems the wrong word and it doesn't specify anything about colour.

Also the keyword  NUMEXP as used for more then one exposure of a single plate is also not appropriate.

Deep sky stacker doesn't write historical data in the export FITS. Same with AstroimageJ. Strange, it looks I have to invent new keywords.

For the next version I will add the following keywords:

LUM_CNT

RED_CNT

GRN_CNT

BLUE_CNT

RGB_CNT or OSC_CNT

The exposure of a single image will stay the same. However if you adding different exposures per colour again a problem arises. So maybe I have to add also:

LUM_EXP

RED_EXP

GRN_EXP

BLUE_EXP

Still assuming for all images of one colour, you have the same exposure time.  This will be transparent from start to end and the process can be done in stages.

More later. Han

 

 

 

 

 

[RolandKol]

Yep Han, please accept my apologies for the image quality, as I still cannot use that bloody coma corrector for my 130PDS!
Bought one spacer, which did not match, - posted back, ordered another... Did not match ever!

I hope the last one I ordered will fit  and I will be able to produce some nicer ones and place them as a ASTAP stacking result example  

someday... hopefully soon

"Making a master using classification could be beneficial." - Yes, especially while creating the first stack from different sessions, - one click an off you go

 

 

[han59]

It is good test material.  I also notice that the light images are 300 and 500 seconds. I you try to stack them, ASTAP will add them equally or throw them out in Sigma Clip . To add them correctly the images should be corrected with the exposure duration. I have never done that but that should be no limitation. It could be handy if you decide to re-use images from the past.

For average they should be added as:

  Average= (image1+image2*exposure1/exposure2)/2

So if image2 has only half exposure duration of image1,  then the pixel values of image2 should doubled.  I have to think about this. If it slows down the stacking maybe it is not so good idea, but the calculation is simple.......

So there is still some programming work to do.

 

[RolandKol]

1 hour ago, han59 said:

It is good test material.  I also notice that the light images are 300 and 500 seconds. I you try to stack them, ASTAP will add them equally or throw them out in Sigma Clip . To add them correctly the images should be corrected with the exposure duration. I have never done that but that should be no limitation. It could be handy if you decide to re-use images from the past.

For average they should be added as:

  Average= (image1+image2*exposure1/exposure2)/2

So if image2 has only half exposure duration of image1,  then the pixel values of image2 should doubled.  I have to think about this. If it slows down the stacking maybe it is not so good idea, but the calculation is simple.......

So there is still some programming work to do.

 

Just keep in mind, I was advised by Carole (SGL member) who is also a member of the local astronomy group, - she usually shoots HA and Luminance with BINx1 500-900sec (depends on an object),

the rest (RGB and OIII/SIII) she BINs x2 and does 150sec - 300sec exposures.

I am not sure if it's the best way to go, I am only 4 weeks on Mono, but she was also advised by someone and she gets really nice results. 
This method will be the first method I will test.

To rescale and match images for Pixel Math is quite a time demanding task, - DSS and Pixinsight do Not automate this step,

She said she uses Registar ($179.00 US), which is able to stack, rescale and blend all together in one go.

No idea how challenging this task from a code perspective, it's just some food for your thoughts and ideas.

P.S.

My spacer came a bit earlier than expected! and it matched!    My coma corrector is ON and ready for some Veil Nebula hunt.

[han59]

The last day I implemented two mayor changes in version ASTAP v0.9.80 just released.

1) Now by default all dark, flats and flat-darks/bias files are added to master files. You can throw all files to ASTAP and they will be organized according their keywords. For this you have to check-mark classify on exposure duration, temperature, flatfield filter. The master files can stay selected and stay as a library the program can access.

2) The colour routine has been also heavily modified.  The colour conversion can bet set by a matrix. For example the test files where made with Ha and OIII. For conversion to colour I have set it for image below as:

Red:=Ha

Green: OIII

Blue:=0.8*OIII+0.2*HA

While testing the files, I have the impression that the OIII filter is wider then the Ha filter. The stars are pretty bright compared with Ha stars. OIII is weak but by increasing the OIII factor the stars become quickly very blue. That can be modified in GIMP/Photoshop but I left it for the moment

Please report any problems. More test sets are most welcome. This will help to improve ASTAP.

p.s. Testing 4656x 3520 pixel files took a lot of time time. This morning I added a tool to bin them 2x2 to speed up.

I hope your coma corrector works well. The center of the images where sharp anyhow.

Han

Part of the result with Ronald's test set:

 

Note the classify by check-marks at the bottom!!

 

 

[han59]

Robert,

p.s. do you observe from Woolwich?  For work, I have lived a few year in Ealing but the skies where horrible.

Han

[Atreta]

Hi, 

I tried astap yesterday but nothing worked for me. I imported the canon raw files, converted them to fits, so far so good. But  when i went to stack the files it kept saying no image files found. Am i doing something wrong? The raw files work with dss. 

Thanks. 

[han59]

Frank, most likely you have selected classify by "image object"  and/or  classify by filter.  See options at the bottom of the stack menu. Just un-check these options.

The imported files have no object name nor filter and the program ignores them to prevent they are mixed with images with a object name or filter.

Imported files should be mono. You can make them vissible by double click on one of the files in the list. The program will convert in the stacking process o colour. So an other thing is to set the bayer matrix correct. If you have a image in the viewer, select in pull down menu the tool convert bayer. The image should becomes colour but if the colours are wrong, hit ctrl-Z to undo and select one of the other four bayer patterns as shown in the screenshot above (stacking tab stack method). As soon the colours are correct, hit the stack button.  You could start with a few images to see the performance quickly.

The stack result is shown in the viewer. Historical result are recorded in the result tab. Double click on any file to show it in the viewer. Removal or rename to *.bak goes with the right mouse button pop-up menu.

Han

 

 

 

[Atreta]

Thank you, Han

Gonna try again after tonight's session and report back to you. 

[Atreta]

I checked again, the problem seems to be the conversion from cr2 to fits.  The pgm files look like to have data in it as the size is larger than the original cr2 file, but the fit file has only 2.81kb (every single one has the same size). 

[han59]

Atreta,

Problem found and fixed in ASTAP version 0.9.81.  The PGM files by the DCRAW conversion program were not read correctly resulting in the tiny FITS files. Can you try again with version 0.9.81?

The old FITS files will be overwritten. No need to delete them.

At what resolution are you imaging? Modern digital cameras have a large sensor of 12 or 20 megapixels which could result in oversampling and long processing time.  Binning the raw in the beginning of the stacking process could improve the processing speed and the signal to noise ratio. Since the raw's are colour images using a Bayer matrix, binning of these raw's should be smart to combine the red, green and blue pixel of the 4x4 Bayer matrix. Or can you set the raw resolution? That would be more convenient and efficient than doing binning in ASTAP.

About the problem, for the record I made a last minute change some days ago with was not thoroughly tested. DCRAW writes the PGM files but at one location it writes non-standard char 32 rather char 10. I was testing with PGM files from an other program and did not notice the problem.

Han