Using Siril with a very large number of images

[Legion50]

Im doing a personal project to take a very good picture of the Andromeda. Im in a location with a lot of light pollution (and no transportation to go elsewhere with short notice) and very wet weather generally. I have to take 2 second exposure images since I don't have a tracker (and any longer the photo gets overexposed due to light pollution). I took a total of 26 minutes of exposure the other day and I was very happy with the results (although there is still a lot of noise). I edited the image in Siril.

Im thinking of getting more images whenever I get clear skies and to refine the images further. So my main question is on how to stack multiple day images. I can do the preprocessing from each session and get the pp_ sequence together. but each photo seems to create a 280MB file in the pre processing step (and another 280 MB file in the registering step) which adds up very fast. Even the 790 images from the last session used above 400 Gbs of SSD storage which is my limit.

So I am wondering if it is possible to optimize this process in any way. Like can I stack each session separately and then stack the stacked images?

[Rallemikken]

If you do it manually, you can preprocess and register in batches. Not sure, but I think once you have the _r_pp-files the _pp-files can be deleted??  Once all are  registred you must do some magic in order to gather all r_pp-files in one sequence-file before stacking the whole lot.

I preprocess, register and stack files from different nights as a whole, with just one set of darks, flats and biases. And that is OLD calibration files. I have a feeling many overcomplicate the whole process. My darks have not changed in two years, my DSLR is pretty consistent. And I have one set flats for each camera/scope/reducer combo. No need to take new, I shoot with reflectors and naked sensors. As long as I keep the reducer clean on startup, I never experience those big donuts some struggle with.

[Elp]

To answer your question, yes, just stack the stacks. I do my workflow after each session so at least you can check there have been no issues during the run. You can do your pre processing like DBE, GNR etc but don't stretch them and leave the file in a linear state. Then you just load each stacked image as you would any of the normal files, once you've loaded them all just go through the process again, save as sequence, no calibration required, register then stack. It's a quick process as you've already done all the hard work previously.

Some prefer to stack everything in one go, for the reason I've specified above I don't think this is necessarily wise, especially if you setup and break down every time like I do.

 

[ONIKKINEN]

One way to reduce file sizes is to reduce the size of your images.

Figure out what image dimensions you would like to have, for example 1920x1080 pixels and make your reference frame this size. First crop the image to the framing you want the final image to be in and then do the resampling.

Import this new tweaked reference frame as the first image into the 'conversion' tab to force Siril to take that as the reference frame. Then run global star alignment and every image is automatically cropped and resized to the reference image, so 1920x1080 pixels. You would only need a small fraction of the hard drive space that way as most of the file size requirements come from the registered images.

There is a hard coded cap to the number images though, and that is 2048. If you have more than that number then you have to make a Fits cube, which is a single fits file with as many layers as there were images. This eats a lot of space no matter how you go about doing this, so try to stay under 2048 images in total.

[Rallemikken]

And isn't it possible to work on 16-bit only? If so, you would half the space needed with just one config setting.

[ONIKKINEN]

6 minutes ago, Rallemikken said:

And isn't it possible to work on 16-bit only? If so, you would half the space needed with just one config setting.

Good call, it definitely is worth it here with very short subs that probably dont need 32-bit precision.

The line to change, or add, in the preprocessing script is in the beginning and says "set 32bit", which if changed to 16bit will export all the calibrated subs in 16bit so half the size.

[Rallemikken]

6 minutes ago, ONIKKINEN said:

The line to change, or add, in the preprocessing script is in the beginning and says "set 32bit", which if changed to 16bit will export all the calibrated subs in 16bit so half the size.

Did a quick peek into my old (but still working) scripts, and that line is missing. Looks like it uses 32-bit as default is nothing else is set.

[WolfieGlos]

6 hours ago, Legion50 said:

I have to take 2 second exposure images since I don't have a tracker (and any longer the photo gets overexposed due to light pollution).

If you can, invest in a basic tracker so you can use a lower ISO and use longer subs = fewer files = less HD space. This will also result in fewer shutter actuations for your DSLR so it will last longer. A lower ISO will also yield better dynamic range.

Just on the stacking side, good advise already. You can “stack the stacks”, although another option you can consider when stacking manually with Siril is holding all of the subs and PP files on an external drive. Then using Symbolic Links, you can create a new sequence for multiple nights using the calibrated subs on your HD; and all of those files are 0kb each. The registered files will still be large but it’ll save a lot of HD space.

@ONIKKINEN assisted me with this here, across I think it was 9 sessions for 425 subs. Although the topic started out on a different subject, and covers “stack the stacks”, it all amalgamated into one, and this is now how I am approaching all of my multi session images.

[ONIKKINEN]

3 hours ago, Rallemikken said:

Did a quick peek into my old (but still working) scripts, and that line is missing. Looks like it uses 32-bit as default is nothing else is set.

Seems that way looking at my scripts too. Anyway, i did 16-bit stacking once a while ago and used a script for that.

Editing the script like below will make all processing happen at 16-bit:

So the very first line in the script as "set16bits" forces 16-bit processing.

[StevieDvd]

If you are using Windows and are happy to activate developer mode then Siril will not make a copy of your files for processing but will use 'symbolic links' to the original files. These are just pointers to the files just like a shortcut in Windows. They are used a lot in Unix & Linux systems.

See https://siril.org/tutorials/tuto-scripts/ for details

 

 

[Legion50]

Wow thanks for all the help. Sorry for the late response.

I got a lot of ideas from this thread. I think I will try stacking the stacked images first. Maybe test it out by splitting my current set into two and comparing the single stack vs batch stacked images for any quality difference. I haven't had another clear sky yet to test this out though.

I will try the 16 bit as well.

I may be wrong here, but would using a tracker and reducing the iso actually change the overall exposure? the total integration time increases, but wouldn't the exposure remain similar? I am going from landscape photography to astro, so I keep thinking in terms of the exposure triangle haha.

Edited February 15 by Legion50

[Elp]

ISO (much like gain with astro cameras) is like an electronic sensor amplifier, it cannot affect the laws of physics and incoming photons. The defining factor for AP is total imaging time as long as LP doesn't infringe upon the target signal, more time generally also means less noise in the stacked image, and you'll also get less read noise by taking longer exposures rather than many shorter ones. Most camera bodies I've used around iso 1600, 30s-120s but this depends on the body, your local LP and what you're imaging.

Edited February 15 by Elp

[Legion50]

1 hour ago, Elp said:

ISO (much like gain with astro cameras) is like an electronic sensor amplifier, it cannot affect the laws of physics and incoming photons. The defining factor for AP is total imaging time as long as LP doesn't infringe upon the target signal, more time generally also means less noise in the stacked image, and you'll also get less read noise by taking longer exposures rather than many shorter ones. Most camera bodies I've used around iso 1600, 30s-120s but this depends on the body, your local LP and what you're imaging.

That makes sense. Thanks for the explainer