Data Limitations

[Clarkey]

I have recently started to image at a remote site and the amount of data I will be getting is going to be a lot more than I get currently. (Hooray - clear sky). However, the negative of this is that I will have a lot more data to process and store.

To reduce the total amount of data, I could limit the flats / bias / darks etc. by saving the master frames only. However, currently I keep all of the original light frames. My question is this; when I have multiple data sets over a number of nights (or scopes), will I see any difference integrating the master light frames only over re-integrating all of the frames together? If there is no difference, I would only need to keep the masters, drastically reducing the overall amount of data stored.

I realise that data storage is relatively cheap, but there is also the backing-up and management of all the data which adds to the time / cost.

What is the general consensus? Do I need to keep everything, or just the master frames?

[vlaiv]

If you want to exploit "advanced" features of stacking algorithms - you should really keep all the files.

Conditions change even during the course of single evening and will certainly be different on a different nights.

Simple / naive stacking algorithms can be used with simply creating sum stacks and noting number of subs in such stack.

Then total stack will be created by summing sub stacks and then divided with sum of total number of subs (regular average). This approach does not let you discard some subs based on statistics (you can for example discard bad sub on single evening - but what happens if that sub is better than say several subs on some other evening - having all the data can let you set rejection threshold more carefully), nor does it let you use weights per sub depending on their quality (no way to assign global quality on a single evening until all subs from future evenings have also been recorded and examined).

You also can't do sigma rejection "en mass" - only on particular evening. Sometimes satellite trail is so faint that you can't form reliable statistics to reject it with only subs on single evening - but total number of subs can help with that.

[Fegato]

As vlaiv says, best result will always come from a single integration of all available subs. If the data is good, I keep all my light and calibration frames. Mind you, I have Welsh weather...  so maybe there aren't so many of them!

Must admit, the thought of having my own scope sitting in Extremedura is so tempting at the moment. Here I shoot F/2 to try and get the most out of poor skies. There - maybe F/7 max, so I don't get too overwhelmed with data! 

[vlaiv]

Here is idea of how to reduce amount of data.

Not sure how long are exposures, but if you simply calibrate your subs from each evening and split them into groups of consecutive say 4 or 5 subs and simply add those subs - it will be like you took longer exposures.

If you image for say 2 minutes - it would be like having 10 minute subs (as shooting for longer is equivalent to mathematical addition - except for read noise, but you should be already exposing for long enough to swamp read noise anyway).

Alternative is to simply do longer subs without math. Integrate with analog device instead of digitally

 

[Clarkey]

I think @vlaiv and @Fegato, you are sort of telling me what I already knew. I was rather hoping I had missed something obvious and there was a way round saving all my subs. I know I am probably going to get at least double the amount of data I have had over the last 3 years - that is a lot of disk space. I even avoided full frame sensors to keep the size down. I will potentially have the option of saving binned files which will help.

The idea of grouping exposures might be a good one. I'll have a thing about the best way to automate the process.

[ONIKKINEN]

Time to invest in new drives i would say, 4TB SSDs are really not too expensive these days and will surely store your raw files for a while. I would recommend storing only the raw data, and not the calibrated (or worse: debayered) files until you are finished with the target and start stacking.

For data handling not sure what software you are using but Siril is the king of large datasets. Really nothing else comes close to its speed when calibrating/registering/stacking.

[Clarkey]

4 hours ago, ONIKKINEN said:

Time to invest in new drives i would say, 4TB SSDs are really not too expensive these days

I already have two SSDs and an additional HDD in the PC giving me 8Tb total. I also have 2 associated external drives to back these up. I use the SSDs for the OS, programs and one is a 'working area'. The rest is just storage. I probably need a couple more 8Tb drives to keep enough space for a couple of years.

[ONIKKINEN]

3 hours ago, Clarkey said:

I already have two SSDs and an additional HDD in the PC giving me 8Tb total. I also have 2 associated external drives to back these up. I use the SSDs for the OS, programs and one is a 'working area'. The rest is just storage. I probably need a couple more 8Tb drives to keep enough space for a couple of years.

Now im curious, just how much data are you planning on getting?

1000 subs with the 571 is only 50gb, and will be a deep image even with shorter (say 60s) subs.

And yeah, i get the aversion to getting more space. I think i am sitting on somewhere around 4-5 terabytes of data (not all of it just raw) for 3 years myself, and because of that the plan is to try and take longer subs to try and not let it get out of hand too much.

[Clarkey]

I'm probably being a little pessimistic in terms of data needs, but I would rather have too much space than not enough. The site expects to get around 200 clear nights per year - which is a lot of data. I generally double what I need to give me a mirror copy. As a syndicate member I have no say on what other people image, or the sub lengths - however, the maximum for the scope would be 180 secs.

[kirkster501]

I keep all the good files - lights, flats, flat-darks, bias, darks.  I use a darks/bias library that I refresh twice per year so it is not that onerous.   I am far from an expert but I have reprocessed my images in light of my improving skills numerous times and I am glad I kept the data.  I keep it all on my NAS.  In 11 years of imaging I now have about 5TB of astro data on my 40TB RAID6 NAS.  Storage is cheap nowadays.  However, I do get rid of the intermediate files that get created as part of processing.  No need to keep those.

I never keep the raw capture files from lunar/planetary/solar.  I only keep the outputs from Autostakkert.