How much intermediate processing data do you keep

[old_eyes]

Using a CMOS OSC camera (QHY168C), I have a lot of large subs. This gets expanded with all the Pixinsight intermediate stages - calibrated, cosmetic correction, debayered and registered. In XISF format these are large files - nearly 200MB each.

I have just doubled the size of my hard disk, but with the clear nights we have been having I am filling it at an alarming rate. Hard disks are cheap (particularly in the context of this mad hobby), but my sense of the ridiculous is kicking in, and backup needs to be considered. People say, just keep the original subs and the final processed image, you can always reprocess, but reprocessing hammers my computer, even with a decent i7 processor and an SSD for swap files.

I need to move away from my previous policy of hording everything, just in case. I am thinking that I just need the raw subs and the debayered version. I can junk the calibrated, cosmetically corrected and registered. My most likely use case for revisiting a set of images, is either to combine them with subs from other nights/sources, hence keeping the debayered, or mess with the calibration or integration parameters, in which case I would have to reprocess from raw or debayered anyway.

What do you do? How do you handle exploding data?

[Dr_Ju_ju]

You'll soon need to get a SAN solution e.g. something like this https://www.synology.com/en-uk/products/series/enthusiast fitted with 4\8 TB disks, but will obviously depend on your pocket depth.

This then just sits on your network and you can store all the images directly to it.... 

I have 2 16 TB stores, a Synology Diskstation and a Netgear ReadyNAS, with a smaller Buffalo for use @ star parties....

[old_eyes]

11 minutes ago, Dr_Ju_ju said:

You'll soon need to get a SAN solution e.g. something like this https://www.synology.com/en-uk/products/series/enthusiast fitted with 4\8 TB disks, but will obviously depend on your pocket depth.

This then just sits on your network and you can store all the images directly to it.... 

I have 2 16 TB stores, a Synology Diskstation and a Netgear ReadyNAS, with a smaller Buffalo for use @ star parties....

Yes I have a synology NAS (DS411+II) that is my backup, plus a cloud solution. So three locations for my data (I'm old school IT - if it is not stored in three separate places it does not exist!). Working on 4TB disks at the moment as it is a price sweet spot and not at the performance bleeding edge; therefore, I hope, better reliability.

It is just that I can see a successful hobby of astrophotography overwhelming any reasonable storage strategy if I keep abolustely everything. We are not talking CERN-LHC here, but it is an astounding amount of data for someone who remembers the wonder of a 1GB mass storage unit that used tape catridges and a robotic arm to stuff them in the slot!

So what do you do Julian? Keep everything or do you have a strategy for weeding?

old_eyes

[Dr_Ju_ju]

I always keep the originals, then I keep all intermediary files till I've produced an image I'm happy with, but after that they get dumped, as I have the originals all intermediaries can be re-created..

[old_eyes]

1 hour ago, Dr_Ju_ju said:

I always keep the originals, then I keep all intermediary files till I've produced an image I'm happy with, but after that they get dumped, as I have the originals all intermediaries can be re-created..

Thanks! I will ahve a look at how much space I can save if I keep the subs that are calibrated and debayered.