How are you storing and archiving data?

[dyfiastro]

Good afternoon everyone

I am trying to figure out the best way long term to Archive and store my data.
having already done this for many years with traditional photography, I already have a set file structure and Archiving process that I follow.
I am wondering how best to approach this with astrophotography and even more so in regards to very large SER and AVI files.

Does everyone save Large SER and AVI files in their native format or is there a better way of doing this?
I was thinking about possibly encoding into a more compressed lossless format for archiving purposes once the final stacks have been processed?
The main reason for asking this is due to realising that the SER files from one of the lunar mosaics I did last year alone currently come in a just shy of 65GB,

Any input or experiences on this would be great.

Thanks in advance

[Alien 13]

I used to save every sub but quickly gave up and now only do the stacked image, a part of me said all this data will be useful in the future when computers and software gets faster/better but I am not sure now I think tomorrows methods of capturing data and sensor design will make today's efforts of little use.

Alan

[cgarry]

Some people archive all their data, some do not bother to archive any data and all the various options in between will be taken by others.  See this https://sites.google.com/site/astropipp/example-uasge/example7 for details of using PIPP to convert your data to an AVI file with a lossless compression.

Cheers,
Chris

[dyfiastro]

2 hours ago, cgarry said:

Some people archive all their data, some do not bother to archive any data and all the various options in between will be taken by others.  See this https://sites.google.com/site/astropipp/example-uasge/example7 for details of using PIPP to convert your data to an AVI file with a lossless compression.

Cheers,
Chris

Thank you very much indeed, looks like I may have to take a closer look at the inner workings of PIPP.

Until now I have saved every usable sub and video file I have taken.
I am very much of a school that saves every raw file and has at least two backups. Trying to do this with data sets this large is proving to be somewhat troublesome.
Looks like a few new hard drives are going be on order and few nights of encoding.

Thanks again
 

[frugal]

I have an 8Tb NAS box that I keep a backup of all of the CR2/FITS files on. My general process is to capture on the cheap toughbook, and then at the end of the session, I copy all of the files to the NAS box, one folder per day. Each file has the target name in the title as well as the date (.../2017-04-08/M101_Light_ISO800_300sec_f0012_2017-04-08.FIT). So when I want to process something, I can search for and copy all of the "M101" files onto the PC to process. I also keep a Flats directory for each day.

[laser_jock99]

Lot's of 3TB hard drives is my solution. Wouldn't dream of deleting any of my hard won data for the simple reason you never know what processing algorithms may come along in the future or what new processing techniques you'll learn. Large hard drives are cheap enough these days,

[kirkster501]

Regarding nightly data sets I keep all original data (lights and flats). I use master dark and BIAS.  I make sure that I save a copy of the master BIAS and Dark on a quarterly basis and can refer back to them with the data sets in case I ever want to rework the raw lights and flats. I also keep the calibrated lights and the LRGB stacked masters before processing.

The way I store it is to have a directory name of the object M1, M42, NGC7000 etc.  Then under that main object directory a sub-directory of the date with the data set under it.

[nightster]

I use a 3Tb WD MyCloud  NAS with 2 WD Red drives in a Raid array on my home gigabit network. Same as others, bring my laptop home plug into the ether and upload the data. That way I can access it from my Win10 box, the Linux Workstation or remotely from my phone or laptop. I'm fortunate to have fiber 1 gigabit networking at home so bandwidth out to the Internet isn't a concern. 

[dyfiastro]

Thanks for the input everyone.

I am just in the process of building an obsy, once that is finished I will be getting another internal hard drive as well as a couple of new external hard drives for backups.
The obsy will have a Gigabit Ethernet connection going to it so downloading images from the obsy computer is not going to be a problem.

Thanks again everyone.