Archive data files, what do you keep?

[tomato]

My portable HDD died last night, and let's just say I did not have the most rigorous back up regime in place.

So going forward, I was wondering what approach do folks take to storing and retaining historic data? Do you keep everything including the raw subs and any interim processed versions right through to the finished article? That is soon going to be a lot of data, so I was thinking of keeping calibrated subs, as this is a sort of milestone point in the processing workflow, then any 'finished' versions post stacking and processing.

Incidentally, my Seagate drive lasted 14 months from new, but it was getting a hard life being out in the cold and bundled from one laptop to another, plus quite a bit of read/write activity. Is that about par for the course or should I have been expecting a bit more out of it?

[vlaiv]

Currently I keep everything, but at some point I will have to start doing some housekeeping on files.

At that time, I'll probably keep only calibrated subs, but that depends on me fully developing best calibration method for different scenarios. Have not yet explored impact of different calibration optimizations and am in the process of writing my own calibration software. Here I think mainly of "additional" features in calibration - like hot pixels removal techniques (replace by mean / median or leave Nan?), any sort of entropy based optimizations (dark and flat optimization), etc ...

I would expect HDD to last at least couple of years (some of my hdds lasted for a decade or so, even powered 24/7), but also I'm aware that HDDs can die on you at any moment. I think that probably best approach for storage is to use SSD as main / work storage, and HDD as "long term" backup / storage. Currently I employ a small storage server that has advanced file system (ZFS) and raid setup on it, so data is fairly protected. I use SSDs as my work drives (both in laptop for field use and in my desktop machine).

This might be a bit of overkill but I'm IT professional, so a lot of data besides astro ...

[wxsatuser]

Had a Seagate drive that ran 24/7/365 for 7years.
It was constant read write with a weather station and a EUMETCAST setup.

BTW
I keep all files I take myself plus everything I get from DSW and ITU, runs into hundreds of gigs.

[JamesF]

I keep everything too.

I'd expect a disk to last longer than that, but perhaps a portable one is subject to a bit more abuse than when sitting still on a desk or in a rack for years.  I'd agree that an SSD is the way forward.

James

[michael8554]

If I give my earliest work an honest appraisal i realise that I have improved and that they're not worth keeping.

So I keep the finished image only for nostalgia.

Later work up to about 2 years old, I keep the DSS Master Flats etc, the Lights and the best result.

Also if you don't  have software that copes with images wiith different pixel counts you might consider data from your superceeded cameras to be obsolete.

Having said thst, some are saying that DSS 4.1.0 is giving better saturation, so keep all your old data for restaking on those infreqent rainy days that we get......!

Michael 

[artoleus]

I have backups running daily to an external drive using Macrium reflect imaging.  I also have a Microsoft Office 365 account so I get 1TB of storage with that. I tend to keep my image files in my Onedrive folder.  This means it is automatically backed up into the cloud too.   I too have started considering what I actually backup, at the moment it is everything, later I may just save the final image, raw files and a Pixinsight project without images.

[Stub Mandrel]

I keep a 'history drive' which is great for when you accidentally edit or delete something, but being permanently attached it's vulnerable to a cyber attack/virus. I also have a full system + data  backup on a separate drive as well as rotating data backups.

If anyone can recommend a GOOD, SIMPLE, RELIABLE incremental backup program, please let me know - my holy grail is a simple password protected encrypted incremental backup.

[Martin Meredith]

11 hours ago, Stub Mandrel said:

I keep a 'history drive' which is great for when you accidentally edit or delete something, but being permanently attached it's vulnerable to a cyber attack/virus. I also have a full system + data  backup on a separate drive as well as rotating data backups.

If anyone can recommend a GOOD, SIMPLE, RELIABLE incremental backup program, please let me know - my holy grail is a simple password protected encrypted incremental backup.

I'm guessing you're not on a Mac? If you were, Time Machine does exactly what you want

https://support.apple.com/en-us/HT201250

I simply plug in a hard disk every now and then and forget about it (it also runs wirelessly if you have a Time Capsule). Not only that, it makes recovering files a doddle, so much so that I no longer worry about versioning any code I write -- I just delve into the backup and choose a date and you see the entire file system as it was on that date. Super reliable.

Maybe there is something similar for other OSs.

Martin

 

[Stub Mandrel]

On 05/05/2018 at 08:19, Martin Meredith said:

I'm guessing you're not on a Mac? If you were, Time Machine does exactly what you want

https://support.apple.com/en-us/HT201250

I simply plug in a hard disk every now and then and forget about it (it also runs wirelessly if you have a Time Capsule). Not only that, it makes recovering files a doddle, so much so that I no longer worry about versioning any code I write -- I just delve into the backup and choose a date and you see the entire file system as it was on that date. Super reliable.

Maybe there is something similar for other OSs.

Martin

 

File History in Windows does that, but I leave it connected 24/7.

I specifically want something to do a straight incremental backup with no frills because it's faster and takes less room.

The Windows a 7 backup & restore does this but is soooo clunky.

I've used Acronis in the past, but it is utterly bloated with useless features in an attempt to make it look better value for money.

Looking at alternatives.

Backup software is the ONE area where simple is best. I want to choose what to backup once, and then simply have to plug in a disk drive and press go every time I want it updated.

 

[Hicks]

I don't really need to keep all my earlier raw exposures, but until HDD space on the server becomes an issue I'll keep everything, the good bad and the ugly Obviously I keep copies of my final processed images too.

If it ever did become a problem I'd likely go through and delete bad frames and archive the rest off to bluray (Couple of copies with redundancy via DVDisaster to free up disk space). If I was desperate for space, I'd go through and generate master bias/flat/darks and delete the raw files for those.

We don't really get enough clear nights in the UK for me to be worried about disk space though plus the FreeNAS server I keep my images on has 3 hdd slots spare so there's ample room to expand further. Main limitation there is cost of extra backup drives to cover any expansion.

[Synchronicity]

Neil

Have you looked at AlwaySync  https://allwaysync.com/ ?

It allows you to set up multiple jobs which can be time based or when a drive is connected.  I used it to sync data from several applications with a local web server and a remote ftp server at the same time.  I'm not sure if you can get it to do incremental backups but it will certainly sync any changed files.

I also have a family subscription to MS Office which gives 5 users 1TB cloud storage each.  I'm nowhere near needing that for imaging files -yet - but when I do I'll fill mine and the spare then start eating up the kids allocations :-)

All the best

Michael

[Stub Mandrel]

1 hour ago, Synchronicity said:

I also have a family subscription to MS Office which gives 5 users 1TB cloud storage each

So do I, but crap broadband means the cloud isn't a reasonable backup option for me.

[ollypenrice]

I can't keep all the raw data. It just isn't possible since there's so much of it with three cameras running all night. Instead I save the linear calibrated stacks, noting how they were made, so it might be Red 10x10, meaning ten subs of ten minutes. That way I can shoot another set and stack the calibrated stacks, weighted according to their sub count.

Not strictly related but worth mentioning: unless I really can't do so because of framing, I image aligned along RA and Dec, not at a random angle. It might be landscape or portrait but not random. Coming back to a target is then much easier.

Olly 

[Kevin Morefield]

On 28/05/2018 at 06:43, ollypenrice said:

I can't keep all the raw data. It just isn't possible since there's so much of it with three cameras running all night. Instead I save the linear calibrated stacks, noting how they were made, so it might be Red 10x10, meaning ten subs of ten minutes. That way I can shoot another set and stack the calibrated stacks, weighted according to their sub count.

Not strictly related but worth mentioning: unless I really can't do so because of framing, I image aligned along RA and Dec, not at a random angle. It might be landscape or portrait but not random. Coming back to a target is then much easier.

Olly 

Olly,

When I want to revisit an object to add more data I do a plate solve of an old sub in TheSkyX and that will center and rotate the current FOV indicator to be identical to the old sub.  Then I impart that FOV as a new target in CCDAutopilot.   Often though, I just leave the target in CCDAutopilot but uncheck it till I want to revisit the following year.

Bisque calls their platesolve “image link” I believe.  

Kevin