Full hard drive - external image data management.

[Ouroboros]

The TB solid state hard drive on my MacBook is almost full. Over 500 GB of it is imaging data - the bulk of it in raw and tiff data files, plus Pixinsight processed files. 

I hate throwing data away and, as they say, memory is cheap.  So what do do?

I could just archive my data pre-two years ago on an external hard drive. Then delete the transferred data from the MacBook.  This would clear a lot of space. Every year I’d just archive data more that two years old.  I’d keep a backup of the archived data too.

Alternatively, I could transfer all my imaging data to an external drive, and delete it in the MacBook. This externally saved data could be backed up using Time Machine. I assume the disadvantage of this would be that Pixinsight would process the externally stored data more slowly than when the data is on the internal hard drive.

A further alternative I suppose is to keep just the data I’m working on on my MacBook’s drive. Then archive it on the external HD when complete, and backup Time Machine as before.

Any alternative suggestions? I’m racking my brain on how best to do this. 

 

 

Edited February 7, 2022 by Ouroboros

[DaveS]

I keep the original data on my 16TB NAS (4 x 6TB in RAID 5)  and only import what I need. My astro data sits on a 500 GB PCI-E NVME card.

[david_taurus83]

I delete the PI preprocessing files. I keep only raw lights, master flats etc and the stacked masters.

[StarryEyed]

Think I have 12 TB in total on two servers. Not all astronomy. I dont throw anything away. The astro photos go back to 2002.

Hoard it! 

Edited February 7, 2022 by StarryEyed

[Ouroboros]

35 minutes ago, david_taurus83 said:

I delete the PI preprocessing files. I keep only raw lights, master flats etc and the stacked masters.

Yes, I’d wondered whether to do this. I tried it once and decided it was a bit fiddly and I was worried in case I deleted the wrong stuff. Which leads me on to StarryEyed’s comment …..

[Ouroboros]

30 minutes ago, StarryEyed said:

Think I have 12 TB in total on two servers. Not all astronomy. I dont throw anything away. The astro photos go back to 2002.

Hoard it! 

That’s my thinking. 

[Clarkey]

I do keep thinking of cloud storage in case the NAS even gives up the ghost. Amazon photos claim unlimited storage - would this count for hundreds of subs I wonder?

[Ouroboros]

32 minutes ago, Clarkey said:

I do keep thinking of cloud storage in case the NAS even gives up the ghost. Amazon photos claim unlimited storage - would this count for hundreds of subs I wonder?

I feel I prefer to have my backup local. I can see cloud storage might work well as a second backup. Blooming slow I’d have thought. I use icloud to store and share photos and a few documents between my iPad, iPhone and MacBook but nothing important. My proper camera photos are saved and backed up locally.

Cloud sharing is not the same as cloud storage of course. 

[caumargu]

Your hard drive will be full sooner or later. I ran into the same problem two weeks ago. However, I deleted all the data from it mistakenly when I wanted to clean it. Therefore, I had to contact data recovery services near me to solve the problem. Btw, you can use special programs too. This is the easiest way to analyze the occupied space. As a rule, they can produce results in the form of a convenient diagram in 1-2 minutes. This way you will find out which folders and files take up the most space.

Edited April 25, 2022 by caumargu

[Ouroboros]

😯 aghhhh! The thought of accidentally deleting everything fills me with horror. That’s why I keep two external backup drives to which I back up everything alternately.  I hope all was recovered satisfactorily, @caumargu. 

What I decided to do in the end was to archive all my astronomy stuff up to two years ago to an external drive, which in turn is backed up to another drive. This freed up a lot of space on my MacBook’s HD. I shall keep my most recent stuff, or what I’m working on, on the MacBook and then archive it at necessary, probably year on year. 

[teoria_del_big_bang]

I decided right from the start to keep at least backups of all my imaging stuff on an external drive, as you say compared to most astro equipment memory is cheap,  so I bought a 8Tb NAS hard drive that is actually two 4 Tb drives and backs up data on both at same time so if one drive goes faulty you still haver it on the other.

But then, not being a computer wizard, at first did not realize why mu PI processing was so slow and thought maybe just that's how PI was performance wise until back end of last year it occurred to me that just as you said the external drive over WiFi was the reason for the slow performance.

On 07/02/2022 at 16:54, Ouroboros said:

I assume the disadvantage of this would be that Pixinsight would process the externally stored data more slowly than when the data is on the internal hard drive.

So I bought one of THESE and what I do now is copy all required data for whatever project I am working on to this SSD (sometimes maybe just the PI saved project) and then work from that as that seems to be just as fast as an internal drive.

Another reason I chose this method is that I have a desktop and two laptops with PI installed and then can use either computer to continue the processing.
Unfortunately this drive had increased in price a lot since I bought it bit it is a great bit of kit, I can also save data direct to is as it is very small (about size of a swan vesta match box) and and easily plugs into my cpu on the mount running NINA.

Yes, its a bit of a faff transferring data back and forth if I want to reprocess some old data but not too bad as transfer is pretty quick and often I only have to transfer a copy of the PI saved project.

Steve

Edited May 16, 2022 by teoria_del_big_bang

[discardedastro]

Bit more of a techie solution - I use a mix of local storage in my desktop (some fast NVMe disks) for stuff I'm actually working on in PI etc, and a file server (some hardware thrown out from a previous job, but it's a ZFS JBOD with about 60TB of disks for 30T of very, very redundant storage) stores all of the data long-term.

I still don't worry about much intermediate data - I store masters, PI projects, and raw subframes/calibration frames.

Performance-wise, PI's major bottleneck in most high-end PCs is usually storage, so fast local storage is definitely order of the day for performance. If you've not got NVMe disks, you can get PCIe to NVMe adapters really cheaply, and then use NVMe M.2 disks - an easy and cheap upgrade!

I have a stack of Python scripts and a PostgreSQL database I built to analyse and store all the subframes in an organised library so I can with a simple script automatically retrieve all the data I have for a given target along with calibration frames and metadata - and then I have a PixInsight script which automatically does all the calibrations for those collections so I can query any sky position and get to a big pile of calibrated subframes ready to process without having to manually go through all of it filter-by-filter, session-by-session.

https://www.talkunafraid.co.uk/2022/02/cataloguing-astrophotography-data-for-fun-and-profit/

Backups - I use Backblaze on the desktop which backs up everything there and keeps it for a year after deletion or moving back to the file store. This is pretty dependent on a fast internet connection but I have a gig symmetrical line. I also use ZFS snapshots, so if I accidentally delete everything on the store I have a full year to get it back from a snapshot. I use borg for most of my off-site stuff but don't currently back up my image data off-site - disks in a datacenter still cost quite a bit! I have pondered Amazon Glacier but it needs a bunch of tooling I've not played with yet.

Moving data from the ZFS pool to desktop is fast enough with gigabit Ethernet between both boxes. Might move to 10G soon but it's pretty decent at 1GE.

[Ouroboros]

All sounds a bit technical for me of little brain.  But you seem to have got it sorted which is good. For me I need to keep it as simple as possible. 

I do wonder though whether I should have some off premises cloud backup in addition - not just for astro stuff but everything. 

[Earl]

We have his and hers NAS full with decades of data, lots of memories.