Low Disk Space - Digital Storage Full!!!

[Buzzard75]

I'm running out of room! I know, I can always just buy more hard drives or thumb drives and they are relatively inexpensive, but it raises a question. What do you do when that's not an option, something you can't afford, or you're out in the field and just need to instantly clean up some space? What data do you keep and what do you dump?

At first I thought maybe just keep the stacked data so you can reprocess it when you learn new skills. You could then delete all the raw, calibraTION and calibratED frames, which is a TON. But if you do that, you wouldn't be able to integrate more data at a later date into the stack. You would probably need to keep the calibratED frames before they're stacked. You could then get rid of the raw and calibraTION frames which is still quite a bit of space.

The question I'm trying to answer personally though is, will I ever go back and revisit old data? There are so many targets out there and not enough time for me to get them all, but I want to get as many different ones as I possibly can. On top of that, both my capturing and processing techniques have changed and improved considerably over the past two and a half years. I can't possibly imagine how different they'll be and how much better my data will be in the future. It wouldn't be wise to incorporate old, lower quality data into a new stack with much better data. So, at that point, are the calibrated files even worth keeping?

 

What data do you keep and why?

[TerryMcK]

I've never got to that situation - I try to start with a blank canvas but aware that can't always be done. However when I used to use primarily SLR I used to take darks before a session so could always delete them.

[Alien 13]

I personally only keep the stacked files, I did think you can still stack multiple stacked files if you wanted.

Alan

[TerryMcK]

Just to add to my previous comment I use a separate imaging computer from the one I use to process the data. Apart from the fact that the imaging box does not need to be particularly high powered mine (the PC) has a 250GB SSD and the Raspberry Pi4 has 128GB card inside. That's what I read as being out in the field and I tend to start those with a clean(ish) slate. Is that what you meant?

My processing computer has lots of local storage in the TB range but I transfer the data I want to keep to a server which has even more space (10's of TB).

Edited March 2, 2020 by TerryMcK

[Buzzard75]

55 minutes ago, TerryMcK said:

Just to add to my previous comment I use a separate imaging computer from the one I use to process the data. Apart from the fact that the imaging box does not need to be particularly high powered mine (the PC) has a 250GB SSD and the Raspberry Pi4 has 128GB card inside. That's what I read as being out in the field and I tend to start those with a clean(ish) slate. Is that what you meant?

My processing computer has lots of local storage in the TB range but I transfer the data I want to keep to a server which has even more space (10's of TB).

I actually just got a mini PC so, yes, my imaging PC is separate from my processing PC, but that wasn't always the case. Before I got the mini PC, I actually did have an instance where I completely ran out of space on my hard disk while I was in the middle of imaging. I used the same PC for imaging that I did for processing. I was frantically deleting old data and other miscellaneous files trying to free up space.

Now I have the mini PC and a new processing PC with a bit more storage, but it's beginning to fill up. I have an external I can dump to for the time being, but I know it's eventually going to fill up and was wondering what was worth keeping. I'm thinking calibrated frames and stacked masters (as there aren't usually very many) are probably the right choice. Everything else is just redundant once you have those calibrated frames.

1 hour ago, Alien 13 said:

I personally only keep the stacked files, I did think you can still stack multiple stacked files if you wanted.

Alan

You technically can, but I don't think that's recommended. When you stack images you're averaging the pixel values of the individual frames to increase the SNR. If you tried to take two stacks and average them or take several raw files and add them to an already stacked image, I would think the results would be less than optimal.

[michael8554]

Over the years have you used different cameras, with different image dimensions?  Some software will cope but mine doesn't.

So since 2014 I've gone from a Canon 300D, 1000D, 450D, 500D, 600D, to currently a 6D.

Including a brief flirtation with a 4/3rd Cooled OSC  (tedious to use, and results no better than the 6D)

So I only keep a years worth of everything, older stuff I keep the Autosave and the processed final images.

And in any case I very rarely go back to them.

That lot is 249 Folders containing 102GB.

Even further back I used to do Planetary, and I have thick DVD storage cases full of DVD's of AVI's and Results.

These take up little room so haven't junked them yet.

Michael

[Buzzard75]

59 minutes ago, michael8554 said:

Over the years have you used different cameras, with different image dimensions?  Some software will cope but mine doesn't.

So since 2014 I've gone from a Canon 300D, 1000D, 450D, 500D, 600D, to currently a 6D.

Including a brief flirtation with a 4/3rd Cooled OSC  (tedious to use, and results no better than the 6D)

So I only keep a years worth of everything, older stuff I keep the Autosave and the processed final images.

And in any case I very rarely go back to them.

That lot is 249 Folders containing 102GB.

Even further back I used to do Planetary, and I have thick DVD storage cases full of DVD's of AVI's and Results.

These take up little room so haven't junked them yet.

Michael

Yes, I started with a DSLR a couple of years ago. I actually lost all that data from a hard drive failure though. Following that, I used an uncooled ASI178MC camera for about six months and now I use an ASI183MM-Pro. I don't see myself going away from the 183 anytime soon. Right now I'm just using a single telescope. I wouldn't try and combine data from two different cameras or two different optical trains. Just too tedious. There have been some other changes in equipment and capturing techniques along the way and I'm sure I'll continue to improve my capabilities.

I too have a ton of SER and AVI files from planetary and solar imaging and doubt I'll ever go back to work with them again. I'll probably keep some though and just compress them down as much as possible for show-and-tell purposes, no need to keep the 100's of GB's worth of SER files. I guess that would probably be a good place to start cleaning things up.