How do you keep data organised?

[Mr niall]

Hello there

i was wondering if anyone had any top tips for keeping all your data organised? I wouldn’t describe myself as particularly untidy but by the time I’ve got from memory card to laptop to DSS to photoshop to Lightroom to online storage(!) Im getting a bit lost!

one thing that particularly annoys is the new adobe creative cloud seems to be backing up and saving things I don’t want it to. I think one of the biggest issues is I’m not very disciplined (or brave??) and am ending up with multiple incarnations of a slightly tweaked finished product and not knowing which  one is which! Also I haven’t really landed on a naming convention I’m happy with for files - something succinct but still detailed and useful.

Anybody use the big photo sharing sites - Flickr... that sort of thing?

A.N

[Demonperformer]

I just keep everything on a local 4tb USB drive  - sorted into folders by date.

[wxsatuser]

15 minutes ago, Demonperformer said:

I just keep everything on a local 4tb USB drive  - sorted into folders by date.

Same but sorted by object.

[Mr niall]

18 minutes ago, Demonperformer said:

I just keep everything on a local 4tb USB drive  - sorted into folders by date.

1 minute ago, wxsatuser said:

Same but sorted by object.

Yes I'd considered that - luckily though I'm getting my hands on a personal laptop this week rather than using the familyone so that should cause a few less arguments! Do you not find you have rogue copies of stuff floating around though? Like file uploads and that sort of thing? Or maybe thats just me!

[Space Oddities]

All my pictures are stored on a Synology NAS. It's convenient, because I can access it via Wi-Fi and Lightroom can also use it as part of its library.

One year ago, I reorganized everything and try to come up with the best way to do it. What works for me might not work for you, of course, depending on what you shoot and how you back things up (hopefully you do backups!).

But for what it's worth, here is the folder structure I use:

• Cloud --> My most precious pictures, that are backed up in the cloud automatically (up to 1 Tb, about 7€/month)
  • Theme --> I do a lot of photography, so it made sense to use a "theme" first: Family, Friends, Travel, Macro, Street...
    • Year --> that's often the only thing I remember. I don't need the month, as I don't have that much folders anyway.
      • Event --> One folder for each "event" or "session". For example: "New York June 2017" or "Afternoon with Grandma May 2018" or "Wedding John & Mary Sept 2015"
• Local --> All pictures that I don't want to back up in the cloud, either because it's not vital, or because it's too big anyway (that includes all the RAW data for astro). I back it up on a local disk instead.

For my astro folder, I simply keep things organized by "Year" and then by session (e.g. "Andromeda Galaxy August 2018").

I also give keywords to pictures. It's great, because then it allows you to search for it, wherever it is! If you're looking for a picture of Mars, and you did some in 2016, 2017 and 2018, the keywords allow you to find them quickly.

You can try to use Adobe Bridge, it's a free software that can help you organize your pictures, browse through them and set up keywords on the fly. It works much better than Windows file explorer or the Mac Finder app  

If you have a lot of astrophotographs, you could also have something by theme. When you look for a picture, there's a 99% chance that you're looking for a what rather than a when. That's why it makes more sense to me to have something like this:

• Deep Sky
  • Galaxies
    • Andromeda
      • July 2017
      • August 2018
      • ...
    • Pinwheel
      • November 2017
      • December 2017
      • ...
  • Nebulae
  • Clusters
  • ...
• Planetary
  • Mars
  • Jupiter
  • ...
• Darks, Bias, Flats

Then, if you need to find all pictures your took this year, you can use a software like Bridge with a filter on the year to find them. That's the most convenient way, as far as I know!

[wxsatuser]

42 minutes ago, Mr niall said:

Yes I'd considered that - luckily though I'm getting my hands on a personal laptop this week rather than using the familyone so that should cause a few less arguments! Do you not find you have rogue copies of stuff floating around though? Like file uploads and that sort of thing? Or maybe thats just me!

When processing I may end up with images that I don't like but keep in a separate folder just in case.
These may be files from PS CC or PI.

I don't seem to have rogue files as when processing everything for a particular object goes back in its folder.
In there will be stacked stuff, master dark, bias and flat, from Pixinsight I make sure the files save back to the
original folder where the are dead easy to find, simply look for the object name all the files will be there.

[Demonperformer]

1 hour ago, Mr niall said:

Yes I'd considered that - luckily though I'm getting my hands on a personal laptop this week rather than using the familyone so that should cause a few less arguments! Do you not find you have rogue copies of stuff floating around though? Like file uploads and that sort of thing? Or maybe thats just me!

I avoid all automatic upload sites like the plague. My imaging laptop does not connect to the Internet at all. So I have total control.

I'm getting a lot less stuff now that I have started using Sharpcap's live stacking feature.

[Spaceade]

I find myself keeping every step of the processing process, in case I want to tweak something down the road. Not sure how common that is round' these parts.

[Ouroboros]

1 hour ago, Spaceade said:

I find myself keeping every step of the processing process, in case I want to tweak something down the road. Not sure how common that is round' these parts.

Same here.  I sometimes remove several steps if it turns I'm going in a wrong direction. But I too keep most stages. 

[Ouroboros]

I keep all my astronomy data in a top level folder rather imaginatively called Astronomy. 

Each evening's data is saved in a sub folder - dated and named for the object.

2018_12_03 M45 Pleiades

2018_11_29 North America Nebula 

2018_11_15 Moon

and so on. 

Each of these sub folders is divided into sub folders Lights, Flats, Bias etc. A sub folder called Output contains the results of callibration and stacking. A sub folder called Processing contains all files at each stage of processing. If I process the same data more than once I create the necessary sub folders for that.  I must say my naming of files is a bit ad hoc. I should probably do something about that. 

I'm getting a bit concerned about data bloat. I dislike throwing data away. So I'm thinking of transferring my Astronomy data to a several TB external drive, and backing up this as now using time machine onto yet another external drive.  

 

[Pompey Monkey]

After a couple of scares, I've learnt that the most important thing is to keep your RAW files and any associated calibration files (flats, darks and bias from the same time). Then you can:

• Recreate your images from scratch and..
• Re-process from scratch!

Anything else can be redone or recreated :)

[don4l]

2 hours ago, Pompey Monkey said:

After a couple of scares, I've learnt that the most important thing is to keep your RAW files and any associated calibration files (flats, darks and bias from the same time). Then you can:

• Recreate your images from scratch and..
• Re-process from scratch!

Anything else can be redone or recreated :)

 

I used to just keep the stacked R, G, B etc.

I bitterly regret not having the original RAW files.

 

[cletrac1922]

8 hours ago, Ouroboros said:

I keep all my astronomy data in a top level folder rather imaginatively called Astronomy. 

Each evening's data is saved in a sub folder - dated and named for the object.

2018_12_03 M45 Pleiades

2018_11_29 North America Nebula 

2018_11_15 Moon

and so on. 

Each of these sub folders is divided into sub folders Lights, Flats, Bias etc. A sub folder called Output contains the results of callibration and stacking. A sub folder called Processing contains all files at each stage of processing. If I process the same data more than once I create the necessary sub folders for that.  I must say my naming of files is a bit ad hoc. I should probably do something about that. 

I'm getting a bit concerned about data bloat. I dislike throwing data away. So I'm thinking of transferring my Astronomy data to a several TB external drive, and backing up this as now using time machine onto yet another external drive.  

 

Do the same, then ZIP each folder

 

[Demonperformer]

7 hours ago, Pompey Monkey said:

After a couple of scares, I've learnt that the most important thing is to keep your RAW files and any associated calibration files (flats, darks and bias from the same time). Then you can:

• Recreate your images from scratch and..
• Re-process from scratch!

Anything else can be redone or recreated :)

I might start doing this.

The sad fact is that most of my images are so bad that I don't think any amount of recreating them would help. But I'm slowly getting better ...