Organising your imaging files

[mikeyj1]

Even after only 15mths of imaging I have many folders and files of various nights imaging, and its all getting a bit messy.   So here's the question, (before I get swamped with unknown and unneeded files):

What's your suggestion for how to catalogue and store the files (eg by date, NGC number, camera type etc..), and should I only keep Lights, or also master Darks and master Bias?  

thanks

Mike

[kirkster501]

Great question.  I struggle with this too.  If anyone has a better idea then i'm all ears...!  It is very messy and difficult to keep track and I need a better system too.

In my case, I have an object folder at the top level, followed by date.  Following that is the scope camera combo I used because maybe I might use a different scope on another night and may want to combine from different scopes.  Following that is lights/flats etc.  I use master bias and darks, which if I refresh I must leave the ones for the past imaging run in the folder too.  My filenames have temperature, filter, time and date, scope etc in the name too so I can find them quickly.

So. 

M51 > 15 May > FSQ_8300 > lights

M31 > 1 Jan > TEC140_Atik > flats

etc.

 

That said, I have forgotten to do flats before and wasted that night's run.  if I do not get flats done I now leave a big sign on the scope saying "need to take flats".

 

[cjdawson]

The answer that I've come up with for this is to use Adobe Lightroom.  I'm sure there's alternatives if you want.    What I've done is create a single catalog for my images.  I import all my raw files into lightroom and tag the files - these tags tend to be imaging session (dateandtime the session started), subject and type of frame (light, dark, etc)   What I'll then do is align and stack the images - using the original files when I can, And import those as well, into the same collection and tag them as aligned, stacked.   Finally, when I process the final image I'll tag that as processed.

Once in a while, I'll break everything up, and start a brand new catalog.  This helps to stop me going nuts with loads of older images.  Also helps to mitigate the "all eggs in one basket" thing.  If I want to keep the old processed images, I'll export them from lightroom, then import them into the new catalog.

 

Of course, this is just one way of doing it, I'm sure that there are others ways too.

[mikeyj1]

OK, well either its wasn't such an interesting question as kirkster and I thought, or your answers coved all possible options? 

I already use lightroom for the processing, but a separate catalogue is a great idea! 

I will also organise my files in a similar hierarchy: OBJECT,  DATE, SCOPE-CAMERA,  TYPE   That will be a change as I currently have YEAR, MONTH, OBJECT, TYPE

Thanks

Mike

[melsmore]

I'm not an imager, so use a notebook. "Observatory" from Codeobsession for the Mac got a very favourable review in Sky and Telescope recently. 

https://codeobsession.com/observatory/

[david_taurus83]

I use APT for capture so I have created a shortcut on my desktop to go straight to this folder. APT automatically creates a folder with the date on when imaging. Once the session is complete I just add the object name to the folder. In the folder I create 4 more folders for raw data, lights, darks, flats, bias. These will be kept if I ever want to add more data in the future. When pre processing with Pixinsight, I still keep everything in the same master folder but I add temp folders for calibration, approved, registered etc. The last session though these temp folders generated around 50gig of data! Once I have a stacked image, I keep this and delete the temp folders as long as I'm happy with the stack. Works for me as I like things in chronological order.

[fireballxl5]

I just use Windows folders to organise my files.

Each imaging run is saved using the folder name format: yyyymmdd_Object_Scope_Camera

Within this folder, sub-folders Date(s) and Processed. Subs go into the Date folder when they were captured.

Cal files are stored in separate folders.

I then have a catalogue folder tree for various object types, e.g. Galaxies where I put copies of my 'keepers' - I do have a few that I like 

Always on the lookout for a better way of managing my data...

CS, Andy

[spillage]

+1 for APT to name files for you. If you do not have to remove your camera from the scope then I would list your target file and sub files per exposure time.

M51

60 180 300 600

then break down each of these to individual light and darks, or filters if you have a mono. You can have a separate file for bias and flats as I do not think you need these to match the light temps.

Light Dark

I also find that moving your images to the final folder asap helps.

Hope this helps and makes sense.

 

Mark.

[carastro]

I keep them all on my main computer where I do the processing, and also on my back up drive.

I have one folder for all the RAW/Fits images and then a separate folder for each target within.  I tend to record the scope and camera used within the capture file name which my capture software allows me to do, albeit an abbrevation, so I know for future reference.  

I then have a separate folder for stacked and processed images (a folder for each target and date, sometimes location too.  

Finally I keep a "final image" folder which I save as a PNG file for uploading to Forums.  

Once in a while I go through the RAW/Fits folders and delete any i think I am unlikely to re-use.

Carole 

[geoflewis]

On 15/05/2018 at 08:24, mikeyj1 said:

What's your suggestion for how to catalogue and store the files (eg by date, NGC number, camera type etc..), and should I only keep Lights, or also master Darks and master Bias?

A great question Mike and one with which I continue to struggle. I've used a range of scopes, cameras and capture software over the past few years and my file storage methodology has developed over time. Currently I use a mono QSI583 camera + filters with SGP for capture control. My current convention is:

1. Firstly I have a dedicated folder tree for all my calibration files, i.e. darks, flats, flat darks and bias (though currently I'm not using bias frames). That folder tree comprises separate folders for each camera, Nikon DSLR, Canon DSLR (which I no longer own) and QSI583 CCD. For the QSI only there are seperate folders for Bin1 and Bin2 and temperature. Currently I capture all QSI data at -10C, but if I ever change that then I will also introduce temperature related folders for each Binning. Under each of those folders I have sub folders for each calibration type (darks, flats, etc) and below that are sub folders by date where I save individual files together with the master calibration file once that has been created. In addition, for the QSI camera there are folders by filter type (LRGB, etc) for Flats and Flat Darks, so that I can accurately calibrate the corresponding filtered light frames. I have attached below a screen shot of my calibration frames folder tree...
2. Secondly I have a separate folder trees for each of the capture tools that I have used, those being ImagesPlus, MaximDL and currently SGP. Since I'm now using SGP I'll explain what I do with that. I configure SGP to save to specified folders by setting the folder and file name structure using the provide directory and file naming tools. This incorpates the target name and I save individual light frames under separate folders organised by target name\filter. I find this works well for the QSI camera as then all files for each target+filter are filed together. See below screenshot...
3. NB When I process the light frames I save those under a sub folder for each target+filter, then when I create the LRGB, RGB, HaRGB, etc., final images, I save those under an appropriately named folder.
4. My file names include reference to Target/Date/Exp/Bin/Filter/Scope/etc., as appropriate, to help me remember what I've actual taken and also in case I accidently move them somewhere else.

As you say it's a messy process that can very quickly get out of control; mine is still a work in progress so may well change over time if I find a better method. I hope this helps (though maybe I have just created more confusion), but I'm also interested to see other folks file and folder naming conventions.

Cheers, Geof

PS The above is for my DSO images, but I also have a separate folder structure for saving and processing video files for planetary, lunar and solar images - phew, my brain hurts....!!

[Joel Shepherd]

For images (not calibration frames), I organize by target (e.g. M42, NGC5676, etc.), then capture date. All lights and flats from one night go in one folder. If I image a target over multiple nights, I add a "+" to the end of the first date directory. E.g.:
   M42/2017-12-04+/...
   M42/2017-12-05/...

When I'm ready to process, in the first date folder I make a "processing" sub-folder, and all of my intermediate and fully processed images go there.

Bias and dark frames are kept in a separate directory tree, organized by type, duration (for darks) and temperature.

Nothing too fancy and I rarely get lost. :-)

[mikeyj1]

Thanks for some excellent feedback all!  So tonight I will set out the new tree structure in Win explorer.  I use APT so the extra naming parts so i already can get that to create both a folder structure and naming convention (very useful feature, although it starts with the camera number first).  My processing is done on a separate machine, so i need to transfer the files and folders from the laptop to the main comp, (gives me a backup) and on the main computer i will use the full OBJECT,DATE+,TYPE.  I like the idea of a separate tree for Bias and Darks, so will include that also.  At the top level ASTROPHOTOGRAPHY i will create a folder called 'Processed images' which is an export folder from Lightroom for JPEGS

thanks again, very useful, but if anyone thinks of anything else, drop it in, it will be appreciated

Mike