What data do you usually keep?

[dannybgoode]

One thing I've noticed from the few processing sessions I've done just the insane amount of data you build up.  Calibrated lights, aligned lights, CosCor lights etc.  So what do folk generally keep and what do they delete?  I know storage is very cheap these days but it seems a bit overkill to maintain a complete library of every step.

I was thinking of keeping the original subs and the aligned subs and binning everything else but is there a compelling reason to keep other pre-processed subs as well?

[ollypenrice]

I keep stacks. So that might be 'red, 12x15 mins' and 'Ha 10x30 mins' etc. I then keep the processing steps till space requires me to delete them. I'll generally keep the un-post-processed RGB but, ultimately, I'll end up binning the rest.

Olly

[Skipper Billy]

I use a CCD camera and relatively small numbers of long subs so I dont generate massive amounts of data.

I recently helped a pal out with a batch of CMOS data 90 each of lights in Lum, Red, Green and Blue plus a full set of darks, bias and flats. The processing folder reached over 300gb !!!!

Personally, at the end of the season I transfer that seasons data to an external drive - £30 for a terabyte !  No reason not to keep the data. The way this season is going I could fit everything on a floppy disc!

 

[dannybgoode]

1 minute ago, Skipper Billy said:

 

Personally, at the end of the season I transfer that seasons data to an external drive - £30 for a terabyte !  No reason not to keep the data. The way this season is going I could fit everything on a floppy disc!

 

Ain't that the truth?!  I am new to it all but this year hasn't gone well for learning - just too much cloud to get a decent few days run at something.

I am also trying to work out a set structure to my files so I can at least find stuff easily.  That's a mission in itself!

[Skipper Billy]

I sometimes forget how old I am and that many members on this site wont have a clue what a floppy disc is!!

https://www.silicon.co.uk/data-storage/storage/tales-tech-history-floppy-disk-209049

It holds a massive 1.2 meg of data - ie you would need 3 or 4 of them to store a single photo from a mobile phone!!

[gorann]

I do my initial calibration and stacking in PI before I move over to PS. I save the original subs but throw out the intermediate files created by PI (calibartion, debayering if OSC/DSLR and star alignment since these can always be recreated). Then I also save all the steps I have saved in PS since I often go back and reprocess.. And I save it all on two 4 Tb hard drives and both are now getting quite full, so I will soon have to move over to two new ones. I should probably check the status of the old ones at least yearly and if one of them appear to fail, which they will do one day, I should back the old data up again.

Edited January 26, 2020 by gorann

[discardedastro]

I keep everything off the camera as FITS, including calibration files, and normally keep the calibrated but not registered XISF files from PI, plus LRGB master files before any processing, cropping etc. That way I can easily re-create the stacks without doing recalibration, repeating cosmetic calibration, etc, or just combine the stacks from a few nights. If I need to go back to the source, it's all there.

Storage is cheap! Only a quarter terabyte so far on the raw datasets, covering a year or two, and I have about a terabyte for all the rest including some stuff like videos from eclipse events etc. I keep most stuff on a ~30TB usable ZFS array I use for all my data storage needs - it isn't fast enough for working storage (just SAS2, and networked at a gig, all spinning disks with NVMe L2ARC/logs) so I copy what I'm working on to a 2TB internal NVMe disk at the start of processing.

Edit: Oh, and for backups, I use Backblaze for everything stored on the PC which retains for >1 year, so that's something. I am planning to spin up some more storage soon and I'll do backups of the raw content to that regularly. Backblaze is great, but does require a fast upstream internet connection - I've got a gig a second up, so it's not a problem. The raws are stored on a ZFS array which can lose about 3 disks before I start worrying about data loss, so it's pretty resilient.

Edited January 26, 2020 by discardedastro

[carastro]

I must confess I keep everything.  Raw/Fits files, stacked files, and processed files and registered and processed files.  I always think I might go back and re-process at some time.   Like Skipper Billy I move a lot of it over to a back up drive when my hard drive gets a bit full.  

In some cases I do reprocess and have been much happier with a result perhaps a couple of years later. 

I ought to go back delete some of the intermediate processing and Raw/fits files though.

Carole 

Edited January 26, 2020 by carastro

[teoria_del_big_bang]

I too keep everything stored on a small server (so everything is backed up as it automatically goes onto two separate drives in the server) I bought off Ebay for about £80.

Issue I have is getting some sort of order to the directories so everything is in a logical order and easily found.

Steve 

[dannybgoode]

Just now, teoria_del_big_bang said:

I too keep everything stored on a small server (so everything is backed up as it automatically goes onto two separate drives in the server) I bought off Ebay for about £80.

Issue I have is getting some sort of order to the directories so everything is in a logical order and easily found.

Steve 

I am starting to develop a reasonable solution to file structure.  The subs for a session are stored in a file with is yyyymmdd - [target] and an L, R, G and B sub folders.  I then create further sub folders for each step in the calibration and integration process.  In here I will also store the master flats that I used to calibrate.

I then have further folders for Darks, Bias and Flats with sub folders for the subs but will generally clear out the subs once I have created a master of each.  I may keep the dark subs just becuase of the length of time it takes to create a good library of them.

I have an 8Tb NAS drive as well where I store my lights from each session as an archive but probably won't keep all the calibration subs etc as these can be reproduced pretty easily - particularly as I will easily be able to locate the master flats for any given target.

[teoria_del_big_bang]

1 minute ago, dannybgoode said:

I am starting to develop a reasonable solution to file structure.  The subs for a session are stored in a file with is yyyymmdd - [target] and an L, R, G and B sub folders.  I then create further sub folders for each step in the calibration and integration process.  In here I will also store the master flats that I used to calibrate.

I then have further folders for Darks, Bias and Flats with sub folders for the subs but will generally clear out the subs once I have created a master of each.  I may keep the dark subs just becuase of the length of time it takes to create a good library of them.

I have an 8Tb NAS drive as well where I store my lights from each session as an archive but probably won't keep all the calibration subs etc as these can be reproduced pretty easily - particularly as I will easily be able to locate the master flats for any given target.

Amazing, but that is not too dissimilar to how I have ended up. I just recently moved everything into similarly structured directories and really is almost same you you have done.

Steve

[dannybgoode]

2 minutes ago, teoria_del_big_bang said:

Amazing, but that is not too dissimilar to how I have ended up. I just recently moved everything into similarly structured directories and really is almost same you you have done.

Steve

I am lucky to have only just started out so I have a blank canvas to work from for such things.  Decided I may as well start off organised - don't know how long it will last though

[teoria_del_big_bang]

I don't know that looks like a good system and hopefully is as I am hoping my similar system will work so I would stick to it if you can otherwise I found it a nightmare if I changed how I stored all this data nearly every time I obtained new data. 

Steve

[TerryMcK]

I keep the lot. I do however ditch the ones that have airplane trails in or are clouded over. Storage is dirt cheap these days so I have live copies on a server. I also have backups on a nearline disk SAN and offline blueray discs.

I work in IT and this is second nature for me.

[dannybgoode]

1 minute ago, TerryMcK said:

I keep the lot. I do however ditch the ones that have airplane trails in or are clouded over. Storage is dirt cheap these days so I have live copies on a server. I also have backups on a nearline disk SAN and offline blueray discs.

I work in IT and this is second nature for me.

I absolutely get having a sensible back up routine in place and would not bin the original subs (except for those that are completely unusable) and yes - I have multiple copies of them.  I am just not sure on the value in keeping say the non-star aligned calibrated subs when you have the star aligned versions which are the same data just a stage on.  For example would you ever go back in and start re-processing old data from that step in the chain?

I suspect I would either start from scratch or from the point of integrating the fully calibrated and aligned subs and not at any point in between.

[discardedastro]

9 minutes ago, dannybgoode said:

I absolutely get having a sensible back up routine in place and would not bin the original subs (except for those that are completely unusable) and yes - I have multiple copies of them.  I am just not sure on the value in keeping say the non-star aligned calibrated subs when you have the star aligned versions which are the same data just a stage on.  For example would you ever go back in and start re-processing old data from that step in the chain?

I suspect I would either start from scratch or from the point of integrating the fully calibrated and aligned subs and not at any point in between.

For me it's the last complex step in the train where I've done nothing drastic to the image data. StarAlign is a 10-second job to repeat, so if I needed to, say, paint over some satellite trails or wanted to use the frames in a mosaic I've got everything "ready to go" and then re-stacking is trivial.

I also use the timestamp format, just ISO 8601: YYYY-MM-DD. If it's an event or not my usual imaging rig I'll append a name or similar. Then projects for specific targets get a YYYY-MM <target name> folder, etc.

[TerryMcK]

4 minutes ago, dannybgoode said:

I absolutely get having a sensible back up routine in place and would not bin the original subs (except for those that are completely unusable) and yes - I have multiple copies of them.  I am just not sure on the value in keeping say the non-star aligned calibrated subs when you have the star aligned versions which are the same data just a stage on.  For example would you ever go back in and start re-processing old data from that step in the chain?

I suspect I would either start from scratch or from the point of integrating the fully calibrated and aligned subs and not at any point in between.

I’ve got a very fast processing system so yes I see no reason to start from scratch again when adding new data to old for instance, it only takes a few minutes to re calibrate with a dual Xeon each with 16 cores and 64GB available. I realise that such a system isn’t in everyones reach though and what I do is not necessarily what others should do.

 

[wimvb]

5 hours ago, Skipper Billy said:

I sometimes forget how old I am and that many members on this site wont have a clue what a floppy disc is!!

https://www.silicon.co.uk/data-storage/storage/tales-tech-history-floppy-disk-209049

It holds a massive 1.2 meg of data - ie you would need 3 or 4 of them to store a single photo from a mobile phone!!

I must be older. To me, these are the only true floppy disks

https://www.renderhub.com/firdz3d/floppy-disk-5-25

They also came as 8 inch disks. I've also stored computer programs on audio tepes.

Anyway, back from memory lane, I keep the original raw files and discard all calibrated/registered files. I don't store intermediate images. PI will store a complete history in a project file.

[Robny]

On 26/01/2020 at 19:14, Skipper Billy said:

I sometimes forget how old I am and that many members on this site wont have a clue what a floppy disc is!!

https://www.silicon.co.uk/data-storage/storage/tales-tech-history-floppy-disk-209049

It holds a massive 1.2 meg of data - ie you would need 3 or 4 of them to store a single photo from a mobile phone!!

I'd forgotten how little they hold...

I remember having to store my course work on those for college.  Before that in my even younger days I remember playing computer games that came o  these:

https://www.computerhope.com/jargon/num/525inch.htm

Before that even I remember bek g sat in front of my Commodore 64 waiting 45 mins for a game to load in a trans watching the flashing screen praying that the TAPE wouldn't stop 3/4 through loading!!!

 

[Skipper Billy]

1 hour ago, Robny said:

I'd forgotten how little they hold...

I remember having to store my course work on those for college.  Before that in my even younger days I remember playing computer games that came o  these:

https://www.computerhope.com/jargon/num/525inch.htm

Before that even I remember bek g sat in front of my Commodore 64 waiting 45 mins for a game to load in a trans watching the flashing screen praying that the TAPE wouldn't stop 3/4 through loading!!!

Remember it well - I had a Vic20 which had a fraction of the power of the Commodore!