Pixinsight WBPP folder organisation

[malftobe]

I have a question about best practice for organising folders for pixinsight for pre processing in WBPP.

 

After a good few clear nights for the first time in forever and a few recent equipment upgrades, I am hoping I have some decent data to process! I am going to finally try the pixinsight trial so want to hit the ground running and have been searching the forums and watching the Adam Block videos on Youtube.

 

Hoping to use the directory button and use the keyword grouping for multiple nights as detailed by @teoria_del_big_bang below

 

I currently use Siril and use the folder structure

 

Target name > Date >Filter > light

                                                 flat

                                                 dark (including dark flats)

 

Would keeping the same structure be the best way using pixinights WBPP so I can use the add directory button and use data across multiple nights?

 

Adam Block seems to use a variety of differently organised data sets of his own and others in the tutorials, so I guess I may be over thinking this!!

 

 

 

Edited January 24, 2023 by malftobe
Edited title

[teoria_del_big_bang]

I can answer this better when I get home from work and look at my directory structure.
Basically I would say yes that level of organisation is fine but you need to add an underscore after the keyword you want to use as PI uses the underscore as a marker to look for the keywords.

Let me get back to you later and explain better.

Steve

[teoria_del_big_bang]

I think i am right in what I am saying here.

To  group images together so you can use WBPP to calibrate images over several nights and group them so images from one night use the flats from that night you need to use keywords.

Just to recap beow is basically what was in one of my replies to another thread (the one you refer to in the OP) there are some added comments to help.

I have data on NGC 7000 for 3 different nights.
I take flats after each session or night.
I make sure the flats go in the same directory as the lights for each night.
Now I used to just name the directories the with the date, such as 22-09-13  *** This I have had to change to use this grouping as I will demonstrate below ***.
I use the format  Year-Month-Day that way the directories are easy to ist in order as wimdows file manager looks at the year first then month and finally day, if use Day-Month-Year then they cannot be sorted  chronologically by the filename, only the date attributes but they can change when you transfer or change any data so not reliable. 

Indecently in the main directory of the target I also keep one of my images without a crop to use for framing up any future sessions if I return to that  target to add further data.
So in each of these directories I have lights and flats foe each night, with further sub directories for filters and exposure times.
You also have to add the darks and Bias if using one, I often like to keep a copy of the masters (not al the raw frames) in the same directory then they all load at once.

So when it comes to using WPBB I can just select the directory "NGC_7000_North_American.

Without us doing anything WBPP will group both flat and light frames by filter and exposure time as it takes these values from the Fits header

But it will not differentiate them by the dates to group into the 3 different sessions.
Now why WBPP is not clever enough to do that I am not sure as the date is also in the header, albeit a date and time. However this may not be too useful as often the date will change at midnight and usually a session spans over two days anyway.
Although in NINA you can select an option not to change date at midnight and so that  would actually work if WPBB was changed to group automatically by date but as it stands it would not do this, also I guess if done in future that is still no good for old data with dates already in the headers.

So we have Flats:-

And Lights:-

As you can see we have all the 300S Ha lights together for the 3 sessions.
But that's not we want because we want the flats from each night to be made into separate master flats and then the correct master used on the correct lights for that night.

So we need to use a keyword to further group these fats and lights.
To do this we use keywords and enter the keyword(s) into WPBB

So I use the keyword "DATE".
But as i have the directories named this does nothing because WBPP cannot find a keyword called DATE and so nothing changes it is still grouping all frames from the 3 nights together.

Sorry for all that waffling you probably already knew but to use keyword the keyword, in this case "DATE", must either be in the Fits header or in the filepath.
It will first look in the header and if it finds this keyword, but there isn't one, there are some date related keywords as shown below but not just "DATE", also these are no good as they also have the time in them.


After looking in the header it will look in the filepath, not just the filename but the whole path, in this instance the filepaths are like below :-

Z:\DATA_and_PROJECTS\LIGHTS\DSOs\QHY268M\NGC_7000_North_America\2022-09-13\LIGHT\HA\300.00\2022-09-13_23-17-18_HA_-10.00_300.00s_0000.xisf
Z:\DATA_and_PROJECTS\LIGHTS\DSOs\QHY268M\NGC_7000_North_America\2022-09-16\LIGHT\HA\300.00\2022-09-16_23-27-55_HA_-10.00_300.00s_0000.xisf
Z:\DATA_and_PROJECTS\LIGHTS\DSOs\QHY268M\NGC_7000_North_America\2022-09-19\LIGHT\HA\300.00\2022-09-20_02-21-26_HA_-10.00_300.00s_0012.xisf

Now although I do have all the frames grouped under three different directories that are dates as it stands WPBB does not know these are what we want to group them to.
So I  have to add the keyword I want to use "DATE" to those directories followed by an underscore. The underscore is very important.
So with renamed directories we get this :-

And now when we load the same "NGC_7000_North_America" directory it will group them as we have the keyword DATE enabled.

 

So that's how WPBB uses keywords.

Now what does that mean how you organise your directories.
Well mine are basically as shown in this post with the change of adding "DATE_"  + the actual date and then making sure all the flats and lights for that night are within that directory somewhere.

So I have the format:-

CCD ---- TARGET  ---- DATE_YEAR-MONTH-DAY ---- FRAME TYPE (Flat, Light) ---- FILTER ---- EXPOSURE ---- FRAME NAME

In NINA to do this automatically I have the following in options:-

\$$CAMERA$$\$$TARGETNAME$$\date_$$DATEMINUS12$$\$$IMAGETYPE$$\$$FILTER$$\$$EXPOSURETIME$$\$$DATETIME$$_$$FILTER$$_$$SENSORTEMP$$_$$EXPOSURETIME$$s_$$FRAMENR$$

So you can use any name you want as a keyword and it can be either in the actual filename, or in any of the directories as WBPP lools at the whole file path.

In actual fact I have only used the "DATE_" keyword and not found a need to use anything else.
Of course it doesn't have to be date you could call them "Session1", "Session2" etc.

I guess if you has some old image files without everything in the headers then grouping them under your own keywords could be a way round that too.

I hope that helps and sorry if a bit long winded, I do waffle on somewhat as I get older 🤣

Steve

Edited January 25, 2023 by teoria_del_big_bang

[malftobe]

Thank you so much for the thoroughly comprehensive explanation and presentation. I really do appreciate all the time, effort and generosity  that has gone into your answer to my question. 

[teoria_del_big_bang]

No problem, that's what SGL is about and believe me I have had more than my fair share of help from these good members 🙂 

Steve

[scotty38]

Just to add to the very extensive reply you've had already, if it's needed and helps you can have NINA create directories by using \\ in the file naming options.

[teoria_del_big_bang]

5 minutes ago, scotty38 said:

Just to add to the very extensive reply you've had already, if it's needed and helps you can have NINA create directories by using \\ in the file naming options.

I did make mention of this at the end of the post, but maybe not in such detail 🙂 
I think it is covered in detail in the excellent NINA documentation.

Steve

[scotty38]

4 minutes ago, teoria_del_big_bang said:

I did make mention of this at the end of the post, but maybe not in such detail 🙂 
I think it is covered in detail in the excellent NINA documentation.

Steve

Apologies I did not spot it 🙂

[malftobe]

My folders are looking a lot more organised now!

 

Are there any files / folders I can delete when the pre processing is finished that pixinsight creates during the WBPP pre processing ?

 

With a small hard drive on the new pc I need to be really well organised!  So I will be swapping the files to process on to the ssd of the laptop to process.

 

(Siril creates a huge process folder that can be deleted but that folder hates trying to be copied to an external hard drive as it creates tiny files and symbolic links)

 

Still not got around to downloading the trial and giving it a go. I thought this weekend would be a good opportunity before the six nations starts, but kids and family have other plans!!

Edited January 29, 2023 by malftobe
added info

[teoria_del_big_bang]

I guess you can delete anything other than the Registered directory.
So long as you do not delete your original images and flats you can always rerun WBPP if you were to lose something.

I would never delete any original images, including the flats and darks, although if you are perfectly happy the master dark and master flat are both good then you could just keep the masters.  
Problem is you can't really just take flats again and be 100% sure they are correct for the nights imaging if done months later, or even days later if anything in the image train has moved.

External hard drives  1Tb and bigger are not expensive now,  SSD can be but even 1TB SSD not too expensive, and I would recommend getting one and do not delete anything until you have have final images you are happy with. Even then it can be good to save what you can as months, years later you often want to reprocess as you either get better at it or new tools become available and it is very frustrating when calibration files are lost.

Always best to have any data you are processing on the main harddrive as accessing a slower external drive can slow things down a lot and get frustrating, but they are great for long term storage or even backing up data from other drives in case one goes down you do not want that to be the only source of your data.

Steve

[malftobe]

Thanks so much Steve for a comprehensive answer once more.

 

And an apology for my laziness in the definition of my questions! Thanks for filling in all the blanks(as always!!)
 

Hoping to keep all original data on an external hard drive, then copying the files/folders over to the laptop SSD when I process them. I will be retaining all original light frames and flats. 
 

Good to know I can delete a few files - the internal hard drive is only 512gb. I’m sure I’ll get a better feel for Pixinsight once I get up and Running and it probably deals with Pre process data a lot differently to Siril. The process folders in Siril fill up hard drive space pretty quickly! 

 

 

[teoria_del_big_bang]

You will get used to it once you give it a try.
I guess with the external drive I would be tempted to do something along the following lines:

Create a darks library for all exposure times you think you will use so that you end up with a library of dark masters. this way you then only have to have the masters on your main pc rather than all the dark frames.
Same goes for a bias if using one.
Update this every 12 months or so.
You can use WBPP to do this and run it just without any light images or flats.

Then each time you process a target:

Copy all the lights and flats from your session, or sessions, you want to process from external drive to PC if there is room on PC.
If not you can process directly from the external drive but it can take considerably longer, i would experiment and see if the time difference is extreme, if not too much more maybe makes sense to process directly from the external drive rather than having copies on main pc drive. 

Once WBPP has run its course if processing from copies on your PC move all unwanted files to the external hard drive just keeping the registered and masters directories on PC.

Carry on with the post processing saving the Pixinsight project ro PC  or the external hard drive, your choice depending on how the emory is going on the PC so you can retrieve your project later to carry on processing or if you have an unexpected shut down of your PC at any time.

 

One thing i have done on my laptop (which is quite old and not an all singing and dancing model) is that it has a CD / DVD drive in what is called a bay. I never really used it anymore as all software comes on line these days. You can but a bay that holds a 2nd hard drive and replace the cd drive with the 2nd hard drive. Cost me about £80 including the SSD hard drive and bay for an extra 1TB of fast SSD memory, so not sure if this is something you could consider.

 

Steve

Edited January 29, 2023 by teoria_del_big_bang

[malftobe]

Absolutely perfect. Thanks so much for the granular detail. It will really be a huge help. 
 

The new laptop has the hard drive and ram soldered in so no upgrades can be done. Can’t seem to get a decent sized hard drive in laptops these days. Quite a few with just 256gb, so considered myself lucky! The processor should be a big improvement over my previous. 
 

I miss my (still working well for many tasks) 2010 MacBook Pro. I did the dvd bay hard drive upgrade on that and maxed out the ram and other hard drive. Even replaced the battery. Can’t seem to upgrade much on most of the new models.