Deciphering computer files

[Demonperformer]

For reasons that are far too complicated to go into in detail here (but basically vizier does not allow me to select the criteria I want), I am looking to download the UCAC4 database. There is an extract that can be downloaded from CdC, but that does not contain all the fields I require.

Obviously this is quite big, so I downloaded zone 900 (nice and small!) to make sure I can translate the data into something usable. Now, the "read me" file says that each record is 294 bytes long. And the z900 file is 13338 bytes long, which means that it contains 45.367 records ... an unlikely figure, methinks!

So, a quick bit of investigation reveals that zone 900 actually contains 171 stars, and 13338/171 is 78 exactly. Well, this could be a coincidence, so I downloaded zone 899. This is 39936 bytes long, which, divided by 78 is exactly 512. And, sure enough, there are 512 stars in zone 899. That is just too much of a coincidence to be a coincidence in my book.

So I have files that contain the right number of stars, but each record is only 78 bytes instead of the expected 294 bytes. This leaves one (fairly important) piece of the puzzle: translating 78 bytes into 294. Unfortunately, I have no idea how to go about this. I realise the files I am downloading are binary files, which means that numbers can be squeezed a bit, but how would I know where one number ends and the next one begins? I use Access as my database program and can see no way of directly importing binary files into it.

So I am turning to the community to see if any of the software experts on here can point me in the right direction.

The end result I am hoping to achieve is to be able to select the stars in a particular zone that have a tycho2 number, and then retrieving selected items of data for each of those stars.

Thanks.

[r3i]

Hi DP,

Just taken a quick look at this.  Where did you see the 294 byte record size?  In the readme that I found here: http://ad.usno.navy.mil/ucac/readme_u4v5  , section 5a appears to indicate the records are 78 bytes long as you suspect.  There's even a bit of example FORTRAN code to demonstrate how to access the records.  Wow, FORTRAN, that takes me back a few decades

HTH.

Cheers,

Mike

Edit: Section 3i defines the record size to be 78 bytes

[Demonperformer]

Thanks. That is weird. I got the file-length from the read me file on vizier. Going to have to investigate that further. 

[JamesF]

There's a useful description of the record format in section 5a of the doc that Mike linked to.  I'm not sure how you can read that into Access directly though.  Having something else turn it into a csv file first might help.

James

[Demonperformer]

Quick look through and it seems I'm not going to be able to do what I was hoping anyway - it does not contain the tyc2 number, just a flag to say if it has a record in that catalogue or not.

Oh well, it was worth investigating.

[stash_old]

They do provide C routines to read the data as well as Fortran and I would bet that something like Python has these routines as well - if not a "wrapper" using SWIG could be used.

quote 

These algorithms were also translated into C thanks to Bill Gray (Project
Pluto, Guide software, www.projectpluto.com/ucac4.htm), who provided the
following files:

makefile,  u4test.c,  ucac4.c,  ucac4.h

This C code has the same functionality as the Fortran code explained above.

 

Here is a bit of code to get you started LOL  https://stackoverflow.com/questions/35282014/convert-ascii-file-to-netcdf-using-python-selective-columns-from-ascii-file

 

[han59]

I have described the UCAC4 header record and decoding at:

https://sourceforge.net/p/hnsky/code/ci/default/tree/hns_Usno.pas

for program:

https://sourceforge.net/projects/hnsky/

It is Object pascal, (lazarus/FPC), record length 78 bytes.

Han

 

 

 

[Demonperformer]

I have had the chance to go through the documentation link provided by Mike above. The tyc2 number is included in a separate (ASCII) file (u4xtyco), so I am going to be able to run with this after all (Access is very good at importing ascii files). And I now have three options to decode the basic info - fortran, c & pascal - I'm sure I should be able to get one of them running!

Still not sure why the info provided in the readme file on vizier says different, but I guess that has become a little academic.

Thanks for the assists.