New file format for Pixinsight

[RikM]

I am currently about 2weeks into the 45 day trial for Pixinsight and fully intended to buy it at the end of the trial. I got a mailshot today that they are introducing a new standard file format in place of .fits. I use a bunch of other software in my workflow as well as PI and all of that works on .fits so if PI is moving away from .fits it will be pretty inconvenient. That makes me wonder if it will be worth buying at all? Anyone else worried about this or am I mistaken?

[StuartJPP]

I would assume that they will still support .FITS files but like Photoshop saves .PSD files with layer information and history and a bunch of other bits I am sure that the Pixinsight team want to save more information than can be stored in a .FITS file. There will probably be an option to "export to" or "save as" FITS file which will "flatten" the image and/or remove history. Who knows, this may be the precursor to decent layer support in PI?

It would be handy to save the entire workflow instead of just the final result.

Though I could be totally wrong...

[Earl]

I think its great if it does lead to what stuart mentions.

I need to update mine soon

[JamesF]

FITS is a very generic container format though.  It's not specifically just for images, but allows all sorts of other metadata to be stored too.  I wonder what they perceive to be the need that FITS fails to meet?

James

[RikM]

Reading through the info on the PI forum it seems that the new file type will be the native file type for PI in the future and is needed to get around the limitations of FITS. All compatibilities will be maintained and FITS header info is stored in the new XISF files so export to FITS will be seamless. I guess this is a good move if it opens up processing opportunities or makes processes more efficient. As Stuart says, it is similar to PS .PSD files. I don't think they will implement layers though. They want to differentiate PI from PS and that is a crucial difference.

[Gina]

I used the trial version of PI for a while and it looked like it could do wonders but I couldn't get my head round it before the trial period ran out.  Think I'll stick with what I've got   Have to admit that Photoshop has been quite a steep learning curve though.

[RikM]

There are a couple of tools that I love. TGVDenoise is great and the linear fit tool does a good job of balancing the channels for me in NB. I like using the curves tool for saturation as well. I have been completely unable to get PI to properly calibrate and stack my images though. I can't make head nor tail of the setting tweaks I would need to make to get it to work. I'm simply not going to bother. The combination of Maxim and Registar just works effortlessly for me, so I think I will stick with that. The STF does a pretty good job of setting and initial stretch as well. On a good image, I think I would still stick with levels and curves in PS but for my short data set NB images the STF does okay.

I'm using PI like a set of 'actions' that happen to run in a different application. It isn't going to replace PS in my workflow anytime soon, but is a great complimentary product. I intend to purchase the full version at the end of the trial.

[Sp@ce_d]

They probably could have worded it slightly better. My first impression too on reading the email was that .fits was going away. But of course that would be self defeating.

I probably would/have spend/spent 80% of my processing time in PI... if I could get the data in the first place!! This year has not been very productive so far..

[NickK]

I saw this then thought - It would be difficult not to have FITS compatibility given it's an industry standard over many systems (spectrometers etc etc).

FITS is a bit old in the legacy stakes so I assume that their internal/native format will move to XISF. It being extendable is their key requirement.

[Earl]

Just in.

esterday we announced XISF, a new file format that we are developing for storage and management of image data on the PixInsight platform. There has been some confusion and concern about this announcement, which we want to address by clarifying the following points:

* The FITS format will always be available in PixInsight.

* We will continue implementing and supporting future versions and revisions of the FITS standard.

* XISF is fully compatible with FITS by design. Existing FITS header keywords are always stored in XISF files, and all FITS keywords are retrieved automatically and transparently when a XISF file is loaded in PixInsight. This means that no data can be lost as a result of FITS to/from XISF conversions. Thanks to this compatible design, XISF integrates seamlessly with existing data sets and tools as an effective replacement for the FITS format.

* XISF is from now on the native file format of PixInsight, and as such will be used for image output by default. However, other formats that are and will be supported, such as FITS, TIFF, JPEG or PNG for example, can be selected as default output formats by simply editing a few preferences options.

* The XISF project has been created to improve the performance and versatility of the PixInsight platform, and hopefully also of other imaging applications. It overcomes a number of important limitations of other formats and provides a powerful, robust and flexible tool for storage and management of image data.

* XISF is a free format. The entire XISF format specification will always be publicly available, and XISF will never be subject to any patents, royalties or restrictions, so that anyone will be able to use it at no monetary cost for any purpose.

Thank you for your attention.

The PixInsight Team at Pleiades Astrophoto

[Earl]

Having PI use its own files leaves the FIT file association free to another application, that could be handy.

[lensman57]

I am currently about 2weeks into the 45 day trial for Pixinsight and fully intended to buy it at the end of the trial. I got a mailshot today that they are introducing a new standard file format in place of .fits. I use a bunch of other software in my workflow as well as PI and all of that works on .fits so if PI is moving away from .fits it will be pretty inconvenient. That makes me wonder if it will be worth buying at all? Anyone else worried about this or am I mistaken?

They have subsequently sent another email that Fits would still be supported . The problem with PI is its vast array of tools and the enormous number of variables that can be adjusted. It takes a life time to get to learn them all.

A.G

[JamesF]

It says in one of their posts about the new format that they're contacting the producers of imaging applications to try to get them to support the format (though they don't actually even have a proper spec. for it yet).  But if it's just so PI can store it's own data, I can't see why anyone would be bothered.  It doesn't appear to buy the imaging application anything particularly obvious.

If there genuinely are problems with FITS, which from my limited knowledge of the format I'm really not sure about as I know it can pretty much be used to store any type of data, then I'd have expected other people to have come across them too and to be interested in jointly agreeing a new format that addresses everyone's concerns and to bring the benefit of plenty of people who can subject the proposed new standard to "peer review".

Seems an odd way to go about trying to create a new standard to me.

James

[NickK]

It says in one of their posts about the new format that they're contacting the producers of imaging applications to try to get them to support the format (though they don't actually even have a proper spec. for it yet).  But if it's just so PI can store it's own data, I can't see why anyone would be bothered.  It doesn't appear to buy the imaging application anything particularly obvious.

If there genuinely are problems with FITS, which from my limited knowledge of the format I'm really not sure about as I know it can pretty much be used to store any type of data, then I'd have expected other people to have come across them too and to be interested in jointly agreeing a new format that addresses everyone's concerns and to bring the benefit of plenty of people who can subject the proposed new standard to "peer review".

Seems an odd way to go about trying to create a new standard to me.

I used to work in a company that originally created what became of the defacto industry protocol in the mobile industry (I'm talking about SMPP for SMS messages).. it's all about the extensions - the unstandard parts of the standard that are used to mess up and make other systems appear to have issues because they can't render the file correctly.

Basically it works like this:

1. Create 'standard', release it to everyone so its 'free'. Even giving away a free open source library to encode/decode the file format..

2. People add it as yet-another-file-format.

3. Vendor then adds extensions that are proprietary.

4. Other vendors then can't render the file in the right way.. either completely fail or not as the user expects - resulting in a 'this other vendor's app is Rubbish..'

5. Users then end up needing to stay with standard vendor to keep all the data in the correct format due to extensions. This is the lock-in.

6. Other companies add their own extensions.. create their own formats as 'standard' .. that are incompatible..

Not that I'm saying a new standard is bad.. but attempting to get all the vendors to support it a possibly a leader in the field is indicative of the above behaviour.

[IanL]

The original announcement was pretty badly worded (have come to expect this to be honest). The follow up makes is clear that FITS will be supported as it is now, and any updates to the standard will too. Makes sense as it would limit the market severely otherwise. Whilst it is planned that the new format will be the default, it will be possible to change preferences so that FITS is the default format if you so wish. I.e. you can carry on exactly as you do now if that is what you want or if you see no benefit in the new format.

I don't know what all the implications are, but the format will support remotely hosted images and local ones embedded in the same "file" as far as I can see. This would make for some interesting opportunities like easier collaboration or combining data from your own and professional sources for example. Combine this with recent changes to add "seals" (i.e. digital signatures) to images and projects and it seems to me a set of moves to enable the product for the collaborative / cloud-enabled age we are now living in.

[JamesF]

Not that I'm saying a new standard is bad.. but attempting to get all the vendors to support it a possibly a leader in the field is indicative of the above behaviour.

Quite.  I'd not want to accuse the PI guys of such behaviour, but when you've been around the block a few times in the software business it's hard not to get that impression.

James

[NickK]

The original announcement was pretty badly worded (have come to expect this to be honest). The follow up makes is clear that FITS will be supported as it is now, and any updates to the standard will too. Makes sense as it would limit the market severely otherwise. Whilst it is planned that the new format will be the default, it will be possible to change preferences so that FITS is the default format if you so wish. I.e. you can carry on exactly as you do now if that is what you want or if you see no benefit in the new format.

I don't know what all the implications are, but the format will support remotely hosted images and local ones embedded in the same "file" as far as I can see. This would make for some interesting opportunities like easier collaboration or combining data from your own and professional sources for example. Combine this with recent changes to add "seals" (i.e. digital signatures) to images and projects and it seems to me a set of moves to enable the product for the collaborative / cloud-enabled age we are now living in.

A file is nothing more than syntax, a container to store data. Rendering data as an image requires processing - all applications currently know how to render a FITS file into an image - including the defined extensions (as it's been around so long). Rendering any file format with proprietary extensions means missing understanding of how to render/recover the data. A proprietary vendor would probably point to using their free-to-use code library but that's nothing more than forcing people to use or applications that are sold have to pay a licence..

Any certificate attached to a file that can be rendered into a separate unprotected image is effectively useless. Protective rendering such as HDMI had to be deployed for blue-ray.. bypassed as history has shown.

Codecs are another issue - patented.. licenced.. etc.. WebRTC for example is a very good example of Apple, Google and Microsoft battling with using patented Codecs to prevent users moving their information without paying for it. Google's own codec is given away to ensure uptake and use of Google based technology..

Quite.  I'd not want to accuse the PI guys of such behaviour, but when you've been around the block a few times in the software business it's hard not to get that impression.

James

Indeed Many shades between black and white.. 

In the end people using PI will use it regardless.. Nothing stopping an XSIF to FITS render batch process.. but it makes it a little more difficult to migrate data stored to a competitor's application even in the event of attempting to prevent churn if they wish to charge for updates. Probably a mountain out of a mole hill.. but the fact that they want everyone to use it has parallels in so many industries with standards..

Next PI will be offering cloud based services to store and collaborate.. but only if you're using/subscribed to PI..  the cloud is just as much a standards battle ground.. 

Sometimes analysis is a bad thing

[IanL]

Nick, with regard to seals/certificates they're not a form of encryption to prevent copying, but a guarantee of authenticity. I.e. Makes it possible to verify that a file has been produced by a given person or organisation and not tampered with. Handy for collaboration I suppose, and should enable one to avoid problems with broken files. I can't really see why it has been included beyond that though since neither the images or PI projects include scripts or code that one would need to verify trust for.

Certifying PI scripts would make more sense since it is easy enough to write one that can alter or damage files on the computer. I haven't looked in to it in detail but given the script engine makes it easy enough to create files and write data in to them I guess one could do some pretty nasty stuff. Unfortunately they don't apply certs to scripts and doing so would hamper people contributing them as we aren't going to fork out money to get a certificate from a CA so they'd be self signed which is as good as useless.

I don't see the logic of arguing that vendors should not create new file formats, otherwise how would any innovation happen? I was perfectly happy downloading .text files from FTP servers back in the day, but look where HTML and everything that followed has got us in the past 20 years. The first versions were open formats, just like XISF is. Standards came in later versions. Yes people do bad things in relation to standards but I doubt we'll see much of that here due to the lack of enough money to pay for the lawyers. Sure PI is a commercial product but the developers have actually been very open in terms of the product and extending it if one so wishes. I think comments to the effect that they are trying to create a proprietary lock-in are very wide of the mark.