Fuji X-T1 Digital Camera?

[The Admiral]

This is all very strange! I converted the RAFs to dng and imported into v3.3.4. Again, all the images were blank (i.e. just black), no stars were visible. Processing them, DSS couldn't find any stars. And yet, if I look at the dng converted files in another browser, I can make out the stars. So what's going on here?

Ian

[happy-kat]

Folloowing posts read on here I use the autosave file that DSS saves and post process that. All is then fine on what it contains.

[buzz]

Ian - did you try PI?    I put a bunch of RAW files into the ImageIntegration tool. It used DCRAW conversion and produced a sensible output as it would with FITS. I didn't see anything unusual in the output.

[The Admiral]

No, buzz, I haven't tried PixInsight.

I must be doing something wrong with DSS. I'm very much a newbie to this. The story is:

1. Convert RAFs to tiffs and import into DSS. If I click on one of the files, the image is there, although it helps if I use the slider at the top right to brighten the image.
2. Import RAFs directly into DSS (v3.3.4). If I click on one of the files, all I get is a black rectangle.
3. Convert RAFs to DNGs and import DNGs into DSS (v3.3.4). Again, if I click on one of the files all I get is a black rectangle.

I even tried this with a conventional photo image and irrespective of whether I import the RAF or DNG, I get a black rectangle. I must be doing something wrong. Looking at the RAW/FITS settings, I don't see what can be changed (none of the Bayer Matrix transformations options works. It isn't Bayer, after all). Even looking at the FITS settings pane, none of the generic matrix arrays match the Fuji X-Trans array.

I'm interested to see that digitalcyanide found that the DNG conversion did work, contrary to what I'm finding. Can anyone who's got DSS to work with RAFs let me know what setting were used please?

Ian

[buzz]

Ian, I think I know what you are experiencing. The RAW file, when it is converted, has no gamma correction. A mid grey on the fuji was 13,000 in 16-bit greyscale and the Canon 60Da, surprisingly, was less at 5000.

My first bench comparison is interesting. I averaged 50 biases: The Fuji has a mean bias of 13.7 and StdDev 2.84. The EOS has 27 and 8.3.  A calibrated 10-min dark frame, at 20C, for the Fuji had mean 39, StdDev 170 and the EOS, 103 and 235. RAW files are never truly RAW and I suspect the Fuji is doing some kind of dark frame subtraction inside but it does appear that the Fuji is quieter for read noise and thermal noise from the StdDev figures.

I want to test the color sensitivity next- My idea is to take an image of my QSI 8-filter wheel against the sky and measure the relative light levels. I have 5 nm SII and Ha in the mix.

[dyfiastro]

I dont know if it makes any difference here but I converted my RAF files using Lightroom.
I already had the images loaded into a lightroom catalog so just exported them as DNG's.
Now I would not thought that this would have made a difference but it may be doing something under the hood.

[The Admiral]

Ian, I think I know what you are experiencing. The RAW file, when it is converted, has no gamma correction. A mid grey on the fuji was 13,000 in 16-bit greyscale and the Canon 60Da, surprisingly, was less at 5000.

My first bench comparison is interesting. I averaged 50 biases: The Fuji has a mean bias of 13.7 and StdDev 2.84. The EOS has 27 and 8.3.  A calibrated 10-min dark frame, at 20C, for the Fuji had mean 39, StdDev 170 and the EOS, 103 and 235. RAW files are never truly RAW and I suspect the Fuji is doing some kind of dark frame subtraction inside but it does appear that the Fuji is quieter for read noise and thermal noise from the StdDev figures.

I want to test the color sensitivity next- My idea is to take an image of my QSI 8-filter wheel against the sky and measure the relative light levels. I have 5 nm SII and Ha in the mix.

Buzz, I'm not sure I really understand what you are saying, but I await your findings. But can I ask, would that mean that the imported images should be totally black, or just a lot darker than you'd expect? As far as I can see, there is no image on them viewed in DSS. But also, when the dngs are viewed in Lightroom, there is an image. I guess the DSS processing is different to LR.

I dont know if it makes any difference here but I converted my RAF files using Lightroom.

I already had the images loaded into a lightroom catalog so just exported them as DNG's.

Now I would not thought that this would have made a difference but it may be doing something under the hood.

Unfortunately my version of LR won't convert RAFs (I switched to Capture 1 when I got my Fujis), otherwise I would have tried that route. It is odd though, because you'd have thought Adobe DNG converter would be identical to the one in LR.

[buzz]

Buzz, I'm not sure I really understand what you are saying, but I await your findings. But can I ask, would that mean that the imported images should be totally black, or just a lot darker than you'd expect? As far as I can see, there is no image on them viewed in DSS. But also, when the dngs are viewed in Lightroom, there is an image. I guess the DSS processing is different to LR.

Unfortunately my version of LR won't convert RAFs (I switched to Capture 1 when I got my Fujis), otherwise I would have tried that route. It is odd though, because you'd have thought Adobe DNG converter would be identical to the one in LR.

Ian - if you bung a RAW file into a dropbox - I can convert it with PI and you can compare results with DSS?  Sensors are linear, but exposure is inherently logarithmic. When PS displays images, it applies a gamma correction to give something that the brain can recognize . I can also compare the result from say PI converting and Nebulosity acquiring an image- it should be the same outcome.

Which LR are  you using? Im using CS6.  DNG is free and you can download the latest.

[dyfiastro]

I am using Lightroom CC and used the Export to DNG function.

[buzz]

I did some work with Ian on one of his FITS files and remembered what I had forgotten from my old photography days.

RAW files are linear - i.e. they assume a gamma of 1.0.  Each 'stop' has half or twice the tonal resolution of its neighbor, just as it comes off the sensor's A/D converter.  Photographic editors (and jpeg conversion) apply a gamma adjustment, normally to 2.2, so they look 'right' on screen. Classical photography (what I call real photography) materials are inherently logarithmic and do the conversion naturally.

To prove to myself, I took an exposure from the Fuji and Canon of a blank white wall. The jpegs were a nice mid-grey.  I took the RAW files into Pixinsight and output as a TIFF.  Reading the TIFFs into Photoshop produces a very dark grey rendering.

I then used levels and moved the middle slider to 2.2 (introducing a gamma correction) and lo and behold, my perfect mid-grey returned.

regards

Chris Woodhouse

[The Admiral]

Buzz, I'm sure that you are correct, but I have my doubts that this is the real issue. Here is a random jpeg taken with a Lumix GH2.

Now, here is a screen grab of DSS with the associated RAW (RW2 file) loaded and pre-viewed at default settings:

I think this clearly shows the difference in gamma.

Now here is the same with an X-T1 jpeg:

Now with the RAF imported into DSS:

There is such a difference in the way these images have processed that I can't help think that there is more than just gamma at play here. Even with the brightness slider at the top right being pushed to the left, the image remains black. But being a novice with DSS, I could easily have got a setting or two wrong, so I cannot be sure it is wholly down to the way DSS behaves with Fuji RAWs.

Ian

[buzz]

Ian, I agree - there is something going on here with DSS. 

For comparison - from RAW - a Jpeg output from Aperture, the JPEG output from PixInsight from the RAW file and finally, PS opening the Pixinsight JPG and doing a 2.2 gamma correction.  It is not quite perfect, as I needed to change my color settings in PI, but it is the overall tonality which I'm trying to show here.

[buzz]

Ian - I just noticed something - on your two screen grabs - the icon next to the GH2 image has a little bayer array - but the one on the XT1 does not - suggesting that DSS does not believe it is a color image. Could that be it?

Funny - I had a GH2 too before getting the Fuji... 

[The Admiral]

Hi Chris (buzz). Don't forget that the GH2 uses a Bayer array, whereas the Fuji X series use a non-Bayer array which they call X-trans (see https://www.fujifilm.eu/uk/products/digital-cameras/interchangeable-lens-cameras/model/x-pro1/features/fujifilm-x-trans-sensor-technology/). If DCRAW claims it supports the X-T1 then it should be able to accommodate that. But just because DSS uses DCRAW, does that mean DSS can accommodate such an array? I don't know what it does under the hood.

I think you said that PI also uses DCRAW. Presumably, it can accommodate the non-Bayer array, as the Fuji RAFs work. A bit of a mystery.

I still use my GH2 occasionally, it has its merits.

Ian

[dyfiastro]

I may just have the solution.

After reading all the posts again I did a little bit of digging this evening.
I knew that the original lightroom exports that I did worked but like everyone else I could not understand why the dng converter was not working for everyone.

I download the dng converter and converted one of the files that I had used before but this time using the dng converter instead of lightroom.

I loaded this into DSS and noticed within an instant that I was having the the same issue as everyone else, no stars and a plain black screen after registering the files.

Just to be on the safe side I re-converted the same file again using lightroom and once again loaded it into DSS, Stars and a correct image as before.

I went back into lightroom and double checked the DNG export settings against the default ones used by the converter, There was one MAJOR difference.
My lightroom export defaults to "Camera RAW 5.4 or later" the DNG converter defaults to "Camera RAW 7.1 and Later".
I changed my lightroom file to the same as the DNG defaults and once again loaded it into DSS, This time I got no stars and the same blank black image as everyone is getting with DNG converter.
Then I changed both my lightroom settings and the adobe dng converter settings to use the "Camera RAW 6.6 and Later". When loaded into DSS these both showed stars and usable images.
There seems to be some kind incompatibility going on.

SO...

In adobe dng converter click on preferences and change the compatibility setting to "Camera RAW 6.6 and Later" or "Camera RAW 5.4 and Later" this should give a usable conversion.

Let me know if this works for everyone else.

[dyfiastro]

As a side note it would be interesting to take this one step further to see if there is any kind of a difference in the data being extracted using both of these conversions. It may be that one gives a better conversion than the other.
I still have not purchased my scope yet so do not have enough of the correct type of images to be able to look into this further, most of my images are very much wide field milky way shots and I was just luck to have a few shots from when I was testing a mates 35mm.

[The Admiral]

Good detective work there, digitalcyanide! Great spot.

Yes it does work for me . However, the dng files are somewhat darker (they are already dark) than when I used TIFFs that I had created in Capture 1. In the light of what buzz was saying, that is not surprising as the TIFFs would have had an associated gamma of 2.2, whereas for the DNGs it will have been 1. Originally,DSS would only see about 4 stars, but I got around that by increasing the RAW colour adjustment setting from 1 to 5 for each colour. I was then able to get 40 stars at a threshold of 20%. Pity it is necessary to have to convert the files to DNG rather than import directly, though.

It is also slightly puzzling that Lightroom CC hasn't used the latest DNG conversion available (I thought that being always up-to-date was one of the selling points of Adobes CC strategy. Mind you, if it did then it wouldn't have worked in this situation - perhaps that's a potential down side of the CC creation!). And in any case, I wasn't aware that Adobe had changed its DNG specification, which is what this behaviour would suggest, I would contend.

[dyfiastro]

Good detective work there, digitalcyanide! Great spot.

Yes it does work for me . However, the dng files are somewhat darker (they are already dark) than when I used TIFFs that I had created in Capture 1. In the light of what buzz was saying, that is not surprising as the TIFFs would have had an associated gamma of 2.2, whereas for the DNGs it will have been 1. Originally,DSS would only see about 4 stars, but I got around that by increasing the RAW colour adjustment setting from 1 to 5 for each colour. I was then able to get 40 stars at a threshold of 20%. Pity it is necessary to have to convert the files to DNG rather than import directly, though.

It is also slightly puzzling that Lightroom CC hasn't used the latest DNG conversion available (I thought that being always up-to-date was one of the selling points of Adobes CC strategy. Mind you, if it did then it wouldn't have worked in this situation - perhaps that's a potential down side of the CC creation!). And in any case, I wasn't aware that Adobe had changed its DNG specification, which is what this behaviour would suggest, I would contend.

Just glad it worked.

CC by default my in fact use the lastest. However, I have used lightroom for the last few years and when I upgrade I tend to just copy folders that have my custom settings and this does tend to replace some of the stock setting as well. this may be the reason why it was not set in lightroom.