PixInsight processing times

[Nikodemuzz]

Hello all!

I am currently using PixInsight with the trial license. I haven't gotten very far yet when it comes to learning the workflow, but something that I immediately noticed was the absolute ages it takes to preprocess the files. Meaning calibration, integration etc. Particularly the debayering process, which to me seems like it shouldn't take long, consumes several minutes per subframe. Arriving from a selection of ~150 light frames and a master bias (no darks or flats) to an integrated image took me a full day yesterday, mostly just computing away. DSS processes the same stack of images in about an hour. If I want to drizzle, that's further several minutes per subframe in PixInsight.

Now, there might be a quality advantage for PixInsight when it comes to the stacked image, but I wonder if it is worth the time expenditure. I mean, I guess you can spend the summer computing the files from the past season, but still. 😃

Is this something you have also noticed, or is it because of my computer? Perhaps due to the files from the Fuji X-T3? If this is the way it is supposed to work, do any of you save time by doing the pre-processing in DSS and post-processing in PixInsight?

[Vox45]

Hello,

It is not my experience. I do all the processing (post and pre) in PI and debayering never took more than a couple of seconds per subframe. Can you tell us more about the computer you are using ? or provide the subframe to debayer so I can test how long it takes on my computer ?

[Nikodemuzz]

On 03/03/2020 at 10:00, Vox45 said:

Hello,

It is not my experience. I do all the processing (post and pre) in PI and debayering never took more than a couple of seconds per subframe. Can you tell us more about the computer you are using ? or provide the subframe to debayer so I can test how long it takes on my computer ?

Hi Vox45!

Your experience sounds like the way I figure it should be. My computer is a Windows 10 machine, i3 8100 processor, 16GB RAM. I have been running a bit low on hard drive space, but have made sure that there has been around 100GB of free disk space when I have been processing images.

I'm attaching one subframe here, so you can have a go as well. Thanks for helping!

DSCF2061.RAF

[Vox45]

It took 20 seconds on my machine from opening to complete debayer (20,247s to be exact)

That single frame is looking good by the way

================================================

Reading 1 image(s):
C:/Users/ydelisle/Downloads/DSCF2061.RAF
Reading metadata: done

Camera ........... Fujifilm X-T3
Timestamp ........ 2020-02-25T21:07:54Z
Exposure ......... 60s
ISO speed ........ 5000
Focal length ..... 21 mm
Aperture ......... f/1 = 21 mm
CFA pattern ...... X-Trans GGRGGBGGBGGRBRGRBGGGBGGRGGRGGBRBGBRG
Raw dimensions ... w=6384 h=4182
Image geometry ... x=0 y=6 w=6251 h=4174
Image rotation ... 90 deg

Raw decoding parameters:
Output mode ............... demosaic
Interpolation ............. X-Trans
Wavelet noise threshold ... 0
White balancing ........... camera
Black point correction .... enabled
Highlights clipping ....... enabled
Auto rotate ............... enabled
Output image .............. w=4174 h=6251 n=3 RGB

Reading RAW data: done
Scaling colors: done
Pre-interpolating: done
Converting to RGB: done

Debayer: Processing view: DSCF2061
CFA pattern (detected): GGRGGBGGBGGRBRGRBGGGBGGRGGRGGBRBGBRG
X-Trans demosaicing: done

Noise evaluation: done
Gaussian noise estimates:
s0 = 1.583e-03, n0 = 0.2847 (MRS)
s1 = 1.536e-03, n1 = 0.4716 (MRS)
s2 = 1.571e-03, n2 = 0.2527 (MRS)
20.247 s

 

I am a bit puzzled by the CFA pattern (detected): GGRGGBGGBGGRBRGRBGGGBGGRGGRGGBRBGBRG

That is quite a pattern !  not sure if this is normal

[Nikodemuzz]

Thanks for testing it out on your computer! It is good to know that the slowness is not related to the software itself. On the other hand, I don't have the faintest clue about how to track down and fix the problem. Must be something about the settings in Pixinsight, because other software seem to handle similar tasks with less difficulty.

The CFA pattern is pretty wild, isn't it? 😃 Certainly not normal to someone who is used to looking at the traditional Bayer array, but there is nothing wrong with it. It's just the way the pixels are in the X-Trans array.

 

[Nikodemuzz]

Thanks for the compliments on the subframe! It is a 60 second unguided sub with the WO SpaceCat. I shot the sequence accidentally at high ISO 5000, but it didn't seem to ruin the image too bad. 😃 

[Vox45]

32 minutes ago, Nikodemuzz said:

Thanks for testing it out on your computer! It is good to know that the slowness is not related to the software itself. On the other hand, I don't have the faintest clue about how to track down and fix the problem. Must be something about the settings in Pixinsight, because other software seem to handle similar tasks with less difficulty.

The CFA pattern is pretty wild, isn't it? 😃 Certainly not normal to someone who is used to looking at the traditional Bayer array, but there is nothing wrong with it. It's just the way the pixels are in the X-Trans array.

 

I did not know about the pattern in X-Trans array  thanks for the info.

"The Fujifilm X-Trans is a CMOS sensor developed by Fujifilm and used in its Fujifilm X-series cameras. Unlike most CMOS sensors featuring a conventional Bayer filter array, X-Trans sensors have a unique 6 by 6 pattern of photosites"

I would suggest that you try your luck on the PI forum, they might be able to help you pinpoint the issue.

Cheers!

[Starflyer]

Try running the PI benchmark from the scripts menu then compare it with benchmarks from other users on the site with similar systems.

I found it painfully slow and put together a decent processing machine last year.

[old_eyes]

I am using a CMOS camera and therefore have loads of subs. Pixinsight does take time, but does a lot of very clever work on your data, much more than tools like Deep Sky Stacker. One thing to note when debayering is that the slowest part fo the process is Noise Evaluation. If you are not going to use that feature or are only having a quick look see to check everything is OK, untick that option. It makes a huge difference.

If I have a lot of subs to process, I find the Batch Preprocessing script invaluable. Set it up and let it run. Plenty of time for a coffee, or in cases of very large numbers of DSLR size subs, time to cook and eat a meal.

[Nikodemuzz]

19 hours ago, Starflyer said:

Try running the PI benchmark from the scripts menu then compare it with benchmarks from other users on the site with similar systems.

I found it painfully slow and put together a decent processing machine last year.

Thank you for the suggestion! I got a score of 4005. My processor is not listed there, but I tried to compare it to other CPU's with similar performance level. It certainly isn't a high score, but doesn't seem abnormally low either for a system with 16GB of RAM. I get a chill up my spine just at the thought of having to start updating the computer to process the files... The expense never ends, does it! 😵 😅

[Nikodemuzz]

7 hours ago, old_eyes said:

I am using a CMOS camera and therefore have loads of subs. Pixinsight does take time, but does a lot of very clever work on your data, much more than tools like Deep Sky Stacker. One thing to note when debayering is that the slowest part fo the process is Noise Evaluation. If you are not going to use that feature or are only having a quick look see to check everything is OK, untick that option. It makes a huge difference.

If I have a lot of subs to process, I find the Batch Preprocessing script invaluable. Set it up and let it run. Plenty of time for a coffee, or in cases of very large numbers of DSLR size subs, time to cook and eat a meal.

I had a go at batch preprocessing the other day. Worked, but took a whole day. 😃 Looking at the bright side, having a software work hard for the entire day without crashing has to be considered a positive. I'll check the noise evaluation next time, although I believe I had it off. Thanks for the hint!

[kbrown]

Back in the days when I used my DSLR (Canon 6D) for AP, PI was rather slow to process the CR2 files. Things were a bit smoother when I first converted everything to fits or xisf format. By default you'll get those anyway as soon as you start outputting new files.

How much RAM do you have? I notice that was a big bottle neck with my old laptop...

[Jokehoba]

Is your hard disk a spinning disk (if so, do you know the rpm?) or an SSD or both?

There are PixInsight preferences (Edit > Global Preferences > Directories and Network) which set the location for temporary swap files. Make sure the swap files are on the SSD.

John

[discardedastro]

SSD vs HDD is the single biggest win you can have for performance in PI. SSD performance relevant to PI is the sequential read-write perf, which many cheaper SSDs won't be terribly good at. NVMe disks like the Samsung 970 EVO series will be absolutely best - and can go into pretty much any machine with a PCIe x4 slot spare with a £10 adapter. That's definitely the single best upgrade I did for my old rig, and of course the disk is modern so can be reused if you do go for an upgrade.

AMD vs Intel is also particularly relevant if you're building a new rig - Intel have much less PCIe I/O on consumer (and even server) parts than AMD, and so I/O can be much faster on AMD platforms. Plus, they're a bunch cheaper - and if you look at the benchmarks site you'll note that AMD systems are currently trouncing Intel systems that cost two to four times more. PI can parallelise very well, which also makes it a good fit for the higher-core-count AMD parts now out like the 3900/3950X series. Well worth looking at that world. Mid-level parts like the 3700 and last-gen 2xxx are also worth a look for a budget build. Just make sure you have RAM to match - 32G is a minimum in my opinion.

https://pixinsight.com/benchmark/benchmark-report.php?sn=AHPMET8XQINSKAD45P47813ZH07815BN is me 

Took me about 14 seconds to open your photo, entirely CPU-bound - I would guess that working with that debayer pattern is quite intensive, since similar frames but from a Nikon Z6 only take a few seconds.

It used to take me (on an i7-2600K) best part of half a day to put an image together using mono frames; I can now do that in under an hour. My laptop, an i7-8650U, is much clunkier but that's partly down to thermal throttling.

[dannybgoode]

Agree with the above. Assuming your motherboard supports NVMe then even a small 128gb drive will make a massive difference for you put your swap files on there. 
 

Note that it is worth experimenting with multiple swap files on the same drive. On my pc I find that 5 or 6 files are optimal with slower performance either side of that. 
 

I run 32gb as well which helps. PI performance scales with what computing power you can throw at it. My laptop is no slouch - a 2017 Dell XPS 13 i7 with 16gb but it’s destroyed by my desktop which is a Ryzen 5 2600 with 32gb and an NVMe SSD. 
 

My laptop gives a benchmark score of 3800 and my desktop around 12000 because I’ve spent a bit of time optimising it for PI. Most tasks take a handful of seconds to complete. 

[Nikodemuzz]

My OS+programs and the processed files live on different hard drives, both of which are SSD's. The CPU is Intel i3 8100 which is a bit slower than the Ryzen 5 2600 that Danny mentioned above, but not by miles. In that light it seems that the biggest difference between our systems would be RAM, I have only 16 GB. Yet our benchmark numbers are quite far apart. It seems plausible to me that most of that would be accounted for by the RAM increase, but what other optimizations can one make? I'm referring to Danny's comment about how he has optimized his machine for PI.

[dannybgoode]

1 hour ago, Nikodemuzz said:

My OS+programs and the processed files live on different hard drives, both of which are SSD's. The CPU is Intel i3 8100 which is a bit slower than the Ryzen 5 2600 that Danny mentioned above, but not by miles. In that light it seems that the biggest difference between our systems would be RAM, I have only 16 GB. Yet our benchmark numbers are quite far apart. It seems plausible to me that most of that would be accounted for by the RAM increase, but what other optimizations can one make? I'm referring to Danny's comment about how he has optimized his machine for PI.

The performance difference is true to an extent as a headline benchmark figure however the multi-core performance difference is huge bearing in mind the Ryzen 5 is 6 core / 12T and PI makes full use of all available threads so I am running 12T against the i3's 4T. :

https://cpu.userbenchmark.com/Compare/Intel-Core-i3-8100-vs-AMD-Ryzen-5-2600/3942vs3955

My CPU score  in 1.8.8.5 is 10485 and the swap score is around 15200.

I also overclock my CPU and have spent time optimising the swap files and slimming down non-essential background tasks etc.  I did play with RamDisks for the swap files and when I was spinning metal storage drives with a SATA SSD boot drive this made a huge difference.  However now I have an NVMe SSD the difference in swap performance between that and RAM is negligible.

 

 

[han59]

I wasn't aware of the X-Trans CFA pattern, but this post triggered me to add experimental X-Trans support to ASTAP (v0.9.351).  Maybe useful for Fujifilm camera owners.

Han

[sloz1664]

In my experience  PI takes a lot of processing power to calibrate your files. I now use APP for calibration & Pixinsight for post processing. I use an i7 Dell precision M700 16Gb ram & an SD disk drive.

Steve

[Whistlin Bob]

On 02/03/2020 at 08:55, Nikodemuzz said:

do any of you save time by doing the pre-processing in DSS and post-processing in PixInsight?

After 2 or 3 attempts at pre-processing with PI which took whole evenings and didn't seem to yield anything much better than what DSS gave me I went for that approach.

[Nikodemuzz]

17 hours ago, han59 said:

I wasn't aware of the X-Trans CFA pattern, but this post triggered me to add experimental X-Trans support to ASTAP (v0.9.351).  Maybe useful for Fujifilm camera owners.

Han

Interesing, Han! Kudos for developing the software!!

[Nikodemuzz]

2 hours ago, Whistlin Bob said:

After 2 or 3 attempts at pre-processing with PI which took whole evenings and didn't seem to yield anything much better than what DSS gave me I went for that approach.

This might be my workflow too unless I find some optimization options in PI that would speed up the certain processes.

12 hours ago, sloz1664 said:

In my experience  PI takes a lot of processing power to calibrate your files. I now use APP for calibration & Pixinsight for post processing. I use an i7 Dell precision M700 16Gb ram & an SD disk drive.

Steve

Thanks for contributing your experience. What is APP exactly? I tried googling it, but unsurprisingly searching with the keyword "app" does return a lot of hits, most of which not very helpful... 😃

[Adreneline]

12 hours ago, sloz1664 said:

I now use APP for calibration & Pixinsight for post processing.

+1 for this approach - it's a winner for me too. I still occasionally use DSS if I just want a quick look at a stack of subs though.

On 06/03/2020 at 18:41, discardedastro said:

NVMe disks like the Samsung 970 EVO series will be absolutely best

Totally agree - transformed my MBP, iMac and Lenovo PC laptop. My NUC has an M2 type drive which is also amazingly quick on read/write access.

6 minutes ago, Nikodemuzz said:

What is APP exactly?

Astro Pixel Processor

 

 

[Nikodemuzz]

7 minutes ago, Adreneline said:

+1 for this approach - it's a winner for me too. I still occasionally use DSS if I just want a quick look at a stack of subs though.

Totally agree - transformed my MBP, iMac and Lenovo PC laptop. My NUC has an M2 type drive which is also amazingly quick on read/write access.

Astro Pixel Processor

 

 

Thank you! 😃 I'll certainly have a look. I'm trying not to steer the conversation too far off topic, but could you briefly share which tasks you do in APP and which in PI?

It seems that Fuji files might not yet be supported. However, I might not be using Fuji exclusively so that might not be such a big issue in the end.

[Adreneline]

42 minutes ago, Nikodemuzz said:

which tasks you do in APP and which in PI?

In a nutshell I do calibration, registration and integration in APP. I then move the individual masters into PI and register them again before cropping edge effects and applying either dynamic or automatic background extraction. I then use PI to complete the post processing and finally tweak the colours in Photoshop - or not as the case may be.

HTH

Adrian