Capture software for Linux

[NickK]

Just a side point about bilinear - GPU graphics cards will do this for free if you decide to use GPGPU (OpenCL etc).

[SnakeyJ]

You're making some excellent progress on this James - despite uncovering a world of gotchas in V4L2, the camera SDKs, video encoding and teaching yourself QT on the fly!   IMO a darn good effort over 2 weeks, sandwiched between work and family life!

Pretty sure that a modern multi core desktop/laptop will have stacks of spare cpu/gpu for doing the debayering and scoring as a pre-process and that the USB bus and disk i/o will be the limiting factors.   With the Pi's you'll be limited by both processor and mediocre i/o to SDHC - which is going to be a challenge to say the least     Though I can see the possibilities for a small form (micro/min ITX) low power system running full time in an obsy under linux.

[JamesF]

Thank you, Jake  I guess progress is a reflection of just how many cloudy nights there have been in the last few weeks, too

I've been quite surprised by how little performance is impacted by some of the image processing, I have to say.  Where I've been doing BGR->RGB conversion, image flipping or histogram creation it's made very little impact to the frame rates possible.  I agree that the data rate from the camera and storage of data on disk are going to be the real limiting factors.  There's always the possibility of using ramfs on Linux I guess, though now single capture runs can get into gigabytes of data you'd need a lot of RAM just to be able to do a single run.

There are so many different debayering algorithms now that it's hard to work out what's worth trying and what isn't.  The simpler ones seem to go a good enough job for alignment and focusing, but the more complex ones which often take multiple passes (up to half a dozen in some I've seen) over each colour appear to produce far better results.  If I've understood correctly there even appear to be some that attempt to account for the fact that the CFA doesn't give a hard cut-off in the frequency response, so "near green" reds and blues actually get a response in the green pixels and vice-versa.  From memory I believe Nebulosity (could be one of the other similar packages though) offers a choice of something like ten different debayering methods!  Obviously it's necessary to debayer for viewing on-screen at capture time and probably for focus scoring, auto-cropping and so on, but this is why I'm thinking it probably makes sense to store the raw data on disk and handle the final debayering in processing.  Or at least have the option to do so.

On the other hand one might argue that if you're that worried about the quality of your debayered output you probably need to be getting a mono camera and filters.  But on the third hand, sometimes being restricted to a colour camera is the best life offers and the lack of availability or affordability of a mono camera/filters/filter wheel etc. shouldn't be a barrier to making the best of the data that is available to you.

James

[JamesF]

Well, I thought it would be relatively straightforward, but adding support for 16-bit mono is getting quite hairy

I have a list of "things to do before making a binary alpha release".  For some reason 16-bit mono support is on it.  I don't know why I have it there rather than in the list of "things to do before making a source release" or "enhancements".  When I write software for paying customers I often end up with a list of "things that must be done before I can claim requirement X is met" and whilst I might add to that list, I rarely remove things from it.  it's a surprisingly hard mindset to get out of

James

[JamesF]

I think I'm just about there with the mono 16-bit support now.  I still have to do the histogram and deal with saving the data.  Actually, I have ffmpeg writing an AVI file which is claimed to be 16-bit greyscale raw video, but when I try to play it back the output says the frame size is 640 x -480  Clearly there's a bug somewhere, but at the moment I'm not sure where.  I've checked everywhere I do anything that makes the frame size accessible and it's always showing correct.

Windows Media Player correctly identifies the frame size but decides it is a colour file, as does Registax.  AS!2 doesn't like it at all.

So at the moment it's anyone's guess

James

[cgarry]

James,

I found the same thing when I investigated 16-bit mono video.  It mostly gets interpreted as RGB 565.  The Y16 codec is defined but I never followed it up as SER is a better format.

Also, a frame size of 640 x -480 is perfectly valid.  It just means a vertical flip is required.

I can't find any recognised pixel format definition for raw video frames, nor a codec that directly handles them.  Is anyone else aware of one, or shall I just shove them in an uncompressed raw video stream and assume the user will sort the details out later?

Mainly raw colour video just uses a mono video codec and it is up to the user/software to organise the debayering.  The exception to this is the BY8 codec which is the same as the Y800 codec (as implemented by TIS, ie not true Y800) but the decoding software knows it needs to be debayered.

Chris

[JamesF]

Thanks Chris.  Reassuring to know that you've hit the same problem  Sounds like I should perhaps leave things as they are at worry about them another time.  16-bit mono performance seems dreadful with the ASI120MM.  I suspect there may be much bit-twiddling going on in the interface library.

Your reference to a vertical flip reminds me that I've not done that for 16-bit mono yet.  I'm half-tempted to implement it only for the preview image.  It's not like it really matters which way up up (or left-to-right) the saved image is.

James

[cgarry]

I thought I was getting about half the 8-bit frame rate in 16-bit mode with my ASI120MM in FC, so I was happy enough.  Though I only tried it briefly so I could be wrong.

The trouble with all these more obscure AVI features such as -ve heights it that other software, especially video players, often fail to support them. 

Chris

[JamesF]

I guess the Windows interface may be in better shape than the Linux one.  Looking at the output from the ASI120MM I'm inclined to think that unless its heavily cooled those extra few bits aren't worth a great deal over the 8-bit image.  There's an enormous amount of noise there.  Ok, so sitting on my desk running at 35C probably isn't the most kind of tests, but even so...

James

[JamesF]

Well, I think we're nearly there.

I have less than half a dozen pretty trivial bugs to squash, a bit of documentation to write, a script to put together to put the files in the write places and I think I might be ready for a binary release.  Could be done before the end of the week

James

[cgarry]

a bit of documentation to write

That is an unusual step for open source software...

[JamesF]

That is an unusual step for open source software...

Sadly so, yes

James

[JamesF]

Nearly there now.  Have one little bug to deal with, a 32-bit build to do and I may well be ready to let people attempt to break it whilst I get on with implementing some of the stuff I've not done yet.

James

[JamesF]

Well, I'm now at the point where I think I've done enough to kick it out there and see if it works for other people.  I've built a 32-bit version and I'm just installing a few different OS releases to see if it works with them and how well.

I should probably concentrate on more functionality than support for more cameras at this point, but I've been looking through all the QHY stuff I've collected.  What a nightmare it is!  So far I've found at least 25 different camera models, and I'm sure I'm done yet.  The open source QHY SDK supports about half of them, specifically excluding the QHY5 (but including the QHY5-II models).  At least some of the cameras present one USB vid/pid pair when connected and then completely change them when their firmware is uploaded, but not all appear to have uploadable firmware.  There's no obvious way I can see to tell which is the latest version of the firmware, either.  All very jolly.

James

[SnakeyJ]

James - might be worth posting in the QHY Forum for some more info on this, they are increasingly more active on the lInux side lately with the development on their Intercam Series cams.   QuiHy I think is the boss and responds to most posts, but LZR is one of the developers and might be more helpful.   Otherwise Torsten @ Firecapture knows a lot of the developers and may be able to point you in the right direction.

How does the SDK support look from ZWO for the ASI120's - seems to me that this is actually in more widespread use as a planetary camera, with most of the QHY5L-IIs being used for guide, though there is little practical difference between the two and both companies seem to have had their own issues with software/driver support.     The Pt Grey cameras are probably more of a niche, probably due to cost - though the software and SDK support from Pt Grey seems very comprehensive, due to their industrial focus.

Good luck on the Alpha launch!

[JamesF]

I've joined the QHY forums and asked for access to their developer area.  We'll see what comes of that.  I suspect their focus is away from supporting the older cameras now, especially the ones that are no longer in production, but there must be an awful lot of them out there so it would be nice to support them in my code.

The SDK for the ASI cameras is basic and not well documented, but functional.  I don't like the fact that it trashes frames if the USB bandwidth control is set too high and I'm inclined to believe that could be entirely dealt with in software, but without seeing the code I don't know and they're far from the only cameras to have the same issue.  It seems to work fine now I have it configured suitably anyhow.

The Point Grey SDK does indeed look comprehensive.  Almost to the point of being overwhelming.  I think they are to be applauded for that.

I can't help wondering though...  So many of these companies claim "trade secrets", "giving away competetive advantage" or other such guff when asked about releasing the source for their drivers.  For the most part I reckon that's a load of cobblers.  Many of the cameras are built around well-known chipsets.  Anyone who is reasonably competent (enough to write their own driver, at least) and desperate to understand what's going on at the PC side of the interface can easily get hold of the necessary tools to reverse engineer the software enough to re-write it.  And re-writing one probably wouldn't take that long either.  I think it's far more likely the story is "we don't want to", "the lawyers won't let us" or "we're too embarrassed to".  The camera firmware may be a different case, but few people are asking for that to be open source.

But that's an argument for another time

Meahwhile, back in Gotham City, my install script appears to work and my Mint 15-built binaries work on Mint 14, as do the SPC900 and ASI cameras.  I'm just finishing installing Ubuntu 12.04 to try it there and I believe I have copies of 11.04 and 10.04 that I can try it on.  Unless they've already been used for pigeon-scarers I might have some of the .10 releases as well.  I'm just trying to decide if I can face downloading a couple of Fedora releases on my 1Mb/s ASDL line to try.  I should get my brother to download them on his fibre link and post them to me.  As a man once said: "Never underestimate the bandwidth of a truckload of CDs".

James

[SnakeyJ]

I can't help wondering though...  So many of these companies claim "trade secrets", "giving away competetive advantage" or other such guff when asked about releasing the source for their drivers.  For the most part I reckon that's a load of cobblers.  Many of the cameras are built around well-known chipsets.  Anyone who is reasonably competent (enough to write their own driver, at least) and desperate to understand what's going on at the PC side of the interface can easily get hold of the necessary tools to reverse engineer the software enough to re-write it.  And re-writing one probably wouldn't take that long either.  I think it's far more likely the story is "we don't want to", "the lawyers won't let us" or "we're too embarrassed to".  The camera firmware may be a different case, but few people are asking for that to be open source.

James

Think you hit a few nails on the head with this - and it certainly seems that Aptina are particularly hot on the NDAs and the documentation for their sensors.   The semi conductor market is certainly cut throat and very protective of its design/competitive edge, which is entirely reasonable, but I also tend to think there's a culture of glossing over and denying the imperfections.    Like so many tech industries the majority of code output relies heavily on historic development and I suspect many companies don't retain the knowledge/staff to go back and chase the bugs out of older portions of code.

If I were president, I'd legislate for all companies release full source for anything over 10 years old - though companies and even governments are likely to end up red faced

[JamesF]

I now have access to the QHY developer board.  It's a bit, err, chaotic, but I now need to spend my time reading through it a bit.

James

[JamesF]

Ubuntu 12.04 just finished installing and I've patched it fully.  The application works, but the SPC900 code throws a wobbler.  It appears that between the v3.2 kernel that is in Ubuntu 12.04 and the v3.5 kernel that is in Mint 14 (and presumably therefore Ubuntu 12.10) the driver was fixed.  I shall have to ponder on the need for a userspace SPC900 driver for the earlier kernel.  If I ponder long enough with any luck there'll be no requirement for one

James

[JamesF]

Actually, it seems that both 12.04 and 11.10 (which I am now installing whilst downloading Fedora 19) are able to operate the SPC900 because they offer me the opportunity to crack the monitor by taking a picture of myself to go with my username, and display what is currently visible from the camera quite happily.  Perhaps I need to look into this a little further.  Could be that support for the SPC900 in earlier kernels is merely a case of tweaking a few things rather than wholesale rewriting.

James

[cgarry]

I have set up one of my old laptops with Mint 15 in anticipation of your release!  Though it does seem to me that you have taken on a long term, full time project with this one...

Cheers,

Chris

[JamesF]

Though it does seem to me that you have taken on a long term, full time project with this one...

Sometimes you just have to throw yourself off and trust that it will all work out ok in the end

But wait no longer.  I have started a new thread so the topic is more relevant:

http://stargazerslounge.com/topic/197489-alpha-release-of-capture-application-for-linux/

James

[NickK]

Awesome job

[JamesF]

Thanks to everyone who has offered feedback thus far.  I've been thinking about another alpha release this evening and am coming to the conclusion that it should include at least:

finishing as much as possible of the functionality in the existing UI -- everything except handling filters, I suspect

restructuring the bits of the code that I'd have done better if I'd know more when I started

and optionally:

Support for the QHY5

Support for those cameras supported by the QHY open source Linux SDK (most importantly for me, the QHY5L-II, but others too)

Support for the TIS cameras, or at least the DxK21s, assuming I can get hold of them to test and any information about making them work

Anything anyone else feels is desperately needed as soon as possible?

James

[JamesF]

I've been thinking about another alpha release this evening

No wonder I have bugs in my software with that sort of woolly writing.  I do of course mean "(I've been thinking about another alpha release) this evening", not "I've been thinking about (another alpha release this evening)"

James