INDI

[JamesF]

Well, I have to admit that I have no idea what's fashionable technically

My interest without doubt lies with Linux though. I've been a user, programmer and admin on UNIX or UNIX-like operating systems for almost thirty years now and trying to work on Windows is like wearing a strait jacket. Perhaps as a result I'm not that worried about what GUIs look like either. One of the prime reasons for windowing systems to exist for me is so I can run lots of terminal sessions concurrently

James

[AstroTux]

I guess what I'm trying to say is that we, astronomer software developers, need to coordinate our efforts and come up with a consistent story when it comes to open source astronomy software. There's just way too many directions people are pulling at it, thus hampering headway. On Windows devs stuck with ASCOM and look at how rich and well supported that platform is. By the way, INDI runs on Windows as well, with support for ASCOM drivers (only on Windows) so there is absolutely no reason for INDI not to take over eventually -- we just need to get our story straight

+1 for that.

We can all get into our "I like this and I don't like that" moments but Linux is developed mainly by those who are directly confronted with a particular problem and need to find a way around it. I have developed software when I have needed it ( something that is not that easy to do on other systems without a great outlay of money first ) and I feel that this is the way a lot of open source code develops. We don't develop it to produce a 'product' but to help out where we can. Also, as James has pointed out, many of us don't like wearing a strait jacket .

It would seem by this thread that there is core of people interested, so maybe we should put our heads together and see what we can come up with.

Alan

[JamesF]

I assume Nick that your OSX drivers for the ATIK cameras are not open source otherwise ATIK wouldn't have released the information required to write them to you?

James

[AlexRoman]

My interest without doubt lies with Linux though. I've been a user, programmer and admin on UNIX or UNIX-like operating systems for almost thirty years now and trying to work on Windows is like wearing a strait jacket. Perhaps as a result I'm not that worried about what GUIs look like either. One of the prime reasons for windowing systems to exist for me is so I can run lots of terminal sessions concurrently

Likewise. Furthermore, getting all of this to work on my desktop PC means I can easily re-compile for <insert one of thousands of ARM boards here> and have an embedded telescope server with support for all my gear -- great for the Canadian winter!

It would seem by this thread that there is core of people interested, so maybe we should put our heads together and see what we can come up with.

Agreed. I've also seen a BUNCH of open source code throughout the web in various repositories, like github, bitbucket, etc. I've also seen duplicated or re-implemented code... What would be great is if we can get all the Linux developers on a forum and they can post what they're working on along with a link to the code. That way there'd be a way to see if something was already done before.

Also attempting to support linux is a nightmare - getting the user to install the right packages and not have issues with the dependencies. Now you could write it all as a wrapper..but the valuable time then moves from resolving the astro problem to solving the linux dependency problem (which varies depending on what other packages the user has installed or the development libraries require).

Lastly it's the end user experience - and I'm sad to say linux apps with QT/vxWidget look ugly on all the platforms but natively the GUI development time than the likes of OSX/Win.

Like JamesF said I'm less interested in GUIs and how they look, although I do try to keep it consistent. At the moment I prefer KDE with its Qt based UI and I have to say I rather like it. I find it very elegant. But again, like JamesF also said I tend to have a bunch of terminals open along with a bunch of vim windows open...

Supporting multiple distros is NOT a nightmare for pure open source. If you have binary-only software, your best bet (seee Perforce) is to package the whole thing as a statically linked binary. If you have open source only, then you just need to tell people what libraries you expect to be available on the system. Maybe I've been at it for way too long, but I don't think installing a few -dev packages is a big deal if you decide to compile from source. Alternatively, provide a repository to support your distro of choice.

Back to INDI, I really like the distributed nature of the protocol. It means I could have local or remote processes saving/cataloging images, logging RA/DEC of where my scope gets slewed to, and last but not least maintaining job queues. I could even make a HTML5 app for smartphones... The possibilities are endless!

[simonct]

Ah, so this is where the Unix guys hang out. I started out on SunOS circa '92 but have been doing Mac and, more recently iOS, since 2000 (the Mac's unix of course so hopefully that still counts). Coincidentally, I also worked at Viables for 5 years but that was for Sony Broadcast R&D - small world.

I really like the basic proposition of INDI but the current implementation does look a bit crufty. However, I also think most of its drawbacks probably aren't relevant if you're planning on stringing everything together with bash scripts - and there's a distro for the Raspberry Pi ;-) I'm actually having far too much fun writing my own Mac framework but INDI interoperability's definitely something I'm looking at and I've already submitted one incredibly minor patch with probably more to follow when I get the time.

It's arguable that the protocol's inefficient but consider whether the web would exist without plain text HTML and email… It could probably do with an update to JSON for maximum interoperability but XML's probably better than some opaque binary format unless you're doing something really performance critical.

So, +1 from me !

Simon

[JamesF]

There's a fair bit I like about INDI and some stuff I really don't. Like Simon, I think the basic idea seems good. But then there are things that really annoy me, such as "Lights". To me, "lights" are a client-side presentation issue. The INDI protocol doesn't have any business involving itself in that and if anything I think they should be replaced by some enumerated value type that makes no judgement about the intended use. The "numberValue" thing makes me unhappy, too. If a device says "for this parameter I accept a numberValue between 0 and 100" the client should be in a position to be sure the data being submitted is valid. If that parameter happens to be a brightness value then it's ridiculous for the client not to know that sending back 55:44:33 is wrong. I can't really see a good reason for not saying "this must be an integer" or "this must be a float" and so on.

And there's also the issue of throughput. If you're doing some sort of long-exposure imaging where large images get pushed over the network once in a while then that's one thing, but if you're doing 60fps planetary imaging with a 640x480 16 bit per colour RGB image that's a big fraction of 1Gb/s and you surely don't want that going anywhere near your network, especially if you're working remotely. I think any piece of hardware that has the capability of producing data at high speed needs some sort of mechanism for saying "Put this somewhere local to you". The implication being that the client will come back for it later, or pick it up out-of-band. And then some method for picking it up later, obviously.

I think there's also a need for a client to be able to "claim" a device to stop other clients using it as long as the initial claimant is still present. You don't, for example, want someone else starting to use your camera just because you're not using it whilst you're waiting for the filter wheel to change filters.

FInally I think there's a need for some sort of override to allow a manager process to decide that everything is going to get shut down regardless of what it is doing because (for example) the rain sensor has tripped and everything needs to tidy itself up right now because the roof is closing.

I'm not sure that XML is the right choice for the wire-format of the data as it seems excessively verbose, but I guess it is easily parsed and there are XML libraries for a huge number of languages. It's the sort of thing you could stick with until it clearly doesn't work, I guess. If the comms side were wrapped into a separate library it wouldn't be that hard to change at a later date. It does strike me that there's no reason not to have an Android/IOS client, so the protocol ought to be lightweight enough for that to work over (say) a 3G network.

Ideally I think I'd want to be able to do things like run an observatory remotely, perhaps starting jobs off from one machine, disconnecting it and reconnecting from elsewhere. I'd like to be able to script imaging runs using Perl or Python (for example) and be able to do something like "Point the scope at the clouds where the moon is and track it. Grab an image until you get one above a certain quality/brightness, then keep the last 120 frames until at least 90 are of a given quality. Stop.". Being able to multiplex the capture of (say) LRGB images from a pair of scopes on the same mount where the exposures may be difference lengths would be good, too. Even better if it were possible to submit a script to some scripting engine that would manage the process and be able to check up on it from the pub. Some hard-core imagers might like to be send an alarm to wake them up if there's a problem or when a run is finished.

James

[JamesF]

I'm still working my way through the documentation, but a couple more things...

The implementation appears as far as I can see to conflate mounts, telescopes and guide scopes into a single object, yet a mount may carry more than one telescope and a guide scope.

It's not clear to me how the property setting system works in terms of handling errors. In fact it really does seem a bit clunky. I can see that sending an alert in response to a command that has been given and failed some time later might be reasonable, but I'd have thought that it ought to be possible for an immediate response to be returned to the client in the event that it has made a request with values that are invalid.

James

[NickK]

[NickK]

Sorry for being cryptic - a mad dash this morning.

There are many issues with the architecture, as I've said - once you start running big data the architecture becomes a liability. There are more issues that you've not identified - and when you look at correcting it.. it's easier to write your own.

However the hardest part is getting people to really buy into the vision and not just attempt to use your code as a convenient repository for their own.. which probably explains the reason ASCOM was a little behind closed doors.

Part of the reason that AOSX has been slow is that we've taken time to think about some of the real gotcha (ie it already has a claim/release with management, the image you interact with is a proxy class so it can be stored local but referenced as an object elsewhere - this is why the ATIKOSXDrivers appear to have some unusual 'complexity'). We still have a long way to go - but I want to get a release out - and no it will cover more than ATIK's cameras!

[JamesF]

So what other issues have you found?

James

[NickK]

So what other issues have you found?

The big one.. from what I can see - everything goes through a single process..

[JamesF]

That's not the impression I have. It looks to me as if it's possible to have every driver as a separate process. Certainly it's clear from the docs that it is possible to have a single process sat in the middle doing mediation, access control and so on, but I don't see that it's a requirement from what I've read so far.

James

[JamesF]

On a related note, what is the situation with AOSX? I can find a web page at sourceforge with very little information on and no source code. Is that because there's none available yet, or is AOSX now a closed-source project? Or am I looking in the wrong place entirely?

James

[AstroTux]

I get the feeling that I need to have a long, long read through the INDI documentation when I get home, if I want to stay in contact with this thread :grin: ......

Alan

[JamesF]

I get the feeling that I need to have a long, long read through the INDI documentation when I get home, if I want to stay in contact with this thread :grin: ......

The place to start is probably the developer's documentation here:

http://www.indilib.org/develop/developer-manual/88-purpose-of-manual.html

James

[AstroTux]

Thanks James, I'll have a long look when I'm back in Englandshire.

[AlexRoman]

I'm still working my way through the documentation, but a couple more things...

The implementation appears as far as I can see to conflate mounts, telescopes and guide scopes into a single object, yet a mount may carry more than one telescope and a guide scope.

It's not clear to me how the property setting system works in terms of handling errors. In fact it really does seem a bit clunky. I can see that sending an alert in response to a command that has been given and failed some time later might be reasonable, but I'd have thought that it ought to be possible for an immediate response to be returned to the client in the event that it has made a request with values that are invalid.

James

You can call IUUpdateNumber (or similar for a different property) and the INDI API will check the parameters against the set min, max, step values. If the values are incorrect it automatically sends the appropriate response to the client.

The big one.. from what I can see - everything goes through a single process..

Somewhat true. The indiserver process is a single process. But that process spawns all the driver processes (command line params to indiserver) and passes everything around through stdin/stdout to the child driver processes. From the looks of it a select() loop on multiple fd's. Fairly standard *nix stuff...

I had an idea to have a controller daemon of sorts that takes in a site setup configuration that tells it what indi server to connect to, what devices to use, etc. Then that daemon takes in job requests and adds them to a queue and serializes access to INDI. A nice HTML5 web interface would be sweet here as it would work with most modern smartphones and browsers. Of course, a non-GUI interface is mandatory

[JamesF]

Somewhat true. The indiserver process is a single process. But that process spawns all the driver processes (command line params to indiserver) and passes everything around through stdin/stdout to the child driver processes. From the looks of it a select() loop on multiple fd's. Fairly standard *nix stuff...

That sounds like it could be possible to convert to the same model as used by Apache where multiple worker processes spawn off listening to the same socket and whichever is free handles the next incoming connection. Or by nginx, which is effectively a single process (actually more than one, but generally a very small fixed number of processes) that handles huge request volumes.

James

[simonct]

You both beat me to the punch in pointing out that it's the standard fork/exec model of spawning driver processes. The thing is, afaik, there's no dynamic detection of devices coming and going. You plug everything in, start the server and off you go. Not very plug and play so not much good for a desktop environment but I'm not sure how much of a show stopper that is in a remote observatory. I'm sure nothing that can't be fixed however.

As for the protocol, ideally there needs to be a bit of a cleanup and possibly even a move to a different transport and format such as WebSockets+JSON. Doing that could bring it into the web-centric network world and make it interoperable with just about everything from web clients and native apps to command line scripts.

I also think the architecture needs a tweak with the introduction of a controller layer at the server end. Instead of the client talking directly to the driver, it communicates with a particular flavour of controller and it's that which deals with one or more drivers to get a particular task done.

I use something like this pattern in my own in-process framework to do things like automatically saving exposures to the internal library and optionally capturing sequences of images to a locally stored video file (and live video would be the same). It'll also handle auto guiding at some point.

The controller then becomes the primary way of interacting with devices using a high level interface and it handles the details of dealing with the drivers themselves locally rather than trying to do this at arms length over a network. It also does the heavy lifting of dealing with exposures by saving them locally rather than streaming everything back to the client; clients get a uuid and can then retrieve the metadata and/or pixels on demand.

Simon

[JamesF]

I have to admit that I don't find the suggestion of moving to websockets and JSON a compelling one. With websockets particularly it just seems like putting an extra layer of complexity on top of the standard Berkeley socket system just to make one means of access easier. I'd have thought a single "proxy" talking websockets to the outside world where required would centralise security and access control and allow normal socket-based applications to carry on as ever.

Regardless, I think there are other things that need sorting first such as how the complete system should be modelled. I'm really not convinced INDI has that right.

James

[simonct]

Fair enough, it was just a thought. But it seems that JSON's now the lingua franca of the web and websockets is a pretty lightweight framing protocol - you have to use something so you might as well use something standard. Nginx supports websockets natively now so you could possibly use that as a proxy, because, in reality, the INDI guys are never going to change the protocol anyway

The modeling question is more interesting and I simply haven't looked into how INDI represents devices in sufficient detail to contribute much right now but it's possible this is where domain specific controllers might add something useful e.g. camera controller, mount controller, etc.

Simon

[StuW]

Just wanted to say I'm following this thread with interest as an ex SCO server admin and *nix user. The main reason I still have a windows partition on my lappy is for Ascom and associated software. I'm not as experienced in *nix dev as some here, but will offer help in any way I can.

[AlexRoman]

You both beat me to the punch in pointing out that it's the standard fork/exec model of spawning driver processes. The thing is, afaik, there's no dynamic detection of devices coming and going. You plug everything in, start the server and off you go. Not very plug and play so not much good for a desktop environment but I'm not sure how much of a show stopper that is in a remote observatory. I'm sure nothing that can't be fixed however.

As for the protocol, ideally there needs to be a bit of a cleanup and possibly even a move to a different transport and format such as WebSockets+JSON. Doing that could bring it into the web-centric network world and make it interoperable with just about everything from web clients and native apps to command line scripts.

I also think the architecture needs a tweak with the introduction of a controller layer at the server end. Instead of the client talking directly to the driver, it communicates with a particular flavour of controller and it's that which deals with one or more drivers to get a particular task done.

I use something like this pattern in my own in-process framework to do things like automatically saving exposures to the internal library and optionally capturing sequences of images to a locally stored video file (and live video would be the same). It'll also handle auto guiding at some point.

The controller then becomes the primary way of interacting with devices using a high level interface and it handles the details of dealing with the drivers themselves locally rather than trying to do this at arms length over a network. It also does the heavy lifting of dealing with exposures by saving them locally rather than streaming everything back to the client; clients get a uuid and can then retrieve the metadata and/or pixels on demand.

Simon

+1

I like the approach of the local controller. Do you mind sharing the source code somewhere? Sounds like it's along the same lines with what I had in mind...

As for device detection, while not too intuitive, the protocol has support for this. When a device is connected, an indi driver that was dormant until now can wake up and say "Hello, we now have device XXX with these properties YYY and ZZZ". I do not think that INDI should bother with device detection. This is an OS-level thing and should be implemented on a per-device driver case...

[Chris Rowland]

I'm looking at this from the perspective of an ASCOM developer with virtually no Linux experience.

INDI seemed to specify a communication method without specifying what command must - or should - be implemented and that seemed to leave that to the driver and application to agree about - not sure how.

ASCOM as implement is Windows only because it's COM based but the set of command definitions could be implemented using just about any underlying communication method - I did an experimental network solution where each command and response was parsed into a text string that could send to a server. I made the mistake of implementing it using UDP and the while thing dissolved into a sea of disagreement about the merits of UDP.

I wonder if ASCOM could be used the define the command set and have other ways of implementing the communication between applications and drivers.

Not trying to impose this but wondering if there's something that will help.

Chris

[NickK]

All good points.

* controllers - this follows the natural requirement for low latency and high accuracy for devices. This is really an example of a service.

* ascom - there are aspects I would change, particularly around the management and some rearchitecturing.

What nobody here has mentioned, and the big reason for V2 of AOSX, is saving the work of a frustrated APer - if a driver fails or a component it should have minimal impact.

Please - think of the user... with a simple experience. Otherwise you'll end up being regarded as difficult to use..