Wednesday, September 23, 2015

Ditched ASN as an IPC Message Format

My early experiments with ASN were encouraging. I wrote a lot of support routines which then gave me the confidence to compile more complex message formats.

Once I got away from simple fixed length structures I immediately ran into problems. The documentation for the open source ASN compiler is nearly non-existent. The memory management is appallingly arbitrary and does not easily avail itself of fixed heap schemes like are often used to create stable embedded applications. After a couple weeks fiddling with it I ditched it and switched to JSON again. Less binary efficient but much easier to work with. I had been using JSON for over a year before I tried to get fancy and implement ASN message schemas. It was a mistake, I could not get it to work reliably without leaving fragments on the heap and corrupting stacks sometimes. The problem with ambitiousness is that it makes you shoot too high before you know how solid your model is.

I have always known it would work in JSON and have encountered no problems coding the message formats up with it. I have the current schema working end-to-end for hardware to communicate over the IPC which is implemented on all platforms as mailslots. The basic format of vanilla daemons which send a descriptive manifest of all their properties and capabilities seems to be working pretty well. At home I have finished off an end-to-end communication transfer and it turns out mailslots can issue acknowledge signals and error messages as well as any other IPC mechanism. So there is some assurance for example that if you issue a message to execute a command on a device or to write a value that you will get a receipt coming back from the daemon telling you mission accomplished or including an error message to help you figure out why it didn't succeed. This could be critical in a system where you do something like turn on a generator. You need a receipt from the daemon verifying the generator is now on and running. Vault-OS does not require the ironclad state transfers that are needed in airplane hardware but it does need to be reliable.

Although the Vault-OS server does not need a visual display to run the daemons could benefit greatly from all having some kind of common UI library to use for configuration and monitoring in a separate process. Right now I am imagining something similar to GTK-Server. X-Forms are now available for GTK as well.


Russell (106) said...

I like the UI.

ASN is... ASN.

Had a job where ASN.1 was the communications packet, and they had stuffed a binary version of a Plain Old C Structure in it. We had to parse the ASN message, then deserialize the POCS before we could start to handle the message. It was a series of spinning plates

JSON, despite being not really optimized for binary storage, as you noted, has been the best of all the formats I've tinkered with, both for personal projects and corporate projects handling tons of data daily.

Sam said...

I'm not sure if I told about this already or you already know. Apologies if I did or you do know already. I heard about it from a Rebol coder that really interested in web interfaces. It's a web kit "Sigma Ajax UI builder" that,"...The whole system is totally server agnostic, using standard http:// protocols and Ajax methods to transfer data in plain text, JSON, XML, and other common data formats. The jsLinb library and Sigma builder IDE are both exhaustively documented, they're free to use for commercial projects (licensed LGPL), and with the help of 3rd party tools such as Cordova, Phonegap Build, Node Webkit, or others, you can easily package the created browser-based client UIs into stand-alone apps for distribution in the app stores of any modern OS platform...".

It's 7MB with a built in GUI visual IDE. Wow! I know this doesn't fit on a floppy but practically nothing does that has a GUI. I know you're going to have a browser so why add another GUI over it? Just use the Javascript built in already. He says,"...This little toolkit has been really productive at creating rich client apps which run absolutely everywhere, on any mobile device, and in virtually every old or new browser (I've tested in the stock browsers on the very first models of iPhone and Android 2.2 devices, in IE6, Firefox 1.5, etc., as well as in the newest Chrome, Microsoft Edge, etc.), and the apps run exactly the same everywhere: ..."

You can use any language I think but here's a rebol tutorial. Should be the same for other languages. He's a really good tutorial writer.tutorial.

This is a link to his forum where I heard about it.

You know this could make you a great deal of money. With your coding experience and this little toolkit you could show a run anywhere GUI interface on anything. Even phones. The Evil bosses

would salivate at the prospect of 24 hour monitoring of all his Minions. You could have a small server at your house and show how everything could be controlled from your phone, lap top or a tablet. Maybe even have a tie in to a database and spread sheet which is what they want to look at.

Texas Arcane said...


I'm going to have a look at that, particularly the X-Forms interpreter. X-Forms is a great way to send an interface to a device that doesn't support a browser but it needs a standardized form entry mechanism that can be described at run-time in text. For example, an entrypad on a staging area vault door. If you want to improve the interface look of the pad you change it on the server and push it down to the door without having to take it off the wall and burn a new program onto it.

Texas Arcane said...


The attractiveness of ASN is a central definition for all messages. In all other regards it is just not as effective as JSON. I gave up because I was spending too much time debugging ASN that was supposed to be automatically generated for me. The memory handling was atrocious and the documentation was so sparse you were left to examining the generated code line-by-line.

Texas Arcane said...


If I really wanted to take the easy way I could use a cross-platform text library UI for all the daemons. These UIs are solely to configure, control and monitor the daemon. The user would need it to change the settings on the daemon to use this or that serial port, this baud rate, etc. with the server itself knowing nothing of these hardware details, only what it gets in the mail box from the hardware as as abstraction.

Sam said...

I hate to bring this up again but it just seems to fit so well. Rebol. In all your work you're having to use complicated libraries. Yes I know a browser has JavaScript built in but it's huge if it's counted in with the rest of the code. A text based interface is fine but it just looks so ancient that no one will pay attention to it. I notice whenever you show a screenshot it usually has nice graphics. Those graphics are going to cost you somewhere no matter what you do. Can't fit the libraries on a floppy...or can you? Rebol 3 is used by a company called Atronix. What they do is use Rebol in factory and warehouse automation. They also tie it in with Programmable Logic Controllers. For those that don't know PLC's are mostly banks of relays controlled by computers.

Now you may think the graphics in Rebol are a toy. They are not. They use the Anti-Grain Geometry(AGG) library. A vector based library that does an incredible amount of stuff. It's built in to Rebol. The higher level commands in Rebol don't use all it's features but it's there. Here's a link to AGG.

Now let's look at this. You're using a ton of libraries, of different languages and all adding complexity. Rebol3's exe file, (the one I have on my drive "r3-32-view.exe") runs at a tiny 1.07MB and does damn near everything. The only thing I could think of you need to add is that excellent control library PVBROWSER.

Growth is there too. Rebol is interpreted but RED programming language is a rewrite of Rebol that uses almost the same syntax. It will be compiled though. It's actually variable. It can be compiled or JIT or even at a higher level. It's the first language to go from systems level programming up to DSL's all in one small package. He's got a lot of funding and will probably finish in a year. And since you hate Americans so bad you can feel at ease it's being done in China. This means you can get the thing going in Rebol and then compile it in RED a short time from now.

And if you just HAVE to have it compiled there's something new for Rebol called Ren/C. It's,"...Ren/C is intended to not have an executable at all...but to be a library for embedding a Rebol interpreter into other programs...". I don't understand all the functions of Ren. It apparently extends it a lot for https and a lot of other stuff I don't know about. Here's a page that has some info on the different branches of Rebol. Down the page it talks about Ren and mentions Ren Garden which I also don't understand. The work on Ren and Ren Garden is done by Hostile Fork who is so smart I don't understand but maybe 50% of what he's doing.

The main problem with Rebol is as it was being open sourced, RED was being worked on at the same time. Well that kept Rebol in limbo because everyone knows RED will be great but it's not here yet so it kind of held up Rebol3. The only people really working on Rebol3 are making money with it. They release their code but not all as they have to make a living.

Link for RED.

I bet you could build Vault OS with Rebol in a couple weekends if you got serious about it.

Sam said...

I think I forgot the link for Ren. My apologies