Friday, August 29, 2014

VOS Work Progressing … Slowly

Only have time to work on it at night for an hour or so.

I have basically rewritten the entire codebase in the last three months and gotten it to compile with Open Watcom to Windows 32, Win 3.1, Linux, DOS, Free BSD all in super generic ANSI C.

I used a lot of the existing code fragments I have demoed screens from up here except I now have the PHP processor compiled into the source. The page content is now created in PHP some of which is also compiled into the executable.

I have realised after examining my previous effort that my message format should be ASN.1 passed to every device for everything that is SCADA related. This permits tiny, minuscule devices to call the VOS web server and to received control messages from it in real-time. The AJAX calls can return JSON, SOAP XML or raw hex for ASN.1 but this way I can define the messages all in one central format and eliminate lots of redundancy. Since I also found a 3K ASN.1-to-Javascript decoder I can use these messages inside client web page logic where needed.

This is what I have compiled into a 1.2 megabyte executable, standalone, no file structure and no support files required (other than optional .INI configuration) :

(Everything below on the client side is aimed at ancient browsers HTML 1.1)

  1. Full featured web server supporting Websockets protocol and UPNP broadcasting 
  2. PHP processor
  3. SQLite Multithreaded
  4. Mailslot Manager (cross platform IPC for daemons running on server to monitor devices)
  5. ASN.1 Message Factory
  6. ASN.1 Javascript Decoder
  7. Code 39 Barcode printer Javascript
  8. Cuecat Barcode Reader Javascript
  9. JQuery 1.9 (Javascript compatible with IE 5.5 for very old thin clients costing $1.00)
  10. JQuery UI
  11. JQuery Sparklines (Single line of real-time monitoring data fits into table row)
  12. JQuery Real-Time Clock and Scheduler (Common shows on all VOS pages)
  13. JQuery Formbuilder (w/PHP interface!)
  14. JQuery Terminal (Console window)
  15. Raphael Vector Graphics Library (Javascript compatible with IE 5.5 W/VML or SVG)
  16. Raphael SVG Importer Library ((Javascript compatible with IE 5.5)
  17. Raphael SCADA Display Control Library (Full suite of gauges, switches, signals and dials)
  18. Raphael "Maphael" Library supporting intelligent maps and diagrams with backend
  19. Cross Platform Character Terminal UI (Fully decoupled, runs as optional monitor)

These will be the pages that come built-in to the executable the instant it is run :

  1. Inventory management with barcoding and tracking
  2. Calendar management with integration with all other modules including jobs and work orders
  3. Personnel management with security levels and access definitions
  4. Medical management with individual checkup records
  5. Diagrams and Maps display editor 
  6. SCADA displays to monitor and control all shelter devices connected to same network
  7. Custom logic to process commands to devices for custom uses and scheduling in-app
  8. Job management for repairs, maintenance and work orders
  9. Full memex library with terabyte archiving of reference manuals and full-text search and soundex
  10. Instant cloud recognition and communication through UPNP with any other VOS nodes
I solved an interesting problem last night - how do you create a cross-browser, cross-platform audio alert signal that sounds on a VOS page, no matter what page it is, when an important alert, warning or emergency condition arises? I figured out how to do it with a 1K midi file and a cross-browser META tag that works on everything going back to Netscape 4.0 when it was the hottest browser in town.

When I get on top of this my real dream is to have a built-in LCARS themed interface as a default for all screens. This will require some greater familiarity with the limitations and speed of Raphael as an interface library to implement. Chances are VOS will be released before this happens but it is a good upgrade path for V2.0 and I might even make a Kickstarter campaign out of it.


Sam said...


Sam said...

Last time you wrote about ASN.1 I started looking at it and somehow got to a page on BEEP(Block Extensible Exchange Protocol)which is a kind of uber network library that takes a lot of work out of networking. ASN.1 sounds smaller and probably better for what you're doing but maybe BEEP will be interesting to you if you haven't heard of it. Excuse me if this is old hat.

I found one library for BEEP and it interested me that it had a server built in. Some features of the library below.

* a server that can accept multiple connections, initiate channels and channel pools.
* a client that can make a single connection with multiple channels and multiple channel pools.
* easy creation of required event handlers for implementing the profiles (protocol specifics)
Link to library

Link to BEEP Wikipedia

More libraries,

What's crazy is it's so difficult to build what you're building. So many different libraries. I can understand maybe the SCANDA with drivers not being normal but a basic server that talks to another server to get data then displays that shouldn't be so hard. It shows how the internet is tied together with gum, spit and paper.

Russell said...

I'm flat out amazed at this.

How are you handling database backup/replication?

Sam said...

Hate to keep bothering you about this but this is too good. I ran across a page of NASA open source software and guess what they have. A no kidding open source "Mission Control". It may be to big or complicated for what you want but it's definitely interesting. Java so it runs on most platforms. You can display data from sensors or whatever in many different ways. I didn't see knobs or sliders like the
Raphael SCADA Display Control Library. Maybe (I know maybe can get complicated) it can be tied together. Anyways it's a major cool thing that you can run the same software as mission control for free.



Oops it does have buttons but I don't know what they look like.

Texas Arcane said...


It is just a cross-platform file copy that can be run manually or scheduled to run at intervals in the background.

I thought of the need to write everything out to a USB and if the machine gets fried from EMP or just dies for whatever reason, that exact same configuration with all existing data can be restored on any compatible device running the VOS server.

Texas Arcane said...


The problem with both BEEP and the MCT library, other than requiring an OS running Java, is that they are both complex to administrate and set up from scratch.

VOS is designed to be the Etch-A-Sketch of vault operating systems. If it breaks, you should be able to give it a shake and make it work again that easily.

Many of these systems assume you have an expert standing by who knows how to install, configure and run the tools. VOS is designed for "IJT" It Just Works.

I am imagining Ma and Pa operations with a terminal in their pantry and monitoring their generator and some environmental controls. If the machine crashes, you have to ask, how hard to make it all run again?

I am aiming at thin clients ($1 on EBay) for a single workstation and multiple workstations ($7 ethernet hub + RJ45 cables) that all plug and play. Zero administration other than the most obvious to people who know how to use a web browser. You browse to the server running on your machine to else type in an address for another machine. I am concentrating on the local network with static IP assignment although I may put DNS in there if I get everything else working.

Russell said...


Yeah, that's sort of what I've ended up doing. I do a .dump to a text file, then tar and gzip it before moving it off the primary storage. It works, but it's clunky. I was hoping you had a figured out a better solution for Sqlite :)

Texas Arcane said...


There is a duplication command in SQLite from one attached database to another but it is slow. I may investigate this later.

I dug into SQLite internals to figure out how to do cross-platform IPC caching of messages from the mailslots using a very, very, very obscure feature in SQLite called shared memory tables which even work across threads. This feature is barely documented and possibly the second most useful feature in SQLite for embedded programming. It is lightning fast mechanism for sharing data between threads using the exact same commands in SQL to access any other database that is persistent instead of volatile.

SQLite = Incredible. The most incredible 2-file solution I have ever seen until I saw "PHPLite" or PH7.

Texas Arcane said...


I had gotten shared memory tables running quite well in VOS 1.0 and recently discovered this incredible shared cache mode in SQLite that speeds up multi-threaded access to data managing all locks transparently for the programmer :

Texas Arcane said...


Although I am currently using threads, my goal is to get rid of them before I put VOS up on github or sourceforge.

If you are trying to build something that runs for a hundred years without crashing, I would much prefer coroutines instead of threads, especially because they port so well to devices with no operating system or mere jump tables like DOS, RTOS and bare metal chips.

This way, if I am selling this device bundled with VOS someday, I might have the ability to forego an "operating system" altogether, at least for the server.

Russell said...

"I dug into SQLite internals to figure out how to do cross-platform IPC caching of messages from the mailslots using a very, very, very obscure feature in SQLite called shared memory tables which even work across threads."

See? I knew you had been poking around in the dark corners of the code. I'll have to read up on that, thanks!

"If you are trying to build something that runs for a hundred years without crashing, I would much prefer coroutines instead of threads, especially because they port so well to devices with no operating system or mere jump tables like DOS, RTOS and bare metal chips."

I know this would be too large for your target size, but have you checked out Erlang? "Erlang's runtime system has built-in support for concurrency, distribution and fault tolerance."

I'm not sure if it could be embedded, but from what I understand, the environment is dang hard to topple over and you can update code as the system is running.

Threads are doable if you have solid GC resource management. But despite being the year 2014, we don't have many of those.