Monday, December 21, 2009

Vault-OS Saved My Life Today. No. Seriously.

I have been so busy the past couple of months I have not been down in Firehold Bravo since first week of February 2009 and then only to install a new entryway ladder into the primary living drum.

I went down today to clean up - I noticed a week ago after opening the hatch it was rife with spiders. Only eight months of neglect and it had turned into kingdom of the arachnids in the primary and secondary access tunnel. We're talking horror movie cobwebs here. The granddaddy longlegs had moved in to capture the ants who have been marching on the dehumidifier lines to get water.

Vault-OS "Sentry" (the DOS based version) had not been run in eight months. After cleaning out the webs sufficiently to gain access to the perpendicular tunnels, I hit the power from the primary hatch control breaker to start the air conditioning system up before I went inside, so it would have fresh air when I got there.

I flipped it on, waited five minutes and then began to crawl down the horizontal access. There was a weird smell in the air. I heard what sounded like an odd buzzing insect talking. I was halfway in when I realized the Dr. Subatzo speech synth that runs on an interrupt in DOS for Sound Blaster was repeating "Air Quality Has Been Compromised. Check for fire or gas." over and over again. Which was odd, because I had programmed it to say it once, then wait for one minute before announcing it again. It was also supposed to sound the siren and flash the warning lights, which it wasn't doing. Great, I thought. I just installed that air quality sensor from Futurlec last year and had spent lots of time calibrating it by first reading the factory setting off it's internal EPROM first. This was a lot of trouble and took me ages. It had barely even been used and had failed already. At least that's the first idea that occurred to me.

Here's the chain of events that happened in order. I know it's hard to believe. I can hardly believe it myself. I assure you this is a true story.

1. The brand new 12 volt auto fan I installed in November of 2008 that turns on when the primary hatch is open (reed switch) to ventilate the shelter by expelling the stale air inside failed. I don't know what caused the failure. I suspect it had something to do with mold or lint in the motor. The motor had locked into this stuttering squeal.

2. The 12 volt motor had jammed and in stuttering it was sending a reverse alternating current that was probably well over 12 volts back over the line it shared in parallel with the led strip lights.

3. The led strip lighting is supposed to have reverse current protection, because the led bars can obviously only work on the correct polarity of current. Whatever the motor was sending out on the common line had fried the tiny cheap circuit in the led strip closest to the distributor box.

4. Now here is the worst part. This led bar, instead of just burning out like you would expect, was glowing an eerie orange color at one end and was heating up the from inside, cooking the plastic sleeve until it was practically molten. This off-gassing from the electrical fault hit me so hard when I dropped into the shelter I almost puked. It was toxic stuff. I'm not sure I could have crawled back out with that stuff pouring into the air behind me, it was real ugly vapor coming off it.

5. Luckily I had two things sitting in the comms station, by purest chance - a mouth/nose respirator hanging there and a backup fan still in the plastic.

I shut off the toggle switch that powered the lights (backup blue lights came on automatically), donned the respirator and quickly pulled the backup fan out and plugged it in. My eyes were stinging and I kept the respirator on while the fan exhausted these ugly fumes over the next ten minutes. Finally, it was okay to breathe again in there.

Dr. Subatzo's mech voice said "Air quality stabilized. Check intake to make certain it is safe."

Vault-OS was working perfectly. It's just that this stupid 12 volt motor had fried it's power supply with a similar AC current that ruined the LED strip lighting. This was causing it to soft reboot again and again, producing the same warning as soon as it was loaded. This cheap 12 volt motor had also blown the fuse on the box that housed the relay that switched the siren and warning lights.

I find this all incredible. I don't blame you if you don't believe this story. I could never have foreseen any of this happening in a million years. It just goes to show you that you can never be too paranoid about anything in a system like this. One malfunctioning 12 volt fan almost ended up killing me. It's possible I could have asphyxiated down there and my wife would not have known for hours.

This has given me a great deal of useful information about how to properly design a system like this. It makes me realize I need reverse current protection on any device that can be damaged or shorted this way. I also discovered that my strategy of putting almost everything on a parallel delivery cable was a good idea in a crisis - even though the first light on the rails above had shorted, the other six light bars were still on. If I'd suddenly found myself plunged into darkness amidst choking fumes, I could have had some serious problems. I might not have been able to find the respirator in the darkness down there.

Honestly, I am humbled by this experience and I plan to make good use of it in my rewrite of Vault-OS. It has helped me to understand how important it is to maintain solid state logs so that Vault-OS can discover it's context ... for example, being soft rebooted every ten seconds. It should be able to look in the log and see this to also notify the user of this fault in the system as well.

You can also understand how complex a sealed environment can be in terms of managing air quality. It's serious business. If you think it's enough to flip the fan switch on and off manually when you need clean air and trust your nose to warn you of smoke or gas, it isn't. You need other kinds of sentries, particularly digital ones, watching your back at all times to cover your human foibles and shortcomings. This experience is a perfect example why.

P.S. I have absolutely decided that my fifth iteration of lighting in the primary drums, the led bars, will have to be completely replaced. There is nothing on the original packaging mentioning that this $18.95 auto light turns into a deadly cyanide gas generator when a reverse current is applied. I think all of them will have to come down and be replaced solely by bayonet LED bulbs. This incident scared the hell out of me.

P.P.S. A master brain that wrapped Vault-OS from the outside like ThinkBoy would have figured out what was happening with proper sensors. I have not built this system yet, at least not the version I want. It does tell me that ThinkBoy will definitely require power management sensors to know what is happening at different junctures in the shelter. A heat sensor in the distributor boxes would give useful information as well. It would be able to determine that this was an electrical fault and would know it was not a coincidence that an air quality sensor was registering a change at the same time. This is what Thinkboy is intended to do when it is working.

P.P.P.S. Also, don't buy those 12 volt fans intended for automobile interiors. I think computer fans will be designed for much longer service with a lower failure rate.


Anonymous said...

I'm glad you're ok.

Anonymous said...

Damn Tex. This is going down in my vault notebook. Hope you're alright, I do appreciate you sharing everything you've learned with us.

One thing in particular that struck me is that you've not been into your vault in 10 months! I'd like to do a quick 5 minute check every month or so for structual integrity, water damage and a full system test.

Anonymous said...

Any motor you put in your circuit needs to be on isolated from the other circuits... At least a relay should power it and you need to filter back emf directly after the motor..

Use a big capacitor to absorb back emf and a diode on your power line to prevent change in polarity and a fuse near to the motor and another one near to the power source.

Also, with long lines running in your vault, you'll have issues with data lines coming from your sensors, you should filter any powered circuit in there using low pass / high pass filters (very easy to learn and to build) ..

Anonymous said...

Btw, talking about Datalines, maybe use optocouplers everywhere.. But I think I remember you writing about this....

Anonymous said...

And thirdly, be careful with motors, check the specs for stalling current, you should monitor the RPM using your OS and you should have overcurrent protection when the current goes over stall current..

Anonymous said...

Thank God it happened now, and not during Code Red. Maybe a long term test is prudent of all the systems...just turn them on and leave them on for weeks. You have an Angel looking over your shoulder, Bud!

Anonymous said...

you could also change your entry procedure to include a torch and hand-held respirator - these could be placed in a box externally for post long-period absence entry

Anonymous said...

Don't you hate it when the shtf when all your trying to do is get ready for when the shtf?

Damn, that's like crawling under your car to drain the oil and having a rattle snake slither up your pant leg.

Man..I am happy to hear you are ok.

Anonymous said...

any pics

Anonymous said...

Well Tex if you're busy you might as well put a gasmask in every room and corridor. Just in case.


Anonymous said...

merry christmas tex.

Anonymous said...

I believe you Tex. Infact, I would have considered it poetically apt for a man obsessed with his self-survival to be prematurely killed by the very thing he thought assured his safety.

God's grace is truly abounding.

Anonymous said...

I am glad you are ok. I've been following your blog and your postings on 2012 Forum for a while. Great insights. Here is my question for you. Do you still believe that it is possible to maintain a fully underground shelter in livable conditions for an extended period of time? Fresh air and humidity control are critical. You need your equipment in working coditions and uninterrupted electricity to power it. My main concern is the reliability of such a system. Even with a high degree of duplication and a stable environment it could be problematic. But in unpredictable environment (EMPs, magnetic reconnections, ashes in the air and on the ground, etc.) it is very hard to see how it will work.
Your opinion and feedback is highly appreciated.

Long Winter.

Anonymous said...