Tech team trapped in data centre as hypoxic gas flooded in. Again Welcome once more to “Who, me?”, in which we help Reg readers to unburden themselves by telling anonymised stories of big, bad, mistakes. This week meet “Luke” who told us that about 20 years ago he was in charge of disaster recovery “for an insurance company in San Francisco” and had “a series of ‘oh shit’ days.” Lue’s …
Banned for new installations, not to the point where it has to be ripped out of existing....
So I imagine a fair bit lying around!
Aren't they the BOFH's former colleagues "lying" around? You know, sales reps, beancounters, this week's boss?
You would have thought that the BOfH would have used up all his halon by now, and have been forced to recharge his systems with something a bit more ozone friendly. Though he's probably got a source down the pub for getting hold of illegal halon. Everyone likes a drop of the hard stuff, now and then...
You can't breath any [Halon] (that's the point) so it's the difference between NATO 5.56 and Russian 5.45 x 39.
Actually, no. Halon does not put fires out by displacing oxygen. It has some kind of funky chemical reaction which reduces the heat from the fire, which puts it out. IIRC, you can breathe the stuff for (IIRC) something like 15 minutes before you start getting a headache from it.
This post has been deleted by its author
CO2 is heavier than air, and is designed to displace the oxygen and starve the fire. So if you're a giraffe you should still find breathable air above the CO2 layer.
Halon is designed to bond with all the free oxygen it can find, and so should mean that there's nothing to breath anywhere - either for you or the fire.
So CO2 sort of might be safter, ish, maybe. Except that you can become unconsious when you're breathing air with more than 4% of it, if memory serves. It's definitely safer to be down the pub...
Ooopsie. Sorry about that. Well you learn something new every day - or unlearn something you'd remembered wrong. Just looked it up and I'd completely misremembered what halon does. It doesn't bind with all the oxygen around, it stops the fire's chemical reactions from working properly. So it's definitely safer than CO2, and can be breathed in low concentrations.
It was apparently first used in extinguishers and fire grenades in the first decade of the 20th century. I'd read mention of those before and sort of vaguely wondered what was in them.
It doesn't bind with all the oxygen around, it stops the fire's chemical reactions from working properly.
Correct. However, I once had a very interesting conversation with an expert in fire detection & suppression systems. He pointed out that while halon (or one of the modern replacements) does indeed stop the fire like that, if the source of heat is still there then you have big problems.
So (for example) you have a cable that's overloaded and would ordinarily be burning. Most modern materials used in electrical and IT stuff is "self extinguishing" - which means that it won't carry on burning once the energy source (heat) is removed. Incidentally, you can easily test this for (say) a bit of cable by holding it in a flame till it starts burning and then remove it from the flame - it should go out on it's own.
So now you have some PVC (where the C stands for Chloride, containing chlorine) still breaking down in the heat, but the free chlorine radicals cannot combine with the oxygen in the air because of the halon. Hmm, chlorine free radicals being stirred around (lots of fans remember) electrical equipment = lots of corrosion of exposed metalwork, especially connector contacts. So your kit isn't destroyed by the fire, but is rendered useless by the chlorine radicals attacking it.
The "self extinguishing" bit is a clue - if you remove the power then the fire stops of it's own accord. Hence by far the most effective (and cheapest) method is to just "switch it off", aka an EPO switch (linked to a fire detection system where one's fitted).
Of course, just "yanking the power" from a server room does have it's own downsides !
when I worked in airline business (in USA) until december 2007 halon was still used in cargo compartment. and as far as I can tell was still (in 2017) used there.
at same time handheld bottles and the like were NOT halon due to the regs.
" the dust that gets blown around is enough to kill a lot of kit"
Not heard that one, but one known problem is that the sound pressure/vibrations generated by the gas exiting the nossles at high speed is sufficient to wreck mechanical hard drives.
Ironically it's potentially possible to loose more data to the fire suppression going off, that if the fire had just been left to burn out.
Ironically it's potentially possible to loose more data to the fire suppression going off, that if the fire had just been left to burn out.
========================================================================
I doubt that.
I was once visiting a company whose business was selling computer systems engineered to run as long as your redundant power and cooling systems were up, including during hardware maintenance and replacement. My company was considering buying some of those systems.
Every part of the system was duplicated, at the least.
When I was there, they were down, and their fully independent mirrored drive systems were down, with major data loss.
One of the drives had caught fire... the smoke killed the rest of them.
Depends. Some types of Halon (there are several) are clean - they leave no residues - that became a requirement for their replacements as well exactly because cleaning or replacing expensive kits, and slow down the recovery, is not nice...
"Some types of Halon (there are several) are clean"
Halon generally works by chemically combining with whatever's burning to extinguish the flames.
It works well, anything near the fire has to be replaced. What you absolutely do not want in a data room is a dry powder extinguisher - something which got demonstrated to the H&S wonk who insisted on having one in there by letting one off in his office during a weekend when he wasn't around.
Whatever you use to extinguish a fire, anything near enough to the flames and heat has a good chance to need replacement - it's the effect on whatever is away enough from the flames and heat that makes a difference, and the procedure needed to purge the room and restore it to working condition after the damaged items (and the fire cause, of course) are removed.
" all the dust that gets blown around is enough to kill a lot of kit."
Yup. So it's worthwhile _insisting_ on a full deep clean of the entire room behind the ceiling tiles and under the raised floor before commissioning (and preferably before the cabling team moves in), then go over it again before actually trundling the racks in.
In a small (8+5 metre) server room I pulled out several vacuum cleaner loads of nasty grit, destroying the machine in the process. Better that, than that the expensive hardware.
The same room has a number of sticky floor pads and dust traps which are periodically cleaned out, as are the bottom of racks.
When was the last time YOUR server room was cleaned?
Reminds me of finding a dead bird in the ceiling void of a prep test kitchen at a food manufacturer.
Was doing a contract running CCTV cable, found a small sparrow(?) up there dry as a husk. Told the supervisor on duty and let them deal with it.
Anon because I don't want to ruffle any feathers...
Reminds me of the occasion (circa 1982 or 83) when Trent Poly in Nottingham relocated its Computer Room into a converted college Library over the summer vaccation. Kit being moved included the single-decker bus sized DECSYSTEM-20 timesharing mainframe, with associated free standing disk and tape drives, all of which required a specialised County-level DEC LCG Field Service Engineer to supervise.
Anyway, the relocation project was planned in detail and the lifting gear etc planned to turn up with the heavy gang with a limited amount of time and, of course, on a fairly tight budget. Time factors also quite critical as the old room had to be handed over to have asbestos cleared out.
So, said LCG engineer turns up day before the big move and comes in to check out the new computer room. Looking around he's generally satisfied with the room - and then he tells us to lift up some floor tiles to check the under-floor void. Suddently he turns round to the Head of Department and sundry others with him and says: "No way is that machine moving in here with that floor!". He bends down and wipes his finger across the real floor and it comes up covered in dust to demonstrate what he means.
Turns out that when the room was convered, the existing parquet flooring was sanded flat, cleaned and vaccumed and then left in that state.
Rapid discussions came up with a plan that the floor could be sealed by varnishing it. Shortly after that four or five of us are piling into a car and haring off to a nearby DIY store and buying up almost all of its stock of large tins of floor varnish and brushes. In the mean time others in the department are busy starting to lift all the floor tiles.
By now its mid-afternoon and M-day starts first thing the next morning.
Teams of members of the department work through the night - with the bulk of it being done by the head of Application Services - and the whole job is just about finished by dawn the next morning. Mind you, anybody spending too long in the room would be high as a kite for the next few days with the vapours given off by the drying varnish.
So, actually a happy ending to the story. All systems were moved during the planned window and we were able to get the service up in time to prepare for the new term. That was until several weeks into running in the new centre, when it was discovered that the "wrong type of air conditioning had been installed" and we had to have a kettle boiling in the room at times to increase the humidity to a level that worked for the kit ... but that's a story for another time!
I didn't quite see it, but I felt and heard the explosion of a flour mill (actually the grain elevator, but the mill disappeared when the elevator did) around three miles away. I hope no one allows that much dust build-up in their data center, but I've seen one that had to be close.
This post has been deleted by its author
And I think you're on to something.
What about designing and selling an actual BOFH/PFY game a la Cluedo?
The Boss in the Filing room with the Faulty CattleProd.
Sharon in the Company Canteen with the Listeriosis Viennas. :)
The Janitor in the Stairway with the PFY's patented slippery stuff.
Now I have a vision of Tim Curry as the BOFH. Or maybe Madeline Kahn.
Cracking film, by the way. Watch it if you haven't already. You're in for a treat.
At the computer centre I was working in during the 80's in the days of 1/2" reel tapes. We had 2 halon systems; a small system protecting the tape library and a much larger system protecting the remainder of the centre.
The tape library was only "manned" during the day shift with the tape librarian preparing the trolleys of tapes for the overnight processing and filling the "Off-Site" storage boxes.
One day (without notice or reason) the Halon in the tape library triggered. This was in the middle of the day when the Shift Supervisor was in the library working with the librarian checking the boxes to be sent "Off-Site". When the Halon went off the librarian and Shift Supervisor raced out of the room. To this day I swear that the librarian had the Shift Supervisor's foot prints up the middle of her back. I have never seen "Big Al" (the Shift Supervisor) move so fast.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
