back to article Tech team trapped in data centre as hypoxic gas flooded in. Again

Welcome once more to “Who, me?”, in which we help Reg readers to unburden themselves by telling anonymised stories of big, bad, mistakes. This week meet “Luke” who told us that about 20 years ago he was in charge of disaster recovery “for an insurance company in San Francisco” and had “a series of ‘oh shit’ days.” Lue’s …

  1. Waseem Alkurdi

    Hasn't halon been banned or something in the '90s?

    And isn't the BOFH's site the only one still going with halon?

    1. Giovani Tapini

      Re: Hasn't halon been banned or something in the '90s?

      Banned for new installations, not to the point where it has to be ripped out of existing....

      So I imagine a fair bit lying around!

      1. Korev Silver badge

        Re: Hasn't halon been banned or something in the '90s?

        Banned for new installations, not to the point where it has to be ripped out of existing....

        So I imagine a fair bit lying around!

        Aren't they the BOFH's former colleagues "lying" around? You know, sales reps, beancounters, this week's boss?

        1. Doctor Syntax Silver badge

          Re: Hasn't halon been banned or something in the '90s?

          "Aren't they the BOFH's former colleagues "lying" around?"

          And is there still enough Halon about to replenish after the BOFH's accidental (every single one of them) discharges?

        2. el_oscuro

          Re: Hasn't halon been banned or something in the '90s?

          I think I have seen the BOFH. Is this her?

          http://dilbert.com/strip/2010-02-06

    2. Anonymous South African Coward Bronze badge
      Pint

      Re: Hasn't halon been banned or something in the '90s?

      Upvoted for the BOFH reference.

      And have one (sans laxatives)

    3. I ain't Spartacus Gold badge

      Re: Hasn't halon been banned or something in the '90s?

      You would have thought that the BOfH would have used up all his halon by now, and have been forced to recharge his systems with something a bit more ozone friendly. Though he's probably got a source down the pub for getting hold of illegal halon. Everyone likes a drop of the hard stuff, now and then...

      1. Peter2 Silver badge

        Re: Hasn't halon been banned or something in the '90s?

        Like a CO2 flood, which is probably more lethal?

        1. Destroy All Monsters Silver badge

          Re: Hasn't halon been banned or something in the '90s?

          Like a CO2 flood, which is probably more lethal?

          You can't breath any (that's the point) so it's the difference between NATO 5.56 and Russian 5.45 x 39.

          1. Peter2 Silver badge

            Re: Hasn't halon been banned or something in the '90s?

            You can't breath any [Halon] (that's the point) so it's the difference between NATO 5.56 and Russian 5.45 x 39.

            Actually, no. Halon does not put fires out by displacing oxygen. It has some kind of funky chemical reaction which reduces the heat from the fire, which puts it out. IIRC, you can breathe the stuff for (IIRC) something like 15 minutes before you start getting a headache from it.

          2. Anonymous Coward
            Anonymous Coward

            Re: Hasn't halon been banned or something in the '90s?

            the difference between NATO 5.56 and Russian 5.45 x 39

            206.99? Of course if you consider the units things get a bit messy.

            1. Destroy All Monsters Silver badge
              Trollface

              Re: Hasn't halon been banned or something in the '90s?

              206.99? Of course if you consider the units things get a bit messy.

              You must be from a country with gun control.

              1. This post has been deleted by its author

        2. I ain't Spartacus Gold badge

          Re: Hasn't halon been banned or something in the '90s?

          CO2 is heavier than air, and is designed to displace the oxygen and starve the fire. So if you're a giraffe you should still find breathable air above the CO2 layer.

          Halon is designed to bond with all the free oxygen it can find, and so should mean that there's nothing to breath anywhere - either for you or the fire.

          So CO2 sort of might be safter, ish, maybe. Except that you can become unconsious when you're breathing air with more than 4% of it, if memory serves. It's definitely safer to be down the pub...

          1. I ain't Spartacus Gold badge

            Re: Hasn't halon been banned or something in the '90s?

            Ooopsie. Sorry about that. Well you learn something new every day - or unlearn something you'd remembered wrong. Just looked it up and I'd completely misremembered what halon does. It doesn't bind with all the oxygen around, it stops the fire's chemical reactions from working properly. So it's definitely safer than CO2, and can be breathed in low concentrations.

            It was apparently first used in extinguishers and fire grenades in the first decade of the 20th century. I'd read mention of those before and sort of vaguely wondered what was in them.

            1. Peter2 Silver badge

              Re: Hasn't halon been banned or something in the '90s?

              No probs. Kind of easy to assume the stuff is lethal given that the BOFH uses it as a mass murder weapon!

              Kudo's for checking and realising (and admitting) you were wrong as well. Increasingly rare these days, unfortunately :)

            2. SImon Hobson Bronze badge

              Re: Hasn't halon been banned or something in the '90s?

              It doesn't bind with all the oxygen around, it stops the fire's chemical reactions from working properly.

              Correct. However, I once had a very interesting conversation with an expert in fire detection & suppression systems. He pointed out that while halon (or one of the modern replacements) does indeed stop the fire like that, if the source of heat is still there then you have big problems.

              So (for example) you have a cable that's overloaded and would ordinarily be burning. Most modern materials used in electrical and IT stuff is "self extinguishing" - which means that it won't carry on burning once the energy source (heat) is removed. Incidentally, you can easily test this for (say) a bit of cable by holding it in a flame till it starts burning and then remove it from the flame - it should go out on it's own.

              So now you have some PVC (where the C stands for Chloride, containing chlorine) still breaking down in the heat, but the free chlorine radicals cannot combine with the oxygen in the air because of the halon. Hmm, chlorine free radicals being stirred around (lots of fans remember) electrical equipment = lots of corrosion of exposed metalwork, especially connector contacts. So your kit isn't destroyed by the fire, but is rendered useless by the chlorine radicals attacking it.

              The "self extinguishing" bit is a clue - if you remove the power then the fire stops of it's own accord. Hence by far the most effective (and cheapest) method is to just "switch it off", aka an EPO switch (linked to a fire detection system where one's fitted).

              Of course, just "yanking the power" from a server room does have it's own downsides !

    4. dmacleo

      Re: Hasn't halon been banned or something in the '90s?

      when I worked in airline business (in USA) until december 2007 halon was still used in cargo compartment. and as far as I can tell was still (in 2017) used there.

      at same time handheld bottles and the like were NOT halon due to the regs.

  2. Phil Kingston

    Seen it in plenty of places but never seen it used. I seem to remember someone telling me that if it goes off, all the dust that gets blown around is enough to kill a lot of kit. So you may as well have let it burn.

    One day they'll let me hit the button.

    1. Phil W

      " the dust that gets blown around is enough to kill a lot of kit"

      Not heard that one, but one known problem is that the sound pressure/vibrations generated by the gas exiting the nossles at high speed is sufficient to wreck mechanical hard drives.

      Ironically it's potentially possible to loose more data to the fire suppression going off, that if the fire had just been left to burn out.

      1. Anonymous Coward
        Anonymous Coward

        Ironically it's potentially possible to loose more data to the fire suppression going off, that if the fire had just been left to burn out.

        ========================================================================

        I doubt that.

        I was once visiting a company whose business was selling computer systems engineered to run as long as your redundant power and cooling systems were up, including during hardware maintenance and replacement. My company was considering buying some of those systems.

        Every part of the system was duplicated, at the least.

        When I was there, they were down, and their fully independent mirrored drive systems were down, with major data loss.

        One of the drives had caught fire... the smoke killed the rest of them.

        1. SuperGeek

          I can imagine. Smoke being sucked into filters of drives only gets filtered till the filter clogs then the drive suffocates and the head crashes due to loss of air pressure in the drive.

      2. ~chrisw
        Coffee/keyboard

        Can confirm. Observed a recent event where an erroneous Argon release caused mass spinny disk crashes due to 200 PSI worth of acoustic shock. This, and the restoration work, sounded painful. If only someone had used the hold-off button...

    2. Anonymous Coward
      Anonymous Coward

      "if it goes off, all the dust that gets blown around is enough to kill a lot of kit."

      Depends. Some types of Halon (there are several) are clean - they leave no residues - that became a requirement for their replacements as well exactly because cleaning or replacing expensive kits, and slow down the recovery, is not nice...

      1. Alan Brown Silver badge

        Re: "if it goes off, all the dust that gets blown around is enough to kill a lot of kit."

        "Some types of Halon (there are several) are clean"

        Halon generally works by chemically combining with whatever's burning to extinguish the flames.

        It works well, anything near the fire has to be replaced. What you absolutely do not want in a data room is a dry powder extinguisher - something which got demonstrated to the H&S wonk who insisted on having one in there by letting one off in his office during a weekend when he wasn't around.

        1. Anonymous Coward
          Anonymous Coward

          "anything near the fire has to be replaced"

          Whatever you use to extinguish a fire, anything near enough to the flames and heat has a good chance to need replacement - it's the effect on whatever is away enough from the flames and heat that makes a difference, and the procedure needed to purge the room and restore it to working condition after the damaged items (and the fire cause, of course) are removed.

    3. Alan Brown Silver badge

      " all the dust that gets blown around is enough to kill a lot of kit."

      Yup. So it's worthwhile _insisting_ on a full deep clean of the entire room behind the ceiling tiles and under the raised floor before commissioning (and preferably before the cabling team moves in), then go over it again before actually trundling the racks in.

      In a small (8+5 metre) server room I pulled out several vacuum cleaner loads of nasty grit, destroying the machine in the process. Better that, than that the expensive hardware.

      The same room has a number of sticky floor pads and dust traps which are periodically cleaned out, as are the bottom of racks.

      When was the last time YOUR server room was cleaned?

      1. Chloe Cresswell Silver badge

        Is this where people look up the time Cambridge Uni had a halon dump in their mainframe room?

        And also found out at the same time there was plumbers grease in the halon vent pipes.. the hard way.

      2. Captain Scarlet Silver badge

        "When was the last time YOUR server room was cleaned?"

        Erm 2009, the henry hoover found a dead rodent which meant lots of paperwork, never again

        1. Destroy All Monsters Silver badge

          Erm 2009, the henry hoover found a dead rodent which meant lots of paperwork, never again

          Shoot, Shovel and Shut Up, but without the Shoot part is the recommendation here.

          1. Anonymous Coward
            Anonymous Coward

            RE: Shoot, Shovel and Shut Up

            Reminds me of finding a dead bird in the ceiling void of a prep test kitchen at a food manufacturer.

            Was doing a contract running CCTV cable, found a small sparrow(?) up there dry as a husk. Told the supervisor on duty and let them deal with it.

            Anon because I don't want to ruffle any feathers...

        2. J.G.Harston Silver badge

          Hoover don't make Henries, they make Hoovers. Numatic make Henries.

          1. phuzz Silver badge

            Numatic make hoovers, but not Hoovers. One's a trade name, the other is a generic name, in the UK at least.

            I'm pretty sure the name 'Henry' was chosen to be alliterative with 'hoover' anyway.

      3. Tony Gathercole ...
        Happy

        re - When was the last time YOUR server room was cleaned?

        Reminds me of the occasion (circa 1982 or 83) when Trent Poly in Nottingham relocated its Computer Room into a converted college Library over the summer vaccation. Kit being moved included the single-decker bus sized DECSYSTEM-20 timesharing mainframe, with associated free standing disk and tape drives, all of which required a specialised County-level DEC LCG Field Service Engineer to supervise.

        Anyway, the relocation project was planned in detail and the lifting gear etc planned to turn up with the heavy gang with a limited amount of time and, of course, on a fairly tight budget. Time factors also quite critical as the old room had to be handed over to have asbestos cleared out.

        So, said LCG engineer turns up day before the big move and comes in to check out the new computer room. Looking around he's generally satisfied with the room - and then he tells us to lift up some floor tiles to check the under-floor void. Suddently he turns round to the Head of Department and sundry others with him and says: "No way is that machine moving in here with that floor!". He bends down and wipes his finger across the real floor and it comes up covered in dust to demonstrate what he means.

        Turns out that when the room was convered, the existing parquet flooring was sanded flat, cleaned and vaccumed and then left in that state.

        Rapid discussions came up with a plan that the floor could be sealed by varnishing it. Shortly after that four or five of us are piling into a car and haring off to a nearby DIY store and buying up almost all of its stock of large tins of floor varnish and brushes. In the mean time others in the department are busy starting to lift all the floor tiles.

        By now its mid-afternoon and M-day starts first thing the next morning.

        Teams of members of the department work through the night - with the bulk of it being done by the head of Application Services - and the whole job is just about finished by dawn the next morning. Mind you, anybody spending too long in the room would be high as a kite for the next few days with the vapours given off by the drying varnish.

        So, actually a happy ending to the story. All systems were moved during the planned window and we were able to get the service up in time to prepare for the new term. That was until several weeks into running in the new centre, when it was discovered that the "wrong type of air conditioning had been installed" and we had to have a kettle boiling in the room at times to increase the humidity to a level that worked for the kit ... but that's a story for another time!

    4. W4YBO

      "...all the dust that gets blown around..."

      I didn't quite see it, but I felt and heard the explosion of a flour mill (actually the grain elevator, but the mill disappeared when the elevator did) around three miles away. I hope no one allows that much dust build-up in their data center, but I've seen one that had to be close.

  3. Pen-y-gors

    New version of Cluedo

    I think an IT-themed Cluedo could be fun

    "The PFY with the Halon in the ops-room"

    1. This post has been deleted by its author

    2. Anonymous South African Coward Bronze badge

      Re: New version of Cluedo

      And I think you're on to something.

      What about designing and selling an actual BOFH/PFY game a la Cluedo?

      The Boss in the Filing room with the Faulty CattleProd.

      Sharon in the Company Canteen with the Listeriosis Viennas. :)

      The Janitor in the Stairway with the PFY's patented slippery stuff.

      1. Martin

        Re: New version of Cluedo

        And presumably it is the Beancounter who is found dead in the cellar.

        1. Doctor Syntax Silver badge

          Re: New version of Cluedo

          "the Beancounter who is found dead in the cellar."

          Isn't the point of a demise in the cellar that the body's never found, it just disappears in a roll of carpet to a landfill north of Leeds?

          1. Anonymous Coward
            Anonymous Coward

            Re: New version of Cluedo

            > to a landfill north of Leeds?

            What, Scotland?

            :-)

          2. bpfh

            Re: New version of Cluedo

            Aha, the joker card - no body no murderer :)

      2. Allan George Dyer

        Re: New version of Cluedo

        But it is always the BOFH/PFY that does it...

        OTOH, the game could be to discover who gets the blame...

        1. Anonymous South African Coward Bronze badge

          Re: New version of Cluedo

          But it is always the BOFH/PFY that does it...

          OTOH, the game could be to discover who gets the blame...

          Something like that...

          1. Anonymous Coward
            Anonymous Coward

            Re: New version of Cluedo

            I figured you were describing a game of identify the body.

      3. Alan Brown Silver badge

        Re: New version of Cluedo

        "The Janitor in the Stairway with the PFY's patented slippery stuff."

        And PFY shouting "boo"

    3. Qwelak

      Re: New version of Cluedo

      Already been done ....

      http://www.theregister.co.uk/2005/11/18/bofh_2005_episode_31/

    4. David 132 Silver badge
      Thumb Up

      Re: New version of Cluedo

      Now I have a vision of Tim Curry as the BOFH. Or maybe Madeline Kahn.

      Cracking film, by the way. Watch it if you haven't already. You're in for a treat.

      1. Swarthy

        Re: New version of Cluedo

        Tim Curry as the BOFH, Madeline Kahn as Gina.

        Or a reasonable facsimile thereof, it has been over 30 years.

  4. Oengus

    Oops

    At the computer centre I was working in during the 80's in the days of 1/2" reel tapes. We had 2 halon systems; a small system protecting the tape library and a much larger system protecting the remainder of the centre.

    The tape library was only "manned" during the day shift with the tape librarian preparing the trolleys of tapes for the overnight processing and filling the "Off-Site" storage boxes.

    One day (without notice or reason) the Halon in the tape library triggered. This was in the middle of the day when the Shift Supervisor was in the library working with the librarian checking the boxes to be sent "Off-Site". When the Halon went off the librarian and Shift Supervisor raced out of the room. To this day I swear that the librarian had the Shift Supervisor's foot prints up the middle of her back. I have never seen "Big Al" (the Shift Supervisor) move so fast.

    1. Alan Brown Silver badge

      Re: Oops

      "One day (without notice or reason) the Halon in the tape library triggered. "

      And THAT is my biggest worry about the bloody things.

      Honestly. I've been in mountaintop "temporary" installations where the halon system was a tank bolted to the ceiling with a mechanical thermal trip (like the heat detectors in fire aoarm systems) and still had to deal with the aftermath of the bloody thing having gone off for no apparent reason with noone around for twenty miles whilst the buillding was 4 foot deep in snow.

    2. Kubla Cant

      Re: Oops

      Maybe it's the font, maybe it's my eyes, but when I read the words "Shift Supervisor" that many times it transmutes into something that must be one of the worst jobs in the world.

      1. MaltaMaggot

        Re: Oops

        ... likewise, double-take on your handle there @Kubla...

        1. This post has been deleted by its author

  5. Mystic Megabyte
    Flame

    Not good stuff!

    A pal had left a tray of cooking oil in the oven of his coal burning stove. Because the oven door wasn't air-tight it kept igniting, using up all the oxygen, extinguishing and then re-igniting when enough air had been drawn back in. You could see the flashes through the gap in the seal. His bright idea was to chuck some earth on it, but every time he opened the door it ignited big time.

    I gave it about a half a second squirt with my elderly halon fire extinguisher, the fire went out but the top half of the room was immediately filled with white fumes. Guessing that these fumes were not very pleasant we crawled out on our hands and knees and didn't go back for quite a while.

    1. Anonymous South African Coward Bronze badge

      Re: Not good stuff!

      Gives new meaning to "Vaping Halon".

      And, yup, it's a BOFH eppysode.

    2. The First Dave

      Re: Not good stuff!

      So why didn't you just leave it to burn out naturally?

  6. Rich 11

    Have you ever breathed halon?

    Yes, but I swear I never inhaled.

    1. J. Cook Silver badge
      Go

      Re: Have you ever breathed halon?

      While I haven't breathed in Halon, I have inhaled some of the dry chemical from an 80's vintage ABC extinguisher. I learned a few good lessons that day:

      1. Don't repeatedly toggle a circuit breaker for no good reason

      2. When putting out the switch that caught fire from #1 that's in a closet, consider not using a vintage 80's ABC 7 pound dry chemical extinguisher and use other means, like a large towel and smothering the flames instead.

      3. Also when putting out the switch that caught fire from #1 that's in a closet using a vintage 80's ABC 7 pound dry chemical extinguisher, stand back a bit.

      4. When pulling the activation (safety) pin on an 80's vintage ABC dry chemical extinguisher, hang onto the pin- You'll want it later.

      5. When triggering the fire extinguisher, do not open the valve all the way initially.

      6. Hold one's breath when triggering the extinguisher to not breath in the inevitable back-blast from slamming the trigger valve to full open and filling the closest full of dry chemical at full velocity. And look away a bit.

      I'm pretty sure that's why I still have a bit of a cough, even 20 odd years later. And we never did find that damned pin.

      1. Anonymous Coward
        Anonymous Coward

        Re: Have you ever breathed halon?

        Those red devices were meant to lure commies to their death, should they ever decide to paratroop down onto suburban family houses.

        It might be the case that Philip K. Dick wrote something about that.

    2. Kernel

      Re: Have you ever breathed halon?

      Aah, memories of following my refrigeration engineer father around as a nipper and the heady aroma of Freon12 that's escaped through a small gap at the end of a red hot, glowing copper pipe - nothing clears the sinuses quite the same, unless it's the hydrogen sulphide gas that was used as a refrigerant in 1950's era domestic fridges.

  7. Nick Kew

    Reverting to type?

    Wasn't the monday column advertised as people 'fessing up to their own blunders? Or at the very least, the protagonist should have unwittingly demonstrated a fault, as in "my fat arse triggered stupidly-placed big red switch".

    Your Luke doesn't appear to qualify. He was there when something happened is an anecdote, not a confession!

    1. This post has been deleted by its author

      1. Nick Kew
        Pint

        Re: Reverting to type?

        I don't see where fabrication comes into it. True, embellished, or false, the story stands on its own to entertain us and provoke "that reminds me" anecdotes from the readers. And I agree, of course many of the best anecdotes appear in the comments.

        That is, however, orthogonal to my point about this "Luke" failing to qualify under Monday's already-loose criteria, by virtue of the story containing no suggestion that he was to blame.

        OK, OK, I'll join you in that pint. Raised to the several comments I've upvoted on this page.

        1. I ain't Spartacus Gold badge

          Re: Reverting to type?

          Who left a ladder precariously propped by the door (plus emergency buttons)?

          1. Anonymous South African Coward Bronze badge

            Re: Reverting to type?

            Who left a ladder precariously propped by the door (plus emergency buttons)?

            The PFY?

          2. Nick Kew

            Re: Reverting to type?

            Who left a ladder precariously propped by the door (plus emergency buttons)?

            No suggestion in the story as to who that was. Hence, no blame to protagonist. At least, no direct blame.

            But re-reading, it says he was in charge of disaster recovery. So I guess he really should have had an eye on disaster prevention, and in turn considered Elfin Safety matters before the event. So on reflection I guess he does deserve blame for setting the scene: the original UPS with no-one responsible, and the culture that left a ladder sitting around.

  8. Gordon Pryra

    I was doing a Windows migration in hemel hempstead at the time Buncefield oil depot went up.

    The server room shifted half an inch and all those loverly 2 inch thick roof tiles ended up on my head and a number of the stand alone servers.

    That nice "snow" stuff went off, and once I found the door, I realised that the frame no longer fit it.

    It was kind of fun kicking the door out :)

    Luckily for me and the migration all that was running were some final data syncs, so although the RSA server had a 4 inch dent in the case with a tile sticking out, no external access was needed and I got a few weeks off work fully paid while they waited to get access to the building again :)

    1. Korev Silver badge
      Pint

      That sounds pretty scary!

      Have a therapeutic pint -->

    2. Manolo
      Mushroom

      Buncefield boom

      "I was doing a Windows migration in hemel hempstead at the time Buncefield oil depot went up."

      Sunday morning 5 AM?

      1. Nunyabiznes
        Joke

        Re: Buncefield boom

        "I was doing a Windows migration in hemel hempstead at the time Buncefield oil depot went up."

        Sunday morning 5 AM?

        Windows migrations: Something best done in the dark of night where no one can see you.

      2. Gordon Pryra

        Re: Buncefield boom

        4:50 am or something.

        emergency services couldn't get in because the locals decided that it was cool to have a street party and block the fire trucks....

  9. Nifty Silver badge

    Hydrogen peroxide

    This does remind me of an office I worked in, building was a kind of factory hall. We started noticing a 'sharp' smell that no-one could quite place. Occasionally a very strong whiff of it would hang around certain spots of the building. It became quite concerning. Management ignored the issue until we researched/guessed that it was a biogas and that was nasty stuff to breathe and highly flammable too.

    The cause? They'd raised the roof some years back by making the building walls taller. Trouble is, the sewage breather stack was left inside the building. Not a problem until a combination of cold weather and a change in the loo cleaner to a strong disinfectant had caused anaerobic digestion to begin in the cess pit.

  10. diver_dave

    Buncefield

    I was living in Watford at the time...

    Felt the windows move as the Munro effect manifested itself.

    That said, I have a friend who had just moved to Adeyfield (sp?) and all they heard was

    a whoosh, then all the leaves were cleared from the street. No damage at all!

    Very scary morning for all, Kudos to the FRS who had no idea how bad it was when they went in.

    DaveA

    1. John Sager

      Re: Buncefield

      Our old boiler used to light up with a bit of a thump, and that morning my wife remarked on how noisy the boiler was becoming. Only later did I realise it was the bang from Buncefield, attenuated a bit over here east of Ipswich!

      1. I ain't Spartacus Gold badge

        Re: Buncefield

        Living 20 miles away I heard a really loud sharp metallic ringing noise, that sounded exactly like hitting the degauss button on one of those chunky old 19" monitors. As they'd been replaced with LCDs, it was only the telly downstairs was still old-school. So I zoomed down to the sitting room, wondering if it had exploded or something.

        An impressively loud noise.

        1. TRT Silver badge

          Re: Buncefield

          Thought one of the kids had fallen out of their bunk bed. Next door's loft hatch fell in to their bedroom. And that was 12 miles away.

          1. Anonymous Coward
            Anonymous Coward

            Re: Buncefield

            In Waltloo, Pretoria we also have a fuel depot (though at a smaller scale). We lived near it at one stage.

            It is things like Buncefield etc which make me wonder how long will it take before the one at Waltloo also decide to do the biggenbang thing (which I hope will never happen).

            Glad we moved and is living somewhere far away from that depot...

          2. Stoneshop

            Re: Buncefield

            Thought one of the kids had fallen out of their bunk bed. Next door's loft hatch fell in to their bedroom. And that was 12 miles away.

            I was living in Enschede when the fireworks storage went up, and in my street there was no damage worth mentioning despite being a mere 1.6km away in a straight line; elsewehere there had been windows blown out at 10km and more.

            1. gryphon

              Re: Buncefield

              If I remember correctly there was the fireworks factory, Grolsch and then Polaroid. A VERY good thing Polaroid didn't go up with the amount of toxic chemicals that were used there.

    2. Anonymous Coward
      Anonymous Coward

      Re: Buncefield

      I'll take your Buncefield and raise it with Flixborough. Living in Scunthorpe at the time, the bang is one of my earliest memories.

      1. diver_dave
        Mushroom

        Re: Buncefield

        *Age Warning*

        I remember Flixborough from my dim and distant youth.

        The odd thing with these kind of incidents that pressure / blast wave propagation is a bit like the three body problem on steroids! https://en.wikipedia.org/wiki/Three-body_problem

        Waves tend to do some very strange things as you get reinforcements and waves cancelling each other out. Some can be predicted such as directing force for demolitions, however, there is always a risk factor.

        IIRC. Buncefield was very directional due to the nature of the initial blast and then the secondary's changing the blast wave following the initial FAE.

        Although even with some predictability, it's still Russian Roulette. Unless of course you are in a lead lined fridge!

        FORD:

        Look I hate to say this lads, but this thing really is going to blow up!

        ZAPHOD:

        Ok, oo-kay.

        [Massive explosion! ! !]

        1. TRT Silver badge

          Re: Look I hate to say this lads, but this thing really is going to blow up!

          Simple. It wasn't a fuel storage depot, it was a hyperspatial field generator. As it overheated it blew a hole through the space-time continuum and you dropped through like a stone through a wet paper bag…. I hate wet paper bags.

  11. Crisp
    Flame

    If something looks like it's about to catch fire...

    Then surely pulling the halon switch is the right thing to do?

    1. I ain't Spartacus Gold badge

      Re: If something looks like it's about to catch fire...

      Depends. If it looks like it's a small fire, you might want to try and fight it with a hand extinguisher first? Depending on what's near it. A bit of dry powder can come in very handy now and again for little fires - as that makes a mess in a well defined area. Whereas foam, water and CO2 have downsides and often spread the mess. I remember watching someone try to put out an alcohol fire with a CO2 extinguisher, and basically spread it all round the room.

      1. Alan Brown Silver badge

        Re: If something looks like it's about to catch fire...

        "I remember watching someone try to put out an alcohol fire with a CO2 extinguisher, "

        It can be done, but you have to know what you're doing.

        Fortunately server rooms tend not to have such things as liquid fuel fires.

      2. Sebastian P.

        Never do this

        No. Never. Unless you have received proper professional training, have the right equipment (including breathing apparatus), and it's explicitly in your job description to fight fires.

        The only exception from that rule is if you need to do that to save another person. Otherwise, just get away and sound the alarm. No equipment is worth more than your health or your life.

        1. Red Bren
          Joke

          Re: Never do this

          @Sebastian P.

          "No equipment is worth more than your health or your life."

          You've never bought a macbook pro then...

        2. MonkeyCee

          Re: Never do this

          "No. Never. Unless you have received proper professional training, have the right equipment (including breathing apparatus), and it's explicitly in your job description to fight fires."

          Don't know why you're getting down votes.

          If there is a fire, get yourself and others to safety and call the experts. The only point in any of that apparatus is to get you out of the joint alive.

          I've done basic fire fighting training, as a volunteer fire fighter and as part of a cookery course. So putting out minor and contained fires if practical is worthwhile doing. If you can smother it with a fire blanket then it's probably OK. If you have to use an extinguisher then call in the fire brigade to do a heat check afterwards.

          My fires at work is alas not tech related, but from my time in kitchens. Deep fat fryer went up, I tossed *my* fire blanket over it (OCD chef kit), killed the gas and power and called the fire brigade. The exec flipped his lid, since service was about to start, and kept trying to remove the fire blanket.

          After the lads showed up, not only did the oil re-ignite when they took the blanket off (in a controlled fashion), but after they had put it out, they found that there were at least two smoldering sites in the ventilation.

          That led to the building being evacuated and a fair amount of mess. No food served that night, bunch of stuff in the chillers had to be binned. Exec very pissed at me, but the owner was happy that the place hadn't burnt down.

          IMHO a fire blanket per floor (or person) is a sound investment. It's far more flexible than an extinguisher for making an exit (you wrap it around you), it'll deal with most smotherable fires, it doesn't lose pressure or need refills, is less reliant on technique and it is less suitable for practical jokes.

          1. Mark 85

            Re: Never do this

            Don't know why you're getting down votes.

            Probably from management or accountant types. Equipment is always worth more than people and with the paperwork, harder to do. A manager I once worked for said: "You people are a dime a dozen. I can get rid of you at any time and have your replacement here in a couple of hours.". To say he was a jackass would be an understatement.

          2. Alan Brown Silver badge

            Re: Never do this

            "Exec very pissed at me, but the owner was happy that the place hadn't burnt down."

            And that's about the point that the exec should be bouncing along the road on his arse and blacklisted from the industry, with a bunch of H&S writeups against his name.

            The fire service have the power to do that. Criminal charges for interfering with fire suppression (ie: attempted arson) have a career limiting effect.

        3. I ain't Spartacus Gold badge

          Re: Never do this

          Sebastian P.,

          You do need to have had some training. As you can make things worse with fire extinguishers if you don't know what you're doing. And you also need to raise the alarm beforehand - unless you're sure you can put the fire out in a couple of seconds (e.g. smothering a small one with a fire blanket).

          But if you've got a small fire in a contained server area, that has fire-suppression gear installed - then there's a small window of time when you can try less drastic measures before hitting the big red switch.

          You can put out a small petrol fire with a water extinguisher - temporarily - if you know what you're doing. But don't try. CO2 isn't much better for liquid fires, but foam, powder and blankets are good.

          You should always look to your own safety, and the safety of others, first. That means getting the alarm raised, people out, and the professionals called in straight away. But with a small fire, you can often tackle it perfectly safely - but if you're not sure then by all means run for the hills and wait for the cavalry. They're trained for it, after all.

          1. Crisp

            Re: Never do this

            I don't know about you guys, but I've definitely learned something new today!

          2. diver_dave

            Re: Never do this

            Agree completely...

            However for shits and giggles the fire marshal course in Malta cannot be beaten.

            You try: Dry powder, CO2, Water and Inert gas extinguishers. A fire blanket on a real oil pan fire. Then AFTER lunch..... They set a wood fire in a cargo container and get three of you to crack the door carrying a hose. Loads of fun(in a controlled situation) but shows exactly what flash over looks like.

            Possible the best course I have ever been on.

            DaveA

            1. DropBear
              Joke

              Re: Never do this

              Wait, now I'm confused - what you're supposed to do with a fire isn't simply putting it next to the rest of the fire...?

  12. Paul Cooper

    A real fire risk

    I once worked in a place that was the offsite store for many large organizations. Not just tapes - paper records of all sorts, too, as this was in the late 70s. The data were all VERY high-value things (though there was a serious temptation to walk down one aisle with a big magnet, as it held the records of a well-known sender of junk mail!). The warehouse must have been one of the most inflammable places in the area! It was protected by a MASSIVE halon system; the warehouse was just that - a warehouse with a vast volume, filled with racks full of paper many meters high, so the volume of Halon required was enormous - I recall cylinders taller than me, and many of them. I remember that we were solemnly warned that if the fire alarm went off, we had ten seconds to get out. I was on the call-out list for night-time alarms, and the few times I needed to attend, I was fervently hoping it wasn't a fire - which it never was! The worst was a piece of loose cardboard setting off the motion sensors (which were cutting edge, state-of-the-art intruder detection in those days), and the police insisted on being in first - I certainly didn't argue!

    1. IanRS

      Re: A real fire risk

      As a teenager, so a long time ago, I had a Saturday job in a clothing chain warehouse. Four floors (with perforated metal grid flooring, allowing for good ventilation / fire spreading, and causing vertigo for those working on the upper floors) full of cardboard, plastic and fabric. The safety briefing for new starters stated that in the case of a fire, the fire crew would come in and try to rescue anybody known to be trapped inside, but otherwise would just keep their distance and let it burn.

    2. Destroy All Monsters Silver badge
      Windows

      Re: A real fire risk

      The warehouse must have been one of the most inflammable places in the area! It was protected by a MASSIVE halon system

      Yeah, sometimes the fire suppression systems are bit .. let's say .. underperforming.

      1996-05-06: Credit Lyonnais suffers further blow as fire guts headquarters

      The fire is the latest blow to the troubled institution, which suffered record losses in 1994 and was bailed out by the government.

      I think no-one really ever found out where 20 billion franc français went exactly.

      It was a sign of things to come.

  13. Wolfclaw

    Working inside the computer room and you didn't switch the gas supression system to manual, you all need to be sacked for H&S breach !

    1. TRT Silver badge

      Ours has a 30 second stand-off.

      1. drand

        > Ours has a 30 second stand-off.

        Our inert gas system does too, plus it has a seven-segment display on the outside of the room counting down the seconds. It's gone off twice (both times over the weekend, with no-one about) but I'm waiting for the day it goes off when I'm in there and I can casually stroll out with one second left on the clock..

      2. Alan Brown Silver badge

        "Ours has a 30 second stand-off."

        So does ours - which you can postpone, but not cancel.

        Stupidity in action. Once the system's triggered (for whatever reason) you can't UN-trigger it.

        1. Antron Argaiv Silver badge
          Mushroom

          IIRC (it was many years ago), ours had a keyswitch. It was normally left on AUTO, but there was an OFF position.

          And the key was always left in the switch.

      3. Mystereed

        30 Seconds to comply...

        I did some work once in a big data centre with a 30 second warning before 10 tons of CO2 would be released.

        We were told if the alarm went off, don't walk to the exit - run as fast as you can, if the underfloor CO2 horns started blasting out the gas, the floor will resemble a Raiders of the Lost Ark scene, with tiles lifting up.

        Luckily I was with someone when the lights on one end wall started flashing and a bell rang - it was just to let the site engineer know his desk phone was ringing in his office outside the hall.

        1. Anonymous South African Coward Bronze badge

          Re: 30 Seconds to comply...

          Luckily I was with someone when the lights on one end wall started flashing and a bell rang - it was just to let the site engineer know his desk phone was ringing in his office outside the hall.

          Downloaded some brownware when that happened? :)

          1. TRT Silver badge

            Re: casually stroll out with one second left on the clock

            After that it's an automatic lock in, and it'll cost you a crystal to get you out again.

  14. Andrew Moore

    Was working in a server room of a government dept. The fire alert kept beeping and then shutting off. I reported this many times (the problem persisted for nearly a year) and was basically told to ignore it. Which was fine, until the day the Halon Deploy Klaxon went off. Once the klaxon went off, you were supposed to have 10 seconds to clear the area before the halon deployed. I don't think I got one second. I hit the doors at speed because they were supposed to electronically lock. Thankfully, the installers fucked that one up too and the lock didn't engage. Spent the next thirty minutes waiting for my hearing to return and picking bits of ceiling tile out of my hair.

    The kicker- the company that installed the system tried to blame it on me, claiming I was smoking in the server room. I pointed out that I didn't smoke and hoped that they had proof to back up their accusation...

  15. IGnatius T Foobar ✅

    No one uses halon anymore.

    No one uses halon anymore. It's been retired for decades. In even an old-ish data center you're going to find fire suppression agents like FM-200, which are (relatively) safe to be exposed to. No one is going to die ... at least not until the PHB kills someone when he finds out how much it costs to recharge the canisters. That stuff is expensive.

    1. Alan Brown Silver badge

      Re: No one uses halon anymore.

      "No one is going to die "

      ALL of the modern systems work by reducing the oxygen content of the room below 12-14% (vs 19-20% in normal atmospheric conditions). Think of it as an instant trip to 15-18,000 feet without the pressure reduction.

      At that point most people are going to be unconscious and anyone overweight/unfit or hypertensive runs a high risk of dying - the risk of pulmonary/cerebral edemas doesn't go away immediately after exposure either.

  16. JeffyPoooh
    Pint

    UPS <- a root cause

    A wise person once wrote...

    You’re concerned about your family’s safety. So you get a guard dog. The dog costs a fortune. It immediately poops on the floor. Then it chews off the entire left side of your Bang and Olufson. It bites the postman’s fingers. It then sleeps through an actual burglary. And finally it eats one of your children.

    This is the UPS experience: If they’re not preoccupied with smoldering their lead acid batteries, then they’re busy buzzing and arcing. Then they blow an internal fuse on the output, and your Great American Novel is suddenly lost, again, for the third time. Then there’s an actually power failure (Yay!), so they turn on their patented 387 volt offset square wave, and your PC is instantly corrupted. Meanwhile battery acid squirts out onto the ceiling, again. Then, while you’re out trying to buy a replacement PC, the UPS catches fire and burns your house down.

    I’d happily pay $800 to not have one.

    1. Destroy All Monsters Silver badge

      Re: UPS <- a root cause

      Where the hell do you get your batteries?

      Mr. Wong's cheap and cheerful shoppe??

      I have never had one of the modern fully-enclosed lead-acid batteries pose a problem.

      1. DryBones

        Re: UPS <- a root cause

        Same. Quit buying no-name UPSes from Alibaba.

      2. Solmyr ibn Wali Barad

        Re: UPS <- a root cause

        About 10 years ago, original APC packs usually contained cells made by La Long. Never saw them catch fire, but they bulged and vented quite happily.

  17. Charles Smith

    Venting

    When you had halon in a room you were supposed to have a venting/extraction system to clear it after a release. I saw a novel approach to this on a seventh floor equipment room, the room had never been pressure/leak tested. We suggested to the client it was needed. On testing we discovered the halon would have vented down the adjoining lift (elevator) shaft if there'd been a release.

  18. Anonymous Coward
    Anonymous Coward

    We had the CO2 equivalent

    For the record the CO2 ones are fine, they take the o2 levels below where a fire burns but still keep it above what a human needs, and include enough other stuff to make the body breathe harder.... I know this from experience. We were having the demo test of our new CO2 system and one of the vents was aimed into a rack of kit and froze a server solid, so I had to go in and fix it. Two hours of breathing like I had just run up 10 flights of stairs, and a replacement psu and all was fine.

    1. HPCJohn

      Re: We had the CO2 equivalent

      In the Oxford Supercomputer centre they have a similar system. An inert gas is used to keep the oxygen content in the server room below the point at which ignition will occur, but you can still breathe.

      Very sensible safety precautions there are never to work alone.

      I do recall feeling a bit woozy after being in the server room for a few hours though.

      1. Olivier2553

        Re: We had the CO2 equivalent

        If the fire suppression gas had been released, I would not consider entering the server room before:

        1) some professionals have cleared the danger of fire

        2) the air has been replaced and all the gas evacuated

        The gas is meant to be harmless to allow human exiting the room, not to continue working like nothing happened.

  19. dmacleo

    have had to take mechanics to hospital due to halon fire bottles in cargo area of aircraft blowing. even if no known inhalation you still need to be put onto O2 and blood gases measured/etc.

  20. Conundrum1885

    Re. biggenbang

    Around the time of the previously mentioned KABOOM my mini seismometer went off, consisting of sample of pyrolytic graphite hovering over four small magnets in a Halbach array.

    Went to check on something else and the graphite was sitting on the window sill.

    Catch is I was hundreds of miles away.

    Any ideas?

  21. Anonymous Coward
    Anonymous Coward

    I lived 90 miles away from Mt. St. Helens when it blew back about 38 years ago. We were almost due south and didn't hear anything - but people hundreds of miles away in other parts of the NW US heard the explosion that signaled the start of that eruption - the biggest natural or man-made explosion in the memory of modern America.

  22. chas49

    Halon toxicity

    It's not lack of oxygen that'll hurt you

    "Although generally considered a safe and inert agent, inhalation of air containing Halon has been associated with a toxicologic syndrome affecting the respiratory, cardiovascular, and central nervous systems, as well as the skin."

    (https://www.sciencedirect.com/science/article/pii/S073567579690087X) (paywall)

  23. ICPurvis47
    FAIL

    Running on the Spot - Begin!

    While I was working in a large chemical factory in northern Germany, installing some new cell line shorting switches, I witnessed a potentially dangerous, but extremely amusing incident. My Boss and I had finished the installation of the twelve switches along the side of the Castner-Kellner chlorine producing cell, and the local operatives had lowered the tank lid into position and clamped it down with several tens of large plastic G-cramps. I was intrigued as to why they were using plastic cramps and not metal ones, I presumed that it was for electrical insulation reasons, but it was soon to become obvious why. I was in the control room observing as the switches we had installed were opened by remote control to divert the electrical current through the brine in the cell to begin producing Chlorine gas by electrolysis, and my Boss was outside on the production floor watching the operatives as they moved around on top of the tank making fine adjustments to the depth of the electrodes into the brine solution to equalise the local voltage drops. Suddenly, an alarm went off in the control room, and the Controller made an urgent sounding announcement (in German) over the Tannoy on the production floor. Every one of the operatives on top of the tank dropped their adjustment keys and turned towards the door into the refuge room. At that moment there was an enormous WHOOSH, and the tank lid, about 100 feet by 30 feet, lifted two feet into the air, firing the plastic G-cramps out sideways, and propelling the operatives vertically into the air, where they all commenced "Running on the Spot" in mid air, before landing in full panic mode back on the tank lid and scarpering towards and through the refuge room door, slamming it behind the last man in. A large brown cloud of Chlorine gas rolled across the production floor after them, engulfing my Boss and the Production Supervisor, who were not fast enough to reach the Control Room door before it did. The extraction system kicked in and soon cleared the gas out into the atmosphere outside the factory, and Boss and Prod Sup staggered into the Control Room and sat down to regain their breath. Although this was an extremely dangerous situation, resulting eventually in the untimely death of my Boss some weeks later, it was obviously well planned for by the Production Staff of the factory, and was, at the time, excruciatingly funny to see several blokes all swivel as if magentically attracted, start running in mid air, and, on landing, all make it to and through a narrow doorway into the refuge.

  24. Richard Pennington 1

    Climate control, 1980s style

    No fire here (sorry), but at my first place of IT employment (mid 80s) they had a machine room with controls for temperature and humidity.

    Rumour had it that by proper (by which I mean "highly improper") adjustment of controls (humidity right up, temperature right down...), it was possible to get it to snow in the machine room.

  25. Skeptic Chicken
    Facepalm

    Been there unfortunately

    Years ago when exiting a comms room, yep hit the button with a ladder.

    Thankfully for me, it had been recently been installed wrong and only called the fire department rather than releasing the gas. Looked like a dam fool, explaining it though.

    (It is now correctly wired and covered with a roll of tape that can be easily removed)

  26. Anonymous Coward
    Anonymous Coward

    I heard a story...

    ...about our head of service who "liked to play" with systems he probably didn't really understand. But because he was the boss, he allowed himself to play. While titting around with the fire suppression system once remotely (or just down the hall from the server room) he set it off by mistake. While the security guide was in the server room doing a check (not sure why the security guide was in the server room). Apparently he passed out but thankfully survived.

    That was brushed under the carpet as far as I was aware. Either that or it was just a made up rumour.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like