back to article Datacenter fire suppression system wasn't tested for years, then BOOM

The "P6 rule" states that Proper Planning and Preparation Prevents Poor Performance. So as you prepare for the imminent weekend, The Register invites readers to run an eyeball over another edition of On Call, our weekly reader-contributed tale of tech support chores that involve another P – as in "going pear-shaped." This week …

  1. KittenHuffer Silver badge
    Flame

    Need more ....

    .... detail about the burst ear drums and associated trauma suffered by the poor peons in the rooms at the time.

    --------> Obv!

    1. I Am Spartacus

      Can Confirm.

      Can confirm that when halon goes off it does sound like a small tactical bomb. Especially if you are not expecting it.

      Under floor halon globe blew the dress completely off one of our female tape operators and another drove the senior operators chair across the consoles and broke his arm. Unions had hell to pay for this.

      No fun was had by anyone that day.

      1. sev.monster Silver badge
        Coat

        Re: Can Confirm.

        Not even the blokes with functioning arms?

      2. Anonymous Custard Silver badge
        Trollface

        Re: Can Confirm.

        And was the female operator named (or thereafter forever nicknamed) Marilyn?

      3. Not Yb Bronze badge

        Re: Can Confirm.

        Somewhat curious, but why would the union have to pay?

  2. b0llchit Silver badge
    Devil

    Tempt fate

    "The abort buttons reset the counter to zero with each momentary press,...

    When you need this button, it has been worn out by the testing (NC gets stuck and NO doesn't close). These buttons tempt fate.

    Murphy will always be around...

    1. FIA Silver badge

      Re: Tempt fate

      I'm pretty sure with a button that 'resets' a 30 second countdown to zero Murphy will be closer than some think...

      1. b0llchit Silver badge
        Pint

        Re: Tempt fate

        Sir, I stand corrected. My hat is off and the bow is deep. Have one on me.

        1. Roland6 Silver badge

          Re: Tempt fate

          I assume the glass contains Murphy’s…

    2. Alien Doctor 1.1

      Re: Tempt fate

      Do you know what that fire extinguisher did? It exploded in my face. I mean, what is the point of a fire extinguisher? It sits there for months, and when you actually have a fire - when you actually need the bloody thing - it blows your head off! I mean, what is happening to this country? It's bloody Wilson!

      1. PerlyKing
        Thumb Up

        Re: Tempt fate

        Derek & Clive?

        1. Alien Doctor 1.1

          Re: Tempt fate

          Basil Fawlty old chap. Derek and Clive would be slightly more rude.

          1. Anonymous Custard Silver badge
            Trollface

            Re: Tempt fate

            Just don't mention the war...

      2. Scott 26

        Re: Tempt fate

        f-f-f-f-fire..... f-f-f-f-fire....

        1. Sgt_Oddball

          Re: Tempt fate

          Does that mean someone has to dial the emergency services?

          So it's... 0118 999 881 999 119 725..........3

          1. Martin-73 Silver badge

            Re: Tempt fate

            Oh, that's just typical.... I'll just put it over here with the rest of the fire

  3. Anonymous Coward
    Anonymous Coward

    I had to deal with that kind of system at a place I worked at many moons ago.

    Every Friday we'd test it by triggering the alarm, with the added pleasure of having to answer a phone call. Fail to cancel the alarm meant a halon dump, failing to answer the phone meant a few fire engines turning up at the front door.

    Never triggered either the entire time I was there (a minor miracle in itself).

    Used to have a guy turn up once in a while with a cup on a stick thing to test the sensors too.

    1. Stuart Castle Silver badge

      Our old fire alarm used to go off regularly. I was friends with the office manager for the facilities management dept, and we were talking about it once. Although she didn't work in out building, she had com over to monitor the drill we were having, so I took advantage of the opportunity to ask her. The alarm was triggering several times a week at that point, and we were getting to the point where the users were starting to ignore it.

      She said the problem was simple. We have several buildings close by. All listed, and due to the listings, when the buildings were refitted for our use, to minimise the damage to the buildings, they went for an alarm that mostly used battery operated sensors, sounders and buttons, all connected together via radio.

      The problem is that the alarms in the different buildings frequently interfered with each other, and when that happened, they tended to go off. The system was ultimately replaced. Not sure how the new ones work, but we have a lot fewer false alarms.

      Of course, the new system won't stop the cause of most of the other false alarms (we have a small kitchen for employee use and people frequently burned stuff while cooking it, which tripped the smoke detectors), but working from home has pretty much stopped that.

      1. Anonymous Coward
        Anonymous Coward

        Why install smoke alarms in kitchen areas? Surely a heat detector is more appropriate (as in the rules for housing, in Scotland at least)…

        1. Sequin

          A colleague of mine once caused the evacuation of a 17 storey office block by burning her toast one morning -after this the toaster was removed and a ban was imposed.

          1. Doctor Syntax Silver badge

            Sometimes shooting the messenger is the right thing to do.

          2. Rich 11

            She was toast.

        2. Richard 12 Silver badge

          Building regs in the rest of the UK as well.

          1. Anonymous Coward
            Anonymous Coward

            The latest regs in Scotland require all domestic properties to have interlinked detectors - heat in the kitchen, smoke on the main living area and smoke in each circulating area on each floor. Detectors can be mains or battery powered but the batteries have to be non-removable, and detectors no more than 10 years old. CO detectors are also required near all appliances that burn gas, though do not have to be interlinked. Commercial and rental properties have had similar requirements for years, but it was added to private dwellings in 2022 - recent new-builds may meet it but there's a lot of older housing needing retrofit...

            1. Intractable Potsherd

              One of the (very few) reasons I'm glad not to live on Scotland any longer. The requirement is totally disproportionate to the risk.

              1. John Robson Silver badge
                WTF?

                Heat detector in kitchen, smoke on each landing, and one in the lounge.

                In what way is that overkill when we're talking about risk to life?

                1. Roland6 Silver badge

                  You missed the one in the hall, for unwanted letterbox deliveries…

                2. Duncan10101

                  "Risk to life" ... I think you're only looking at one side of the equation. There are risks to life caused by installing too many sensors: an obvious one being that "More sensors" = "More false alarms". This makes people insensitive to them, and they might well ignore a real fire. In fact, that's a fundamental theme of the article (and the many many stories in the comments too).

                  So, yeah, I believe the sensors shouldn't be disproportionate to the actual risk.

                  Exactly what level is "Proportionate" is another question. When governments get involved in "Making people safe", I doubt very much if anybody has really done that calculation.

                  1. Dave314159ggggdffsdds Silver badge

                    WTF? The level of tinfoilhattery here is off the scale. They're fucking fire alarms. There is almost zero downside to going completely OTT, let alone to making a sensible, normal level of protection mandatory to stop idiots cutting corners.

                    1. Martin-73 Silver badge

                      Yes indeed, plus the fact that with the demise of ionization detectors, or at least deprecation, the level of false alarms has dropped to near zero unless you are the kind of person who eats truly burned food, with the kitchen door open. In which case you deserve the minor inconvenience

                  2. John Sager

                    They can't do that calculation properly - it's always heavily weighted by CYA considerations by the Pols and TPTB generally.

                    1. Dave314159ggggdffsdds Silver badge

                      What calculation?! They're smoke alarms ffs.

                3. Martin an gof Silver badge

                  I have no problem with a requirement for interlinked smoke (and heat) alarms. Smoke alarms are known to have saved many lives over the years since they were commonly available for home use, and making them mains-powered, battery-backed and interlinked removes one of the biggest problems with domestic alarms - that people do have a habit of removing the batteries, or simply not maintaining the units.

                  We have seven alarm heads in our house altogether. The problem is the very occasional false alarm, which can often be attributed to Baby Spider Season. Ours are also interlinked with a small control panel in the utility room which has buttons for "test" and "silence", with the latter silencing all alarms except that one which triggered, meaning you can in theory trace it. In practice by the time you get to the button, the spider has moved on and the alarm isn't in "trigger" mode any more. Nevertheless it's still a lot more convenient than the way it worked in our old house, which required climbing up on steps to push the button on the alarm head itself. Even so, the children have still been known to sleep through the racket of seven sounders.

                  Building regulations in Wales changed from 2016 to include a requirement for sprinklers in all private domestic properties (as well as an existing requirement for multiple occupancy buildings). This one is a bit more onerous as it adds a minimum of £1,000 to the build costs, possibly substantially more if you need boost pumps or storage tanks, and in theory you need an annual drain-down and check. Problem there is that installers are stretched enough doing installations and don't have a lot of time for maintenance.

                  While the benefit of domestic sprinklers - particularly in smaller houses - is debatable (they are not designed to put fires out - they are designed to suppress the heat and smoke to make escape easier for longer, and possibly reduce spread until the fire brigade arrives, thus reducing overall damage) what is surprising and, I'd say, shameful is that while most new-build and re-build public buildings in Wales (schools, hospitals, libraries etc) have been required to be fitted with fire suppression systems since 2011 (I think - couldn't find a reference), proposals for similar regulations in England are constantly defeated.

                  M.

                  1. John Brown (no body) Silver badge

                    I'm not so sure about a mandated sprinkler system in private residences, especially single family dwellings. That could be a very, very expensive false alarm.

                    1. Martin an gof Silver badge

                      I didn't say I was an out-and-out advocate of sprinklers in the home, but this false alarm business must be put to bed:

                      • Sprinklers do not all go off at once like in the movies - only the sprinklers in the immediate vicinity of a fire will activate. They are not and cannot be interconnected. Each one works entirely independently (see below).
                      • Sprinklers cannot "false alarm", though I suppose there is potential for mechanical failure causing an individual head to leak.
                      • Sprinklers - as normally installed - are a small bulb filled with a liquid, blocking the outlet of a water pipe. If the temperature of the liquid in that bulb passes a certain threshold (IIRC 56C), its expansion will break the bulb and unblock the water outlet in that one - and only that one - sprinkler head.
                      • Domestic sprinklers are usually recessed into the ceiling and covered with a metal plate held on with solder which melts at around that same temperature so that the plate falls off to allow the sprinkler to activate.
                      • A pressure switch can be fitted to the supply pipe, which can be connected to a conventional fire alarm panel in order to activate the alarm in the event of a sudden pressure drop indicating activation of a sprinkler head, but it cannot be made to work the other way around.
                      • In most domestic fires a working smoke and heat detector system will trigger well before the sprinklers do.

                      This (pdf) document sets out reasonably clearly how these things work.

                      They are not easy to retrofit which is why the building regulations are not retrospective (building regulations very rarely are retrospective), but when major works, rebuilding etc. is carried out it is not actually terribly difficult to run the pipes. Domestic systems often use a heat-resistant plastic pipe (commercial systems are often metal pipe) which has a certain amount of flex and is glued together. You can make it even easier (and other services too) by using open web joists (just an example manufacturer) if you are replacing floor joists.

                      The main problem in a retrofit to be perfectly honest is likely to be upgrading the mains water connection. Leaving aside situations where a storage tank may need to be fitted and / or a booster pump, many houses in the UK are fed from the water main in the street through a 15mm Copper pipe (or ½" Lead still in some cases) which is inadequate to supply sprinklers directly. Newer houses might have 20mm or 25mm plastic supply (because 15mm can also be inadequate when there's a combi boiler and more than one bathroom), but a sprinkler system really needs a 32mm supply. Obviously the water company will charge to upgrade this (and fit a new meter) though in our area it's not an exorbitant charge.

                      M.

                      1. TheWeetabix Bronze badge

                        Dry, standpipe systems are and must be interconnected, all the sprinklers come on at once. Dry standpipe systems do work the other way around. Once the alarm comes on the water flows.. you can also set up pressurized dry standpipe system, where the water only comes on after the gas pressure is let out of the system there by triggering an alarm. These systems also have much lower maintenance costs and risks of leaking. Is there any particular reason you didn’t go over any of that?

                        1. H in The Hague

                          Not familiar with the Welsh regulations in detail, but used to be familiar with the field.

                          "Dry, standpipe systems are and must be interconnected,"

                          Not sure that is correct, I think those can work with a single head as well. When dry the system is filled with compressed air, if the pressure suddenly drops that means a sprinkler head has actuated and then water is let into the system and escapes through the opened sprinkler head. Used in cold warehouses, etc. where a system permanently filled with water might freeze. Disadvantages: more complex and needs an air compressor to occasionally top up the pressure if that drops through small leaks.

                          " all the sprinklers come on at once."

                          That's a deluge system. Normally only used over fuel tanks, offshore installations, etc. Requires a v high water flow. Not used in dwellings, offices, etc. where a fire normally only triggers one or a few heads. The idea is to suppress a fire at its incipient stage, before it spreads. (And the water supply would be inadequate to supply a larger number of heads.)

                          " gas pressure is let out of the system there by triggering an alarm. These systems also have much lower maintenance costs and risks of leaking."

                          Why would the maintenance costs of a dry system be lower than those of a wet system? For a start you need a compressor to maintain the pressure in the system while dry. Or did I misunderstand your comment?

                  2. Anonymous Coward
                    Anonymous Coward

                    "Smoke alarms are known to have saved many lives "

                    Apply *anything* to 50 million people and hand pick the cases "anything" have "saved" lives and then mandate that to everyone because it "has saved lives" is an excuse to collect bribes from the manufacturer. No more, no less.

                    Every one 'saved' dies anyway. That's the harsh reality. Living longer to collect pension is a cost, not a saving. So bribes are the *only* rational reason for this BS.

                    Sprinklers are absolute madness, so you can bet any amount that the people mandating it got bribed very well.

                4. Aitor 1

                  Not proportional to cost.

                  Having it fitted we are looking at £500 or about.

                  I did install it because even if overkill, hey, it is my life, and I can afford it.

                  But considering how many fires are, and how many lives this would save.. I think this is overkill.

                  There are about 44 deaths by fires in Scotland per year.

                  Lets assume the US number of 55% reduction is correct.

                  There are 2.53 million households in Scotland.

                  Let's assume all those 44 deaths are residential, and no firefighters, etc (so best case scenario for alarms)

                  This would save 24 lives.

                  The battery ones last about 6 years really, not 10, and the installed ones are very expensive to install.

                  The cost? 8.8 million pounds per life.

                  While life is quite valuable, there are many things than can be done for less money to save lives, and actuaries would agree.

                  I would say, fix potholes, put lights on dangerous intersections, do colon cancer screenings sooner, etc etc.

                  If you fit them yourself, it is better value, still quite expensive, but I did it for £200

                  Sources:

                  https://www.nfpa.org/News-and-Research/Data-research-and-tools/Detection-and-Signaling/Smoke-Alarms-in-US-Home-Fires#:~:text=The%20risk%20of%20dying%20in,to%20trigger%20a%20smoke%20alarm.

                  https://www.nrscotland.gov.uk/statistics-and-data/statistics/statistics-by-theme/households/household-estimates/2021

                  1. Dave314159ggggdffsdds Silver badge

                    Re: Not proportional to cost.

                    What an insane bit of nonsense. The deaths per year are so low because we already have smoke alarms fitted by everyone who isn't mad, and we already have fireproofing regs. Your political nutcasery is brainworms.

                  2. Richard 12 Silver badge

                    Re: Not proportional to cost.

                    I just replaced all the detectors in my home.

                    Mains interlinked detectors do not cost £500 per ten years.

                    The initial installation cost might be £500, but almost all of that is running the wires and making good afterwards. That set will last 10 years - £50pa.

                    The next set don't need installation because all the wiring is already done. They are literally plug and play, I didn't need any tools at all.

                    They're £20 each, a "typical" home has three, and they'll last another ten years.

                    So £6 a year per home after initial installation costs, or £632k per life once installation is amortised.

                5. Anonymous Coward
                  Anonymous Coward

                  "Risk to life" is technically zero: These devices save literally zero lives and there isn't even a theoretical chance to do that: Everyone will die. Some sooner than others. Which part of that you decided to ignore?

                  Therefore they are almost always overkill and benefit only the company who makes them and the authority who got paid to make them mandatory: Bribery at its finest.

                  The *only* benefit is that some people live longer than they would have otherwise lived and there's a price tag for that. It is *a lot* less than mandated alarms in every room.

                  For a society that's typically a major *cost*, not a benefit : Old people get pensions and get sick a lot. I should know, I am old people.

                  1. Anonymous Coward
                    Anonymous Coward

                    I realise you are probably trolling, but by that logic the moment someone stops working and starts taking their pension, society should treat them as a burden rather than a benefit. Don't fit smoke alarms because if Joe Bloggs's electric blanket shorts out it's better for society that he dies in a house fire than he is able to escape without injury and live another 10, 15, 20 or more years, increasingly burdensome to "society" in general. (Let's just ignore the innumerable benefits retired people bring to society, eh?)

                    Have you ever watched the films or read the books Logan's Run and Soylent Green? I'm sure there are plenty of other such imaginings of dystopias created by overenthusiastically allocating monetary or societal "value" to individual lives. Brave New World springs to mind (though I'll admit it's a boring read).

              2. Smndnm

                Maybe… one needs to understand what risk is, I’ll help you. Risk is the situation of being exposed to uncontrolled energy, termed the hazard. Risk is quantified by the product of probability and consequence. An analogy:- the hazard is the shark in the water, the risk is going swimming in that water. The breed of shark and the chosen swimsuit/diving cage/chain mail suit all contribute to risk calculation. In the case of a house fire, the probability is low, it is not something that happens in every house every 5 years, the consequence however, is total loss of building and total loss of life. Interlinked detectors in the home have zero impact upon probability, but they do decrease the consequence by providing time and awareness to escape the hazard and save lives, pets and buildings. Not totally disproportionate in my view, the alternative is captain obvious at the funeral saying “ oh if only they had installed…..”

              3. Martin-73 Silver badge

                I see you value your life very little sir/madam. going above and beyond any rules and having an optical in every bedroom/circulation area/living area, plus heat and CO where appropriate, all interlinked, is cheap compared to losing your home/family.

                A small example (I am an electrician so i fit the things on a daily basis), but my own home, 2 bed bungalow: ... Optical in each bedroom/ Multisensor* in the hallway, ionization in the attic**, and heat in the kitchen, and CO in the boiler cupboard.

                *an Aico thing, it compares sensor types to determine if it's likely a false alarm or not...

                ** i want a fast alarm if the fire starts in the roof space, so it's one of the few places imho an ionization is still a sensible choice.

                Overall system cost, about 350 quid... bought as a bundle, excluding the fixed wiring, which cost me half a day and about 50.... lifespan, 10 yrs, less than 10p a day. Fire is NOT your friend, i know we hereabouts hate being told what to do, but scotland has the right attitude on this.

                /2 cents

                (edit) for non electricians that 50 quid might be significantly more, so i see why scotland put it into the legal requirement...

            2. J.G.Harston Silver badge

              So what happens, do I get fined if I don't have a detector in my kitchen? I get fined for not living my life as the authorities demand I do? Isn't that..... that political fad that was popular in the '30s that we must not name.

              1. John Robson Silver badge

                No - it's called building regs. They don't apply retroactively.

                If you build a new house, or significantly alter an existing one, then you put the detectors in at that point, or it doesn't get signed off.

                Possibly also if you let a house?

                And mine are interlinked by their mains supply cable (triple and earth) which is fed from a lighting circuit (because you're not likely to leave that switched off).

                You can also have them linked by radio signals.

                1. This post has been deleted by its author

                2. J.G.Harston Silver badge

                  Ah, so it's if you *build* a house, it must have x, y, z. That's very different to what the OP said.

                3. Roopee Silver badge
                  Coat

                  Incorrect

                  The new Scottish rules apply to all domestic properties, not just new builds, from Feb 22. As pointed out however, very difficult to enforce unless there are other agencies involved...

                  In Feb 22 and for months after it was almost impossible to buy the approved battery type alarms (which were around £50 each) as they were out of stock everywhere. Mains alarms were much cheaper to buy, but very difficult and expensive to retro fit in most properties because of the interlinking, making the 10-year sealed battery ones a no-brainer.

                  Mine’s the one with a Scottish Power of Attorney in the pocket for a relative who moved to Scotland...

                  1. Anonymous Coward
                    Anonymous Coward

                    Re: Incorrect

                    The point where enforcement occurs is when someone is buying the house and they do the survey. The results come back as not compliant, and then the buyer says you must fix this before i can buy the house. I doubt there will be a price negotiation "fix", e.g. well lets knock 5 grand off the price and you do it when you move in, as the mortgage company wont accept the house as security for the loan.

              2. Dave314159ggggdffsdds Silver badge

                "I get fined for not living my life as the authorities demand I do? Isn't that..... that political fad that was popular in the '30s that we must not name"

                No, you fucking Holocaust denying idiot.

                1. Martin-73 Silver badge

                  To be fair it was co-incident with that and done by the same bunch of w*nkers...

        3. General Purpose

          He didn't say the smoke detectors were in the kitchen. Properly burnt toast can set them off at quite a distance.

          1. keithpeter Silver badge
            Windows

            @General

            About 20 feet easily. Good pavlovian conditioning to close kitchen door when heating croissants on the toaster wire thingy.

            I call it the monthly test.

        4. Zenubi

          I have a friend who was a fireman - he said the local nurses home - quite a large block - was subject to kitchen and other false alarms.

          He said the tender was crewed in record time for these - especially at night.

          One imagines many sleepy nurses in the parking lot in less than normal clothing but he would not be drawn further.

        5. Anonymous Coward
          Anonymous Coward

          Here the vagariies and variences of stateside fire code kick in

          For you see, a heat sensor simply won't do, you need a smoke sensor. It must be (according to the last fire inspector) immediately adjacent to any cooking surfaces. There may be a slightly different model allowed for use by gas ranges, but the break room only has microwaves, so it isn't eligible.

          Instead what is three feet in front of a bank of 3 microwaves is a sensor worthy of an airplane bathroom. 6 fire engine roll-outs in four years since the remodel. 3 of which were from microwave popcorn, which will still draw a steely glare of disapproval from management to this day.

          Reason be damned in the face of the National Electric Code. I think the same sensor was installed in the kitchen in my last apartment. It would reliably let your neighbors know when your bacon was just shy of the crispy phase.

          1. M.V. Lipvig Silver badge

            Re: Here the vagariies and variences of stateside fire code kick in

            My house had one of those stupid in the kitchen smoke alarms, and you don't even have to be burning food to set it off. Just a little oil vapor was enough. Being a USAian, I removed it, and false alarms went to zero. There are plenty of other smoke alarms in the house, more than enough to warn of fire anywhere in the house.

    2. Xalran

      That reminds me of my time in Priory Hall of Residence at the then Coventry Polytechnic...

      While no Halon were involved, the Fire Alarm going off was a classic as soon as you put 4 or more students in a room doing what students do in the evening...

      It went off several time a weeks, and in some cases several times in a single night. ( when some were in playful mood. )

      Every time it meant :

      - evacuation of everybody willing to evacuate.

      - wait for the fire trucks to fill up Priory Street.

      - wait for the check that everybody has evacuated, and evacuating those that didn't.

      - wait for the all clear from the firemen.

      - pile up back into the buildings, using emergency exits as entrances... because they were still open. ( and thus bypassing the brand new RF card reader )

      1. Alan Brown Silver badge

        " the Fire Alarm going off was a classic as soon as you put 4 or more students in a room doing what students do in the evening..."

        The $500 charge per callout from the local fire brigade for repeated false alarms was a strong dissuader once that fee started being passed to the occupants of the rooms in question (or floor, for communal space triggers)

        (that's $2500 in today's money)

  4. diver_dave

    Err

    From my military days there's 7 P's

    Proper Planning and Preparation Prevents Pi** Poor Performance.

    1. lglethal Silver badge
      Go

      Re: Err

      I've always preferred "Poor Planning on Your Part, Does NOT Constitute an Emergency on Mine".

      Although maybe that's not quite so accurate in the above story, since it would definitely be an emergency for whoever was in that room!

      1. chivo243 Silver badge

        Re: Err

        My framed version states "Bad planning on your part does not necessarily constitute an automatic emergency on my part."

        1. nintendoeats

          Re: Err

          There was one manager at my old workplace who had that on his cubicle. He was head of marketing. I always found that odd.

          1. Doctor Syntax Silver badge

            Re: Err

            Don't expect marketing to read any meaning to a slogan. How would they be able to do what they do if they did that?

      2. Spanners
        Happy

        Re: Err

        We have had that on our department wall for years!

        Sometimes, someone will point it our to a visiting manager...

        1. A.P. Veening Silver badge

          Re: Err

          Sometimes, someone will point it our to a visiting manager...

          Meaning every time there is one of those extremely rare visiting managers.

    2. WonkoTheSane
      Headmaster

      Re: Err

      As quoted in "Carry On England" (1976)

      1. John Brown (no body) Silver badge

        Re: Err

        Many of the Carry On team, actors and writers, had WWII military experience, or at least ENSA experience so quite possibly/probably picked up common military phrase for use later in their careers :-)

    3. Doctor Syntax Silver badge

      Re: Err

      That's what I was thinking. It's not exclusively military.

    4. Handy Plough

      Re: Err

      “No plan of operations extends with any certainty beyond the first encounter with the main enemy forces” or, in simpler terms “Everybody has plans until they get hit for the first time”.

  5. Anonymous Coward
    Anonymous Coward

    death trap

    The whole setup sounds like a death trap to me:

    - office also had detectors and would be flooded by Halon ? WTF ? Not so long ago, people would routinely smoke in offices !

    - the whole 30 s counter and reset button (inside the area concerned by Halon) also sounds lethal to me by keeping people in the fire/halon area

    - the above counter thing doesn't sound like something people wouldn't forget, 2 decades after setup

    Hopefully, no-one was hurt this time, but like someone in road security in France I know, is used to say: "There are actually very few accidents vs. the amount that are just waiting to happen (due to people carelessness)".

    1. chivo243 Silver badge

      Re: death trap

      Makes you understand why the BOFH and PFY like Halon! Speaking of those gents... perhaps today we get an episode?

      1. Joe W Silver badge

        Re: death trap

        Halon is much less immediately deadly than the now (apparently) preferred method of just flooding the entire room with CO_2

        1. Anonymous Coward
          Anonymous Coward

          Re: death trap

          modern fire suppressants don't use halon and DC's don't normally use or CO2 its generally either FM200 which works by absorbing heat or IG55 which works by reducing oxygen levels. Both are safe and you can be in the DC if they go off, they won't kill you! Althought weatehr you would want to be in there when it discaharges is another question! ;o)

          1. Phil O'Sophical Silver badge

            Re: death trap

            I've never really understood why the discharge has to be so violent. It's obviously important to get a lot of gas everywhere in the room, quickly, but would a gradual 20 seconds versus explosive 2 seconds really make much difference?

            1. jollyboyspecial

              Re: death trap

              "I've never really understood why the discharge has to be so violent. It's obviously important to get a lot of gas everywhere in the room, quickly, but would a gradual 20 seconds versus explosive 2 seconds really make much difference?"

              A slow release will not flood every nook and cranny with gas. An explosive release will come close. especially if more gas is emitted than the volume of the room.

              1. Catkin Silver badge

                Re: death trap

                Plus, you want a good mix so you don't get a pocket of pure halon that drops a person in the same fashion as any other non-oxygen gas would.

            2. Peter2 Silver badge

              Re: death trap

              I've never really understood why the discharge has to be so violent. It's obviously important to get a lot of gas everywhere in the room, quickly, but would a gradual 20 seconds versus explosive 2 seconds really make much difference?

              I'm not a particular fire specialist but a fire is detected through smoke making it from inside the box of a burning bit of equipment to outside of it, to outside of the rack, to the smoke detector. This is going to cause a delay after the fire has started at which point it's likely to start spreading. An alarm sounds giving a 30 seconds notice that you would really like to leave the area. Do you then want a nice gentle gas release in one corner risking a dense cloud of suppressant in one area of the room, which would slowly (very slowly) be pushed around with air currents and to start taking effect within a few minutes and maybe start damping down the fire within 5-10 minutes?

              I mention this because your not only fighting the fire; your racing the fire brigade to doing it.

              The fire brigade average response is something like 6 minutes in urban locations to 8 minutes in more rural settings. (and if your in an urban location with a DC then chances are that you are probably nearer their station so their response will be well under the average) and their usual MO for dealing with a fire is to cut the power, drag a hose in smashing down any doors in their way and then they'd spray your server racks with several tons of water per minute.

              Most IT equipment does not like water much, and this sort of treatment is liable to have adverse impacts on uptime.

              So if I did have the option of having a gas suppression system like this then i'd spec it with about twice the amount of fire suppressant needed for the size of the room to allow for leakage out of the room and then have an explosive deployment to cause an overpressure wave to flood the room with suppressant near instantly so that the fire is out near instantly preventing it from spreading and causing further damage, with the aim of any fire being out at about the same time the fire brigade chaps have their fire engine out of the door.

              This way you might just be able to get away with turning either an individual bit of equipment or a rack off instead of the entire building. ;)

              1. John Brown (no body) Silver badge

                Re: death trap

                "Do you then want a nice gentle gas release in one corner risking a dense cloud of suppressant in one area of the room, which would slowly (very slowly) be pushed around with air currents and to start taking effect within a few minutes and maybe start damping down the fire within 5-10 minutes?"

                While I take you point, I think he's suggesting something less violent, but still relatively violent, something akin to power of gas or powder fire extinguishers, ie the gas gets there pretty quickly, but not violent enough to itself cause damage and injury.

                1. Peter2 Silver badge

                  Re: death trap

                  While I take you point, I think he's suggesting something less violent, but still relatively violent, something akin to power of gas or powder fire extinguishers

                  And I take that point, but how well would you expect a CO2 extinguisher to work if instead of basically shoving it into the fire before pulling the activation levers, you pulled the activation levers while standing by the door of the room the fire was in?

                  I'm going to go out on a limb and suggest that it wouldn't work very well.

                  1. John Brown (no body) Silver badge

                    Re: death trap

                    I'd assume it would have to be designed into the room. Assuming the gas used is heavier than air, then maybe the equivalent of a sprinkler system blasting down from multiple nozzles across the room. I think the point isn't so much to target the fire as flood the room with fire suppressant and/or remove/reduce the oxygen content. Dunno, not my area, just making suggestions. Maybe it would be too expensive or some other reason it's nor been done. I'm just trying to imagine a system that does the job without itself causing damage when triggered.

                    1. Alan Brown Silver badge

                      Re: death trap

                      That's exactly how modern suppression systems work

                      Inergen is mostly argon with some CO2 to keep people's breathing relfex active, and specced to drop oxygen levels to around 12% - which is (barely) survivable by hoomans but will stop all conventional fires (except metal fires)

                      Overspeccing in such instances is a bad idea as it moves the systems from "dangerous" to "lethal"

                2. Alan Brown Silver badge

                  Re: death trap

                  Let's not forget that gas suppression systems are piped to multiple outlets - usually one every couple of metres along the length of the room - both above and below the floor

              2. Alan Brown Silver badge

                Re: death trap

                Just as a FYI, an 6 bottle Inergen suppression system (enough to cover a room with 16 racks) costs about £5k per fill

                Overspeccing the things is usually bounced by the beancounters

            3. Dave314159ggggdffsdds Silver badge

              Re: death trap

              "would a gradual 20 seconds versus explosive 2 seconds really make much difference"

              Yes. There are plenty of videos of fire progression on YouTube. It's incredible how fast it happens once it gets started. Or, not so incredible once you realise it's an exponential increase.

    2. Peter2 Silver badge

      Re: death trap

      With apologies to BOFH fans: Halon is not lethal upon inhalation (really; go and look it up) and if you remained in a room flooded with in for a few hours then the worst you'd get is a headache at very high concentrations.

      Halon does not work by displacing all of the oxygen in the air (that's a CO2 flood and those are genuinely dangerous) Halon just has a very useful chemical reaction; which is when it has more than a certain concentration in the air then it messes with the chemical reaction that is fire by essentially removing the heat from the flame, which then puts the fire out for as long as the gas concentration remains above whatever the required percentage is. (and at which point this concentration drops hours or days later you'd hope that somebody has removed the source of ignition...)

      Alas, it's equally as good at eating through the ozone layer so it's been ditched in modern years. Off the top of my head, modern equivalents are FM200 and Novatec 1230.

      1. Arthur the cat Silver badge

        Re: death trap

        Halon is not lethal upon inhalation

        But it makes coffee taste weird. A friend of mine had forgotten there was going to be a halon flood test so was working in the machine room when it was due. We had a sane machine manager who stuck his head round the door to check, so my friend was out of the machine room when the floods went off, but he'd left his coffee cup behind. When he went back in later to finish off what he was doing his cup was surprisingly in the place he'd left it so he automatically took a sip and promptly spat the mouthful of coffee out all over the operator's console.

        1. Sudosu Bronze badge

          Re: death trap

          Did that cause a fire?

          1. Arthur the cat Silver badge

            Re: death trap

            Just soggy paper.

      2. Alan Brown Silver badge

        Re: death trap

        "Halon is not lethal upon inhalation (really; go and look it up)"

        However when exposed to flames/red hot surfaces, it can form hydrofluric acid vapour. Inhaling that is seriously not recommended

    3. Anonymous Coward
      Anonymous Coward

      Re: death trap

      - the whole 30 s counter and reset button (inside the area concerned by Halon) also sounds lethal to me by keeping people in the fire/halon area

      eh? the reset button (normally by an exit) is standard issue in a DC, if you need to abort a gas release in case of a false trigger, they cost a sh1t tonne to refill those gas bottles! And modern VESDA's are VERY sensitive, hence why lots of DCs have a no cardboard box rule or anything else that might be dusty in the DC. We nearly had a false release a few years ago. A contractor was in the DC to work on a CRAC unit, SOP said that the VESDA should have been switched to manual before work took place but the contractor said he wasn't doing anything to create smoke so a colleague let him in. But what he did do was recharge the gas in the CRAC when he connected his gas tank to the system a small amount of gas was released (bit like when you connect your car tire up to pump it up) and that was enough to trigger the VESDA. Lucky we had a guy from the installers of the fire alarm system on site that morning and I was in the reception area talking to him when the house alarm went off! He immediately knew what it was and dashed up 2 flights of stairs and in to the DC just in time to press the magic button!!!!

    4. Doctor Syntax Silver badge

      Re: death trap

      "Not so long ago, people would routinely smoke in offices !"

      I'd vote for a Halon system just to stop that, let alone putting out fires.

      1. sev.monster Silver badge
        Holmes

        Re: death trap

        Anyone that smokes in a side office connected to the DC, presumably with the shared air circulation system pumping under the raised floor, deserves to have a tank of halon shotgun blasted into their face.

        Hopefully they quit [smoking] and don't come back.

      2. Outski

        Re: death trap

        I smoke like a beagle, but wouldn't spark up anywhere near a server room

    5. Elongated Muskrat Silver badge

      Re: death trap

      Not so long ago, people would routinely smoke in offices

      The 1970s ended 43 years ago.

      I've never, in my entire working life, worked in, or even visited, an office where people smoke in the actual office. One place I used to work at had a smoking room, but even that was got rid of long before the ban on smoking in the workplace was introduced (over 15 years ago), in favour of a "bus shelter" outside.

      1. Anonymous Coward
        Anonymous Coward

        Re: death trap

        Very late 80s, Toronto. Brit consultant, smoking his expensive smokes, feet up on table, regaling us about the expensive wines he'd had last night, while the stupid also-Brit vendor worker who would not ask for help stumbled onward, while we watched helplessly as we were not allowed to improve the situation (cuz foreigners). I think the customer allowed this to continue from the pure enjoyment of the vendor's self-inflicted pain, week after week after week.

        But, yes, every job situation since then was consciously picked with smokeless in mind. You were just lucky(?)

      2. Emjay111

        Re: death trap

        The Japanese company I used to work for had a 'smoking canteen' and a 'non-smoking canteen'.

        At times, you could barely see from one side of the 'smoking canteen' to the other, and it was a pretty small room. I left in the late 1990s, and as far as I know, this facility continued to exist until the ban kicked in during 2007.

      3. Roland6 Silver badge

        Re: death trap

        The UK only legally banned smoking in offices etc. in 2007. Prior to that smoking in offices was down to company policy…

      4. Andy A

        Re: death trap

        One place I worked forced smokers out into the fresh air. The facilities were strategically placed squares in the yards, each 4m on a side, with a bin for the fag ends.

        No shelter. No seating. Very attractive in the middle of winter.

        1. Arthur the cat Silver badge

          Re: death trap

          No shelter. No seating. Very attractive in the middle of winter.

          You have obviously never suffered from nicotine addiction. It's a bastard(*) – took me 11 years from first trying to give up smoking to actually succeeding (which was 30 years ago about now), and that was after a medical emergency. When I was still hooked I'd have been out there smoking even if I was accreting a thick layer of ice from freezing rain or being scorched by a solar flare.

          (*) I've been told it's worse than heroin by people who have given up both.

          1. Doctor Syntax Silver badge

            Re: death trap

            "You have obviously never suffered from nicotine addiction"

            You don't have to be an addict to suffer from smokers' addiction. It's why smokers get no sympathy from the rest of us.

      5. Dave@Home

        Re: death trap

        BBC Scotland, circa 1997, several senior managers still smoked in the office.

        I routinely refused to go to calls in those rooms, and got all sorts of earache

      6. Anonymous Coward
        Anonymous Coward

        Re: death trap

        i can remember in a ship yard I worked mid 90's people still smoking in the office :o( And a Lab I worked at from 1998 had a smoking room inside the building in the same physical space as the stationary store, all that paper etc!

      7. Alan Brown Silver badge

        Re: death trap

        In Kiwiland it was the late 1980s - and the smokers in our offices not only deliberately ramped up their activity prior to the law kicking in , but would keep lighting up long after the laws were activated (until a couple of them were sacked)

        I experienced something similar in the Netherlands and UK (only less obnoxiously) when laws were passed much later

  6. Catkin Silver badge

    There's a wonderful promotional video from DuPont titled "halon 1301 the instant firefighter" that features a demonstration of a lab tech in a plastic box trying to light a fire in a halon filled environment (that he's breathing in). Given DuPont's track record with worker safety, how long he survived afterwards is another question but it certainly doesn't instantly kill or impair.

    The mechanism is ingenious: it reacts with burning compounds to render them non flammable. It's really annoying that it's an ozone killer because having a breathable gaseous fire suppression compound that causes no damage really was a boon. I'd have it in my house if it were possible.

    Edit: sorry, this was meant to be a response to Peter2

    1. Catkin Silver badge

      The video in question:

      https://www.youtube.com/watch?v=NrP5-E9jmas

      Wonderfully 60s.

      1. J.G.Harston Silver badge

        Ooohh! Those waistcoat-suit combos! It's like going back in time!

        1. C R Mudgeon

          Indeed! That the guy in the box is wearing a tie just seems so strange -- even though I wore one, along with the suit jacket, for work in the corporate IT world, until I got hired at a start-up in 1982. (I never bothered with the vest though.)

          1. Roland6 Silver badge

            I think any one not around in the 80s wouldn’t comprehend the shock of new and the idea that you didn’t need to wear a suit to work in an office.

            In some respects the modern comparator is working from home…

      2. C R Mudgeon

        That demo setup made me think of the disintegration chamber from the Star Trek episode A Taste Of Armageddon.

    2. Alan Brown Silver badge

      "it reacts with burning compounds to render them non flammable"

      Whilst forming a bunch of nasty neurotoxins in the process. They won't kill instantly but I wouldn't hang around them

  7. Potemkine! Silver badge

    Business as usual

    The component you don't test will mandatory fail, generally in a spectacular manner and always at the worst moment.

  8. jollyboyspecial

    Many years ago I attended a fascinating demonstration.

    The first part of the demonstration was in a dummy room with normal atmosphere. A match was struck and dropped on a sofa. The resulting first was rapid and terrifying.

    The second part was an identical chamber with an identical sofa, this time however the atmosphere was different in an unspecified way. Matches were struck repeatedly and dropped on the sofa. The matches would initially flare as normal but go out almost immediately. Even when the match was struck right against the sofa the match would go out without any more than scorching the upholstery. A blowtorch was applied to the sofa, the upholstery started to burn but the flames died as soon as the blowtorch was removed. The mystery atmosphere was revealed to be that oxygen content was sligthly reduced. IIRC the concentration was between 17 and 18%.

    This was incredibly effective at preventing fires. Some stuff would burn at these concentrations of oxygen but not nearly as quickly or violently. A system to achieve this would not be difficult to build for a contained environment like a datacentre. An office building or residential space may be more difficult. And you wouldn't even need to use any noxious gasses. Nitrogen would do the job. Another good thing is that this wouldn't be harmful to humans, maybe it wouldn't be comfortable for somebody with low blood oxygen levels but I'd go out on a limb and say that firstly somebody in that condition shouldn't be at work and secondly even if they were it's not something that's suddenly going to kill them. They'd probably get short of breath and need to leave the room.

    I was always surprised that there wasn't a wider uptake of this sort of solution for environments like DCs. Sure it may cost a little to run 24/7 but I would have thought preventing fires from starting in the first place was preferable to waiting for them to start and then putting them out. I remember one DC having a minor fire, not only was the power to the DC cut as a precaution but when fire fighters turned up they wouldn't let anybody back into the building until they had completed safety checks. This level of downtime was very costly. There was also the cost of getting the halon tank refiled. And the cost of getting somebody in to repair the damage done by the halon system being triggered. And of course the risk of running without a fire suppressant system for a couple of days until the tank could be refilled.

    1. Anonymous Coward
      Anonymous Coward

      air exchange and air management systems to keep oxygen levels low I expect would cost a fortune to run! Don't forget air exchanges in buildings are part of building regulations. We are doing some load testing in our DC in a few weeks using rack mounted heaters and one thing we need to keep an eye on are CO2 levels in the DC during the test. And I'm not sure working long term in such an environment would be very healthy for you either!

      1. Peter2 Silver badge

        And I'm not sure working long term in such an environment would be very healthy for you either!

        Carrying around your own oxygen supply would sort that out quite easily, and it wouldn't matter if it stopped working as you'd still be able to get to the exit without it.

      2. sev.monster Silver badge
        Boffin

        Not sure if these exist on the market, but why not the same system in a rack? Environment controlled racks do exist, having one adjust oxygen level/pressure/mixture sounds doable. And you don't have to worry about human safety, as even if the system is not turned off and the closet is opened, the presumably small BTU system won't be able to affect the rest of the room.

      3. Anonymous Coward
        Anonymous Coward

        "air exchange and air management systems to keep oxygen levels low I expect would cost a fortune to run! Don't forget air exchanges in buildings are part of building regulations. We are doing some load testing in our DC in a few weeks using rack mounted heaters and one thing we need to keep an eye on are CO2 levels in the DC during the test. And I'm not sure working long term in such an environment would be very healthy for you either!"

        Where does C02 come into the above?

        Reducing oxygen content would not increase the CO2 content of the atmosphere. And a healthy human would not be harmed in air with 17% oxygen content.

        1. Anonymous Coward
          Anonymous Coward

          one of the main things air handling does is to help control the build up of CO2 in the room. How about non-healthy humans working in the DC?

        2. Anonymous Coward
          Anonymous Coward

          The air we breathe has approximately 21% of Oxygen. Oxygen is the key factor in sustaining a fire and the key factor in keep us alive too!

          By removing the oxygen, we will certainly extinguish a fire, but that comes with obvious problems. How do we sustain life at the same time?

          Fires need more than 15% Oxygen to combust. Anything below this level of oxygen will not be enough for a fire to sustain combustion.

          Luckily, we only need 12% plus of oxygen to survive and this is where the answer lies. To extinguish a fire and sustain life, we need to reduce oxygen from 21% (assuming were at sea level) to below 15%, but not less than 12%. This will extinguish a fire and sustain life at the same time.

          I actually can't believe people are discussing working in areas with reduced oxygen levels like it would be a good idea!!!!

          So the issue is you are working in a DC and the Oxygen level is maintained artificially at a low level of less than 15%, and then there's a feck up with the air handling system, and the oxygen level slowly reduces further and due to the effects of oxygen starvation you're not actually aware that its happening! This is why the regulations for working in "enclosed" spaces are so strict the lack of oxygen and build up of CO2 kills people!

          1. Elongated Muskrat Silver badge

            Exactly this. If I could upvote it more than once, I could. If you're doing anything to screw with the normal, atmospheric, levels of gases, you need to be monitoring it very well.

            I recall a story a while ago, where some physicists working with liquid nitrogen managed to asphyxiate themselves when they were carrying a dewar in a lift and the lift broke down. You won't even be aware of it if you are breathing pure nitrogen, you'll just pass out, continue breathing normally, and die.

          2. C R Mudgeon

            Even without an equipment failure, a 15-12=3% margin of error seems dangerously tight.

            1. Paul Hovnanian Silver badge

              How much time do IT people actually spend in the racks of a data center? The simple solution would be to provide them with breathing gear (an Inogen or similar would do).

    2. Flightmode

      I saw a similar demonstration some twenty-five years ago . I can't remember now if it was a match or a lit cigarette that was dropped into a faux leather sofa in a controlled room. Nothing happened at first, but after a minute or two, you could see one of the seats starting to bulge slightly. It expanded a bit, enough for most people to notice that something was going on. Those that WEREN'T paying attention to the seat cushion caught on when the damn thing ruptured and the whole room EXPLODED into fire in three seconds flat; carpet, curtains, furniture, books and all. That demo gave me a whole new respect for fires, I'll tell you that.

    3. Anonymous Coward
      Anonymous Coward

      This is a really bad idea, as impaired thinking and attention starts kicking in between 15% and 19% O2. While it won't kill you (that's typically <10%), it means you're likely to make significant mistakes in that environment, and have no idea because your thinking is impaired. You literally have no idea you're affected, because the awareness that something is wrong is one of the things being impaired. Most low-oxygen alarms kick in at 19.5% for this reason.

      Then there's monitoring and control. If you're pumping nitrogen into a room, there's likely to be some pockets of air with MUCH lower O2 concentrations, potentially lethal ones. Sensor placement and proper airflow becomes a life-and-death matter. And the sensors need routine (think every month or two) accuracy verification. I think I'd rather have the risk of fire, especially given that we're talking about rooms without much in the way of combustibles (no wood, cardboard, very little paper, etc.)

      1. C R Mudgeon

        Just out of curiosity, what's the situation in a commercial airliner? What's the effective oxygen level, given the lower air pressure they maintain?

        1. Roland6 Silver badge

          Google is probably your friend as you can select sources likely to be reliable, unlike ChatGPT.

          “ the lowest oxygen concentrations noted were about 15.2% ”

          Which is probably why airlines and insurers are wary of elder passengers:

          CONDITIONS FOR FIT TO FLY OLDER ADULTS AND AIR TRAVEL

          1. C R Mudgeon

            "Google is probably your friend"

            I did try a DDG search, but without success.

            Thanks.

    4. david 12 Silver badge

      I've been told that ICBM Submarines do run at reduced oxygen. Interesting that fire suppression is one of the possible benefits. Poor wound healing is one of the drawbacks. Also, from the same source, that "make and mend days" (when you have to do the stuff you've been putting off, instead of the stuff you'd rather be doing), are also "extra oxygen days" as a bit of reward and compensation.

      1. A.P. Veening Silver badge

        I've been told that ICBM Submarines do run at reduced oxygen.

        Why would they, the disadvantages far outweigh the advantages (as far as I can see) and the oxygen is easily extracted from seawater.

    5. Anonymous Coward
      Anonymous Coward

      I was always surprised that there wasn't a wider uptake of this sort of solution for environments like DCs.

      IBM was keen on reduced oxygen workplaces until legal told them they couldn't charge the employees extra for it.

    6. Alan Brown Silver badge

      Reduced oxygen environments are relatively common in large DCs

      They come with reduced human cognition levels too. It doesn't take much reduction in O2 concentration to fuddle your thinking

  9. keith_w

    One company I worked in the for 80's had added a computer room late in the planning stages so the only place available for it was between 2 sets of elevators on the 53rd floor. They had a Halon system which had a don't flood button, which need to be held until the cancel button was activated, on a wall about 20 feet from the phone. Unfortunately the cancel button was in a sister company's computer room 2 floors above. Fortunately we never had a fire in there.

    1. sev.monster Silver badge
      Megaphone

      And the phone to buzz them was across the room.

  10. Tim99 Silver badge

    Er P7

    The version we used also referenced urine - In this case micturition might be an appropriate response.

    1. M.V. Lipvig Silver badge

      Re: Er P7

      "So I'm relieving myself on the cabinet (with the Power OFF, of course) when the user returns to the office. ..

      Slightly shocked by the look of it.

      "Thank goodness you're here!" I cry, adlibbing like a daemon, "Quick, get a fire extinguisher - I think I'm holding it at bay!!!"

      https://www.theregister.com/2000/08/30/bofh_takes_the_piss

  11. Anonymous Coward
    Anonymous Coward

    Soldering in wrong place

    Many years ago I'd gone to a site to commission a small installation. The fitting party was finishing up when the fire alarms went off so we evacuated and dutifully closed and locked the room as it was a very secure area inside a secure site. I'm sure you've guessed it - the soldering fumes had set the alarm off (someone forgot to disable alarms) and it took them 30 mins to find someone with access to the room - fortunately it was a sunny day so standing outside wasn't too much of a chore.

  12. kezersoze

    What are the odds?

    I was involved in a maintenance window where some of the underfloor gear was being replaced. Due to the large number of floor tiles being removed at the same time the fire suppression was turned off. During one of my walks from the work area back to the control room I thought I smelled smoke. I called a couple others over to confirm my sense of smell. They agreed that something was off but we couldn't determine where it was coming from. I felt it was strongest in one are so I put my head behind a storage array and took a deep whiff, nothing. Turned the the AS400 in the next isle and did the same. Just as I was about to say I think it's this one, a huge cloud of black smoke came out. I immediately unplugged the power, ripped the network cable out of the back and we pushed that system out of the data center into the parking lot. We checked and the fire suppression system would have tripped if we hadn't disabled it for our maintenance. That data center has been running for 30 years and has never had to dump it's halon. What are the odds it would have an event in the 6 hours we had it disabled?

    1. Outski

      Re: What are the odds?

      What are the odds it would have an event in the 6 hours we had it disabled?

      Isn't it Captain Carrot who's the expert at precisely calculating million-to-one odd?

    2. Doctor Syntax Silver badge

      Re: What are the odds?

      Could your maintenance work have interfered with its cooling?

      1. kezersoze

        Re: What are the odds?

        That's a common question when I tell the story. We weren't working in the area the AS400 was in. We were working around 8 rows away so I doubt it had anything to do with it but anything is possible

    3. david 12 Silver badge

      "we pushed that system out of the data center into the parking lot"

      My dad reckoned that was standard for nitrocellulose film back in the day. The operator was in a booth hung of the back wall partly for that reason (you can still see the design on old theatre buildings). Anyway, when the film caught fire at his High School, the projectionist pushed the fire out the window, pulled the fire alarm, and called the school office on the other side of the campus.

      The school office pulled the fire alarm on the other side of the campus, and the fire dispatcher quickly formed the opinion that the entire three-story school, full of hundreds of school-age children, was on fire from one end to the other.

      Massive fire response that moved every fire company in town.

      1. Allan George Dyer

        Re: "we pushed that system out of the data center into the parking lot"

        Does that explain the overhang on the old Regent Theatre in Norwich? I've wondered why it was like that. Bit of a problem for the house underneath, if they had had a film fire. Presumably they never did, given that the house is still standing.

    4. Dan White

      Re: What are the odds?

      I'd say, given your experience, about 1 in 1...

  13. Boris the Cockroach Silver badge
    Flame

    Ahhhh the

    mis-wired switches.... and the day of fault finding one of the robots.... follow wire '94' to its relay as the symptoms of failure indicate relay '94' has gone ka-put.

    And replace relay '94' , button it all up and hit the power button........ and the fault is still there. red light flashing while your comrades all go "ha-ha"

    Open it up again... and do a full trace of the cabling and you find that whoever made the wiring loom labeled the wire '49' and its relay '49' should be replaced, sadly relay '49' do not exist and wire '49' ends in relay '84'.... where upon a deep dark desire comes over you to fly to Kobe, Japan, march into the factory where they make the robots, grab some wire and strangle the guy who makes the wiring looms.

    PS I could tell of more wiring nightmares... sadly the official secrets act stops me

    1. Anonymous Coward
      Anonymous Coward

      Re: Ahhhh the

      In industry, I was trying to trace the wiring to a measurement instrument. The red-and-black wires went into a conduit, up to the ceiling, through a small junction box, then down to the instrument - where they were white-and-black. The junction box did NOT contain a splice or connection; the wires (red-and-black) went in one conduit and out another toward the instrument.

      As is common in industry, the red wire had +24V on it, the black was the return, and the signal was the current (4-20 mA = 0-100%) going through them. Typically the instrument would be powered off of that 24V, which would throttle the current to produce the needed signal level. As I eventually discovered, this particular instrument was independently powered and providing its own 24V in addition to what was being provided on the wires, so whoever had wired it years before had done something "clever" and confusing with the wires (I think it was +24V source to negative terminal of instrument, + instrument terminal to 500 ohm resistor to ground, or something like that) to get it to work while double-powered ...

      ... inside the conduit, a foot from the instrument.

      1. NITS

        Re: Ahhhh the

        Saw something similarly strange. Ringdown panic phone system in a dollar store. Splices where they shouldn't be.

        Three phones: one at the cashwrap, one in the stockroom, one in the office. Each one's supposed to be on its own home run cable. They weren't; the office phone was tapped into the stockroom phone's cable.

        The phones have an audio connection, and a connection to the alarm panel that signals when the handset's lifted off the hook. Each phone's supposed to be on its own zone so the mothership can tell which phone's in use, even if the user's not in a position to talk. They were all wired in parallel instead. The switchhook contacts included. For the talk path, they're normally open, so that's OK. But the contacts to the alarm system are normally-closed. So the alarm wasn't getting tripped when a handset was lifted.

        Moreover, the wire colors in the blue-jacketed Cat5 cable to the cashwrap phone magically changed colors from one end to the other. I toned and traced the cable in the ceiling. Some dimbulb didn't understand the purpose of twisted-pair wiring (cancelling out common-mode noise, e.g. hum). Instead, they'd doubled the copper at a splice in the middle of the ceiling, with pairs subbing for individual wires. So it was white/blue to blue+white, blue/white to green+white, etc. I put it back to twisted-pair all the way.

        I eventually got it sorted. And labeled the snot out of the weird stuff that remained, to help the next person who has to deal with it.... who might just be me.

  14. An_Old_Dog Silver badge

    Auditors

    At an engineering consulting firm, they had a computer room with an extensively-large DECsystem10, and a Halon fire supression system. An insurance company auditor came round with a large briefcase of papers, which he laid on a table and opened, pulled out a clipboard with a checklist, and started walking around, asking some questions, checking some things, and marking up his list. A machine operator saw him standing next to the "Fire Suppression" Big Red Button and reaching out his hand. The operator only got out, "DO NOT" before the auditor pressed the button. WHOOM! The tornado-like effects created a blizzard of paper, the auditor jerked back, stumbled, and fell to the floor. He shakily picked himself back up, searched for his clipboard, hunted up a pen, made a mark on his checklist and said, "Works."

    1. jollyboyspecial

      Re: Auditors

      Reminds me of the insurance inspector who came to our works one day. Given how accident prone he turned out to be I think he was probably in the wrong job.

      The inspection was for our traveling crane to confirm repairs had been carried out correctly following an insurance claim. The injector managed to take the cover plate off the control for the travelling crane without first isolating the power. And if you guessed that received an impressive shock and that the resultant burns ended in a trip to A&E then you guessed right.

      More was to follow however. He'd parked his car right outside the loading bay in an area covered with yellow cross hatching and "No Parking" signs. While he was at A&E somebody swung out of the loading bay in a fork lift full of of castings and clipped the car. This resulted in relatively minor damge to the car. Minor that is until two heavy castings rolled off the forklift onto the car, one landed on the boot lid and the other went through the back window.

      1. C R Mudgeon

        Re: Auditors

        Carma [sic]

  15. trevorde Silver badge

    Testing to destruction

    Was told about a data centre contract for a high security installation, possibly front line military. Part of the spec was to securely erase *all* the data in 30 seconds. This was probably to prevent it falling into enemy hands. The solution they came up with was to install thermite grenades to melt the disk arrays.

    It came time for the Site Acceptance Test and the customer was told: "Yes, we can verify that requirement but it will cost you £100k to do so". The customer took them at their word.

    1. ricardian

      Re: Testing to destruction

      I've told this story before but it is relevant to the topic.

      My friend worked for a high security Government department and had managed to create a small piece of security-related hardware that matched all the specs - works with wide range of voltages both AC & DC, not polarity sensitive, not bothered by moisture, heat, vibration or being dropped from several feet onto hard surface. However, he failed on the final item in the spec - had to be easily & quickly destroyed in case of an emergency

  16. Paul Hovnanian Silver badge
    FAIL

    At a certain little ...

    ... aircraft manufacturer (I'm not even sure they are still in business) they had the forward cargo bay fire supression switch wired to the aft bay and vice versa.

    1. Anonymous Coward
      Anonymous Coward

      Re: At a certain little ...

      Sounds like a Boeing design...

  17. Sudosu Bronze badge

    Ahh the 90's

    Halon release+gas mask+glow sticks=Halon Rave

    Wait a minute, that sounds like a great band name...

  18. DS999 Silver badge

    Poor design

    Why does the human inhabited area have a raised floor? It sounds like the DC used to be larger or was planned to be larger so they built a raised floor bigger than they needed, then decided to occupy the spare area rather than do the sensible thing and use it for storage or keep it empty.

    Since it had a raised floor (which I'm willing to bet big money would allow a person to pull a tile in one area and crawl under the glass window wall into the other area...so much for security!) it couldn't use ordinary sprinklers because the water would get into the datacenter's under floor area and cause big problems.

    1. Anonymous Coward
      Anonymous Coward

      Re: Poor design

      Yes, you CAN lift the tiles out and crawl from the unlocked space under the door into the locked space. Please don't ask me why I know this is VERY possible. It was a while ago, and the statue of limitations has long passed, but still...

  19. Frogfather

    Something similar

    Something very like this happened when I worked at Scottish TV in Glasgow in the 90s except that the dump was caused by a techie working on the system crossing some contacts with his scope probe. The gas release is achieved by a small explosive device so yes, it would go Booooom!

    1. John Brown (no body) Silver badge
      Mushroom

      Re: Something similar

      Speaking of Glasgow, didn't the city council DC have a gas release, possibly a false alarm, but the shock of the explosive gas release killed most of the hard disks? I think El Reg may have reported on it.

  20. Missing Semicolon Silver badge
    Happy

    Wrong number of P's

    It's 7!

  21. IceC0ld

    Fool - Proof

    so the creation of a FOOLPROOF system can be considered, even aimed for,

    what they appear to have forgotten is that you need to make it USER PROOF :o)

    1. call-me-mark

      Re: Fool - Proof

      I've posted this before, last time it got 42 upvotes which I thought appropriate:

      "A common mistake that people make when trying to design something completely foolproof is to underestimate the ingenuity of complete fools." ― Douglas Adams

  22. an.other_tech

    Back in the late 90s

    Ah Halon, so useful yet so bad for us humans.

    Came across these systems at a Sony building, had a couple of server rooms. Never heard about it going off.

    However, did get given a halon extinguisher for home, and luckily never had to use it, although did have several CO2 extinguishers, they were used many times.

    To clear wasps, and to put out the odd accident when someone decided to leave a soldering iron on and it, cough, apparently slipped !

    Then when work decided to upgrade the whole fire alarm system that had to be linked to the fire brigade.

    Before the new system, a few call outs, when someone forgot to turn the isolate key.

    When the new addressable system went in, it went off almost daily for several weeks due to a faulty board, so when the firemen turned up on the third day, they said we would get a fine and if it kept happening, we would be removed from automatic call outs!

    So whilst the engineers were trying to sort it out, we had to isolate the auto callout link.

    But, every time the system went into fire, there were several automatic things that happened, day or night.

    Road barriers opened, magnetic locks released (and wouldn't you know it, every single one needed to be manually secured and reset, and large smoke vents opened around the site. Plus the sirens and external klaxons went off which of course woke the neighbours.

    We were never so fit as that month !

  23. Linker3000

    The newly-installed fire suppression system in the computer room next to me was test-fired with a cylinder of CO2.

    That's when the contractors found the uncoupled pipe in the false ceiling - mostly because the entire ceiling, dust, metalwork and all, was now resting across the floor and the IBM mainframe, while the air in the room was doing a good impression of the smoggiest day in history.

    At least all personnel had left the room for the test. The cleanup bill was expensive because as well as replacing the entire ceiling, the mainframe had to be cleaned and all dust filters changed. Fortunately, no asbestos found.

    1. Anonymous Coward
      Anonymous Coward

      yes normally the gas release is activated via a small cylinder of CO2 that knocks a valve open to release the gas. So as a total fail safe for working in a DC you can disconnect the CO2 cylinder as well as putting the VESDA to manual, remembering of course to reconnect it!

  24. Scene it all

    My first job out of college was in a medium sized IBM shop with the raised floors. The 'smoke' sensors in those days were optical and could be triggered by too much dust. Some of the sensors were mounted under the raised floor, where all the cables were. Their locations were marked by a small colored dot on the ceiling immediately above. Needless to say, when we had to use the "floor suckers" (big suction cups) to lift a panel and snake new cables, we were very careful.

    I was not there when it was tested, but the machine room supervisor told me the whole building shook. I think they used compressed air for the tests, as real Halon was expensive.

  25. Griffo

    Had the same thing!

    We had the exact same thing happen, except in our case it was during the handover from the supplier. He was demonstrating the yellow "disable" button by pressing it.. and boom off it went.

    We had to clear out the 20+ story building and basically it was shut down for the day until the floor was sufficiently vented.

  26. Jim Whitaker

    It is a P7 rule.

  27. Someone Else Silver badge
    Coat

    I always knew it as the P7 Rule: Proper Prior Planning Prevents Piss Poor Performance.

    YMMV, of course.

  28. ricardian

    Halon in action https://youtu.be/Jx1s8u2Mpmo

  29. Anonymous Coward
    Anonymous Coward

    Office

    My last place of work wanted to move me and my team into a spare room to give us and them more space. Turned out it was the former data centre room which, whilst very spacious still had the *mahoosive* halon tank and pipework proudly on display.

    I asked when this was going to be removed, the facilities manager told me it wasn't as it was still charged, had never been decommissioned and that moving it could be tricky. We could live with it. I was being unreasonable in not considering his generous offer to move us there, apparently....

    Anyway I told him to naff off and not long after I walked out with no job to go to before my probation was up (not for this reason, but was indicative of the quality of their leadership and employee care).

  30. Smndnm

    Different P6 down under

    That’s not the P6 we use here…

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like