Datacenter fire suppression system wasn't tested for years, then BOOM The "P6 rule" states that Proper Planning and Preparation Prevents Poor Performance. So as you prepare for the imminent weekend, The Register invites readers to run an eyeball over another edition of On Call, our weekly reader-contributed tale of tech support chores that involve another P – as in "going pear-shaped." This week …
Can confirm that when halon goes off it does sound like a small tactical bomb. Especially if you are not expecting it.
Under floor halon globe blew the dress completely off one of our female tape operators and another drove the senior operators chair across the consoles and broke his arm. Unions had hell to pay for this.
No fun was had by anyone that day.
Do you know what that fire extinguisher did? It exploded in my face. I mean, what is the point of a fire extinguisher? It sits there for months, and when you actually have a fire - when you actually need the bloody thing - it blows your head off! I mean, what is happening to this country? It's bloody Wilson!
I had to deal with that kind of system at a place I worked at many moons ago.
Every Friday we'd test it by triggering the alarm, with the added pleasure of having to answer a phone call. Fail to cancel the alarm meant a halon dump, failing to answer the phone meant a few fire engines turning up at the front door.
Never triggered either the entire time I was there (a minor miracle in itself).
Used to have a guy turn up once in a while with a cup on a stick thing to test the sensors too.
Our old fire alarm used to go off regularly. I was friends with the office manager for the facilities management dept, and we were talking about it once. Although she didn't work in out building, she had com over to monitor the drill we were having, so I took advantage of the opportunity to ask her. The alarm was triggering several times a week at that point, and we were getting to the point where the users were starting to ignore it.
She said the problem was simple. We have several buildings close by. All listed, and due to the listings, when the buildings were refitted for our use, to minimise the damage to the buildings, they went for an alarm that mostly used battery operated sensors, sounders and buttons, all connected together via radio.
The problem is that the alarms in the different buildings frequently interfered with each other, and when that happened, they tended to go off. The system was ultimately replaced. Not sure how the new ones work, but we have a lot fewer false alarms.
Of course, the new system won't stop the cause of most of the other false alarms (we have a small kitchen for employee use and people frequently burned stuff while cooking it, which tripped the smoke detectors), but working from home has pretty much stopped that.
The latest regs in Scotland require all domestic properties to have interlinked detectors - heat in the kitchen, smoke on the main living area and smoke in each circulating area on each floor. Detectors can be mains or battery powered but the batteries have to be non-removable, and detectors no more than 10 years old. CO detectors are also required near all appliances that burn gas, though do not have to be interlinked. Commercial and rental properties have had similar requirements for years, but it was added to private dwellings in 2022 - recent new-builds may meet it but there's a lot of older housing needing retrofit...
"Risk to life" ... I think you're only looking at one side of the equation. There are risks to life caused by installing too many sensors: an obvious one being that "More sensors" = "More false alarms". This makes people insensitive to them, and they might well ignore a real fire. In fact, that's a fundamental theme of the article (and the many many stories in the comments too).
So, yeah, I believe the sensors shouldn't be disproportionate to the actual risk.
Exactly what level is "Proportionate" is another question. When governments get involved in "Making people safe", I doubt very much if anybody has really done that calculation.
I have no problem with a requirement for interlinked smoke (and heat) alarms. Smoke alarms are known to have saved many lives over the years since they were commonly available for home use, and making them mains-powered, battery-backed and interlinked removes one of the biggest problems with domestic alarms - that people do have a habit of removing the batteries, or simply not maintaining the units.
We have seven alarm heads in our house altogether. The problem is the very occasional false alarm, which can often be attributed to Baby Spider Season. Ours are also interlinked with a small control panel in the utility room which has buttons for "test" and "silence", with the latter silencing all alarms except that one which triggered, meaning you can in theory trace it. In practice by the time you get to the button, the spider has moved on and the alarm isn't in "trigger" mode any more. Nevertheless it's still a lot more convenient than the way it worked in our old house, which required climbing up on steps to push the button on the alarm head itself. Even so, the children have still been known to sleep through the racket of seven sounders.
Building regulations in Wales changed from 2016 to include a requirement for sprinklers in all private domestic properties (as well as an existing requirement for multiple occupancy buildings). This one is a bit more onerous as it adds a minimum of £1,000 to the build costs, possibly substantially more if you need boost pumps or storage tanks, and in theory you need an annual drain-down and check. Problem there is that installers are stretched enough doing installations and don't have a lot of time for maintenance.
While the benefit of domestic sprinklers - particularly in smaller houses - is debatable (they are not designed to put fires out - they are designed to suppress the heat and smoke to make escape easier for longer, and possibly reduce spread until the fire brigade arrives, thus reducing overall damage) what is surprising and, I'd say, shameful is that while most new-build and re-build public buildings in Wales (schools, hospitals, libraries etc) have been required to be fitted with fire suppression systems since 2011 (I think - couldn't find a reference), proposals for similar regulations in England are constantly defeated.
M.
I didn't say I was an out-and-out advocate of sprinklers in the home, but this false alarm business must be put to bed:
This (pdf) document sets out reasonably clearly how these things work.
They are not easy to retrofit which is why the building regulations are not retrospective (building regulations very rarely are retrospective), but when major works, rebuilding etc. is carried out it is not actually terribly difficult to run the pipes. Domestic systems often use a heat-resistant plastic pipe (commercial systems are often metal pipe) which has a certain amount of flex and is glued together. You can make it even easier (and other services too) by using open web joists (just an example manufacturer) if you are replacing floor joists.
The main problem in a retrofit to be perfectly honest is likely to be upgrading the mains water connection. Leaving aside situations where a storage tank may need to be fitted and / or a booster pump, many houses in the UK are fed from the water main in the street through a 15mm Copper pipe (or ½" Lead still in some cases) which is inadequate to supply sprinklers directly. Newer houses might have 20mm or 25mm plastic supply (because 15mm can also be inadequate when there's a combi boiler and more than one bathroom), but a sprinkler system really needs a 32mm supply. Obviously the water company will charge to upgrade this (and fit a new meter) though in our area it's not an exorbitant charge.
M.
Dry, standpipe systems are and must be interconnected, all the sprinklers come on at once. Dry standpipe systems do work the other way around. Once the alarm comes on the water flows.. you can also set up pressurized dry standpipe system, where the water only comes on after the gas pressure is let out of the system there by triggering an alarm. These systems also have much lower maintenance costs and risks of leaking. Is there any particular reason you didn’t go over any of that?
Not familiar with the Welsh regulations in detail, but used to be familiar with the field.
"Dry, standpipe systems are and must be interconnected,"
Not sure that is correct, I think those can work with a single head as well. When dry the system is filled with compressed air, if the pressure suddenly drops that means a sprinkler head has actuated and then water is let into the system and escapes through the opened sprinkler head. Used in cold warehouses, etc. where a system permanently filled with water might freeze. Disadvantages: more complex and needs an air compressor to occasionally top up the pressure if that drops through small leaks.
" all the sprinklers come on at once."
That's a deluge system. Normally only used over fuel tanks, offshore installations, etc. Requires a v high water flow. Not used in dwellings, offices, etc. where a fire normally only triggers one or a few heads. The idea is to suppress a fire at its incipient stage, before it spreads. (And the water supply would be inadequate to supply a larger number of heads.)
" gas pressure is let out of the system there by triggering an alarm. These systems also have much lower maintenance costs and risks of leaking."
Why would the maintenance costs of a dry system be lower than those of a wet system? For a start you need a compressor to maintain the pressure in the system while dry. Or did I misunderstand your comment?
"Smoke alarms are known to have saved many lives "
Apply *anything* to 50 million people and hand pick the cases "anything" have "saved" lives and then mandate that to everyone because it "has saved lives" is an excuse to collect bribes from the manufacturer. No more, no less.
Every one 'saved' dies anyway. That's the harsh reality. Living longer to collect pension is a cost, not a saving. So bribes are the *only* rational reason for this BS.
Sprinklers are absolute madness, so you can bet any amount that the people mandating it got bribed very well.
Having it fitted we are looking at £500 or about.
I did install it because even if overkill, hey, it is my life, and I can afford it.
But considering how many fires are, and how many lives this would save.. I think this is overkill.
There are about 44 deaths by fires in Scotland per year.
Lets assume the US number of 55% reduction is correct.
There are 2.53 million households in Scotland.
Let's assume all those 44 deaths are residential, and no firefighters, etc (so best case scenario for alarms)
This would save 24 lives.
The battery ones last about 6 years really, not 10, and the installed ones are very expensive to install.
The cost? 8.8 million pounds per life.
While life is quite valuable, there are many things than can be done for less money to save lives, and actuaries would agree.
I would say, fix potholes, put lights on dangerous intersections, do colon cancer screenings sooner, etc etc.
If you fit them yourself, it is better value, still quite expensive, but I did it for £200
Sources:
https://www.nfpa.org/News-and-Research/Data-research-and-tools/Detection-and-Signaling/Smoke-Alarms-in-US-Home-Fires#:~:text=The%20risk%20of%20dying%20in,to%20trigger%20a%20smoke%20alarm.
https://www.nrscotland.gov.uk/statistics-and-data/statistics/statistics-by-theme/households/household-estimates/2021
I just replaced all the detectors in my home.
Mains interlinked detectors do not cost £500 per ten years.
The initial installation cost might be £500, but almost all of that is running the wires and making good afterwards. That set will last 10 years - £50pa.
The next set don't need installation because all the wiring is already done. They are literally plug and play, I didn't need any tools at all.
They're £20 each, a "typical" home has three, and they'll last another ten years.
So £6 a year per home after initial installation costs, or £632k per life once installation is amortised.
"Risk to life" is technically zero: These devices save literally zero lives and there isn't even a theoretical chance to do that: Everyone will die. Some sooner than others. Which part of that you decided to ignore?
Therefore they are almost always overkill and benefit only the company who makes them and the authority who got paid to make them mandatory: Bribery at its finest.
The *only* benefit is that some people live longer than they would have otherwise lived and there's a price tag for that. It is *a lot* less than mandated alarms in every room.
For a society that's typically a major *cost*, not a benefit : Old people get pensions and get sick a lot. I should know, I am old people.
I realise you are probably trolling, but by that logic the moment someone stops working and starts taking their pension, society should treat them as a burden rather than a benefit. Don't fit smoke alarms because if Joe Bloggs's electric blanket shorts out it's better for society that he dies in a house fire than he is able to escape without injury and live another 10, 15, 20 or more years, increasingly burdensome to "society" in general. (Let's just ignore the innumerable benefits retired people bring to society, eh?)
Have you ever watched the films or read the books Logan's Run and Soylent Green? I'm sure there are plenty of other such imaginings of dystopias created by overenthusiastically allocating monetary or societal "value" to individual lives. Brave New World springs to mind (though I'll admit it's a boring read).
Maybe… one needs to understand what risk is, I’ll help you. Risk is the situation of being exposed to uncontrolled energy, termed the hazard. Risk is quantified by the product of probability and consequence. An analogy:- the hazard is the shark in the water, the risk is going swimming in that water. The breed of shark and the chosen swimsuit/diving cage/chain mail suit all contribute to risk calculation. In the case of a house fire, the probability is low, it is not something that happens in every house every 5 years, the consequence however, is total loss of building and total loss of life. Interlinked detectors in the home have zero impact upon probability, but they do decrease the consequence by providing time and awareness to escape the hazard and save lives, pets and buildings. Not totally disproportionate in my view, the alternative is captain obvious at the funeral saying “ oh if only they had installed…..”
I see you value your life very little sir/madam. going above and beyond any rules and having an optical in every bedroom/circulation area/living area, plus heat and CO where appropriate, all interlinked, is cheap compared to losing your home/family.
A small example (I am an electrician so i fit the things on a daily basis), but my own home, 2 bed bungalow: ... Optical in each bedroom/ Multisensor* in the hallway, ionization in the attic**, and heat in the kitchen, and CO in the boiler cupboard.
*an Aico thing, it compares sensor types to determine if it's likely a false alarm or not...
** i want a fast alarm if the fire starts in the roof space, so it's one of the few places imho an ionization is still a sensible choice.
Overall system cost, about 350 quid... bought as a bundle, excluding the fixed wiring, which cost me half a day and about 50.... lifespan, 10 yrs, less than 10p a day. Fire is NOT your friend, i know we hereabouts hate being told what to do, but scotland has the right attitude on this.
/2 cents
(edit) for non electricians that 50 quid might be significantly more, so i see why scotland put it into the legal requirement...
No - it's called building regs. They don't apply retroactively.
If you build a new house, or significantly alter an existing one, then you put the detectors in at that point, or it doesn't get signed off.
Possibly also if you let a house?
And mine are interlinked by their mains supply cable (triple and earth) which is fed from a lighting circuit (because you're not likely to leave that switched off).
You can also have them linked by radio signals.
This post has been deleted by its author
The new Scottish rules apply to all domestic properties, not just new builds, from Feb 22. As pointed out however, very difficult to enforce unless there are other agencies involved...
In Feb 22 and for months after it was almost impossible to buy the approved battery type alarms (which were around £50 each) as they were out of stock everywhere. Mains alarms were much cheaper to buy, but very difficult and expensive to retro fit in most properties because of the interlinking, making the 10-year sealed battery ones a no-brainer.
Mine’s the one with a Scottish Power of Attorney in the pocket for a relative who moved to Scotland...
The point where enforcement occurs is when someone is buying the house and they do the survey. The results come back as not compliant, and then the buyer says you must fix this before i can buy the house. I doubt there will be a price negotiation "fix", e.g. well lets knock 5 grand off the price and you do it when you move in, as the mortgage company wont accept the house as security for the loan.
I have a friend who was a fireman - he said the local nurses home - quite a large block - was subject to kitchen and other false alarms.
He said the tender was crewed in record time for these - especially at night.
One imagines many sleepy nurses in the parking lot in less than normal clothing but he would not be drawn further.
For you see, a heat sensor simply won't do, you need a smoke sensor. It must be (according to the last fire inspector) immediately adjacent to any cooking surfaces. There may be a slightly different model allowed for use by gas ranges, but the break room only has microwaves, so it isn't eligible.
Instead what is three feet in front of a bank of 3 microwaves is a sensor worthy of an airplane bathroom. 6 fire engine roll-outs in four years since the remodel. 3 of which were from microwave popcorn, which will still draw a steely glare of disapproval from management to this day.
Reason be damned in the face of the National Electric Code. I think the same sensor was installed in the kitchen in my last apartment. It would reliably let your neighbors know when your bacon was just shy of the crispy phase.
My house had one of those stupid in the kitchen smoke alarms, and you don't even have to be burning food to set it off. Just a little oil vapor was enough. Being a USAian, I removed it, and false alarms went to zero. There are plenty of other smoke alarms in the house, more than enough to warn of fire anywhere in the house.
That reminds me of my time in Priory Hall of Residence at the then Coventry Polytechnic...
While no Halon were involved, the Fire Alarm going off was a classic as soon as you put 4 or more students in a room doing what students do in the evening...
It went off several time a weeks, and in some cases several times in a single night. ( when some were in playful mood. )
Every time it meant :
- evacuation of everybody willing to evacuate.
- wait for the fire trucks to fill up Priory Street.
- wait for the check that everybody has evacuated, and evacuating those that didn't.
- wait for the all clear from the firemen.
- pile up back into the buildings, using emergency exits as entrances... because they were still open. ( and thus bypassing the brand new RF card reader )
The whole setup sounds like a death trap to me:
- office also had detectors and would be flooded by Halon ? WTF ? Not so long ago, people would routinely smoke in offices !
- the whole 30 s counter and reset button (inside the area concerned by Halon) also sounds lethal to me by keeping people in the fire/halon area
- the above counter thing doesn't sound like something people wouldn't forget, 2 decades after setup
Hopefully, no-one was hurt this time, but like someone in road security in France I know, is used to say: "There are actually very few accidents vs. the amount that are just waiting to happen (due to people carelessness)".
modern fire suppressants don't use halon and DC's don't normally use or CO2 its generally either FM200 which works by absorbing heat or IG55 which works by reducing oxygen levels. Both are safe and you can be in the DC if they go off, they won't kill you! Althought weatehr you would want to be in there when it discaharges is another question! ;o)
"I've never really understood why the discharge has to be so violent. It's obviously important to get a lot of gas everywhere in the room, quickly, but would a gradual 20 seconds versus explosive 2 seconds really make much difference?"
A slow release will not flood every nook and cranny with gas. An explosive release will come close. especially if more gas is emitted than the volume of the room.
I've never really understood why the discharge has to be so violent. It's obviously important to get a lot of gas everywhere in the room, quickly, but would a gradual 20 seconds versus explosive 2 seconds really make much difference?
I'm not a particular fire specialist but a fire is detected through smoke making it from inside the box of a burning bit of equipment to outside of it, to outside of the rack, to the smoke detector. This is going to cause a delay after the fire has started at which point it's likely to start spreading. An alarm sounds giving a 30 seconds notice that you would really like to leave the area. Do you then want a nice gentle gas release in one corner risking a dense cloud of suppressant in one area of the room, which would slowly (very slowly) be pushed around with air currents and to start taking effect within a few minutes and maybe start damping down the fire within 5-10 minutes?
I mention this because your not only fighting the fire; your racing the fire brigade to doing it.
The fire brigade average response is something like 6 minutes in urban locations to 8 minutes in more rural settings. (and if your in an urban location with a DC then chances are that you are probably nearer their station so their response will be well under the average) and their usual MO for dealing with a fire is to cut the power, drag a hose in smashing down any doors in their way and then they'd spray your server racks with several tons of water per minute.
Most IT equipment does not like water much, and this sort of treatment is liable to have adverse impacts on uptime.
So if I did have the option of having a gas suppression system like this then i'd spec it with about twice the amount of fire suppressant needed for the size of the room to allow for leakage out of the room and then have an explosive deployment to cause an overpressure wave to flood the room with suppressant near instantly so that the fire is out near instantly preventing it from spreading and causing further damage, with the aim of any fire being out at about the same time the fire brigade chaps have their fire engine out of the door.
This way you might just be able to get away with turning either an individual bit of equipment or a rack off instead of the entire building. ;)
"Do you then want a nice gentle gas release in one corner risking a dense cloud of suppressant in one area of the room, which would slowly (very slowly) be pushed around with air currents and to start taking effect within a few minutes and maybe start damping down the fire within 5-10 minutes?"
While I take you point, I think he's suggesting something less violent, but still relatively violent, something akin to power of gas or powder fire extinguishers, ie the gas gets there pretty quickly, but not violent enough to itself cause damage and injury.
While I take you point, I think he's suggesting something less violent, but still relatively violent, something akin to power of gas or powder fire extinguishers
And I take that point, but how well would you expect a CO2 extinguisher to work if instead of basically shoving it into the fire before pulling the activation levers, you pulled the activation levers while standing by the door of the room the fire was in?
I'm going to go out on a limb and suggest that it wouldn't work very well.
I'd assume it would have to be designed into the room. Assuming the gas used is heavier than air, then maybe the equivalent of a sprinkler system blasting down from multiple nozzles across the room. I think the point isn't so much to target the fire as flood the room with fire suppressant and/or remove/reduce the oxygen content. Dunno, not my area, just making suggestions. Maybe it would be too expensive or some other reason it's nor been done. I'm just trying to imagine a system that does the job without itself causing damage when triggered.
"would a gradual 20 seconds versus explosive 2 seconds really make much difference"
Yes. There are plenty of videos of fire progression on YouTube. It's incredible how fast it happens once it gets started. Or, not so incredible once you realise it's an exponential increase.
With apologies to BOFH fans: Halon is not lethal upon inhalation (really; go and look it up) and if you remained in a room flooded with in for a few hours then the worst you'd get is a headache at very high concentrations.
Halon does not work by displacing all of the oxygen in the air (that's a CO2 flood and those are genuinely dangerous) Halon just has a very useful chemical reaction; which is when it has more than a certain concentration in the air then it messes with the chemical reaction that is fire by essentially removing the heat from the flame, which then puts the fire out for as long as the gas concentration remains above whatever the required percentage is. (and at which point this concentration drops hours or days later you'd hope that somebody has removed the source of ignition...)
Alas, it's equally as good at eating through the ozone layer so it's been ditched in modern years. Off the top of my head, modern equivalents are FM200 and Novatec 1230.
Halon is not lethal upon inhalation
But it makes coffee taste weird. A friend of mine had forgotten there was going to be a halon flood test so was working in the machine room when it was due. We had a sane machine manager who stuck his head round the door to check, so my friend was out of the machine room when the floods went off, but he'd left his coffee cup behind. When he went back in later to finish off what he was doing his cup was surprisingly in the place he'd left it so he automatically took a sip and promptly spat the mouthful of coffee out all over the operator's console.
- the whole 30 s counter and reset button (inside the area concerned by Halon) also sounds lethal to me by keeping people in the fire/halon area
eh? the reset button (normally by an exit) is standard issue in a DC, if you need to abort a gas release in case of a false trigger, they cost a sh1t tonne to refill those gas bottles! And modern VESDA's are VERY sensitive, hence why lots of DCs have a no cardboard box rule or anything else that might be dusty in the DC. We nearly had a false release a few years ago. A contractor was in the DC to work on a CRAC unit, SOP said that the VESDA should have been switched to manual before work took place but the contractor said he wasn't doing anything to create smoke so a colleague let him in. But what he did do was recharge the gas in the CRAC when he connected his gas tank to the system a small amount of gas was released (bit like when you connect your car tire up to pump it up) and that was enough to trigger the VESDA. Lucky we had a guy from the installers of the fire alarm system on site that morning and I was in the reception area talking to him when the house alarm went off! He immediately knew what it was and dashed up 2 flights of stairs and in to the DC just in time to press the magic button!!!!
Not so long ago, people would routinely smoke in offices
The 1970s ended 43 years ago.
I've never, in my entire working life, worked in, or even visited, an office where people smoke in the actual office. One place I used to work at had a smoking room, but even that was got rid of long before the ban on smoking in the workplace was introduced (over 15 years ago), in favour of a "bus shelter" outside.
Very late 80s, Toronto. Brit consultant, smoking his expensive smokes, feet up on table, regaling us about the expensive wines he'd had last night, while the stupid also-Brit vendor worker who would not ask for help stumbled onward, while we watched helplessly as we were not allowed to improve the situation (cuz foreigners). I think the customer allowed this to continue from the pure enjoyment of the vendor's self-inflicted pain, week after week after week.
But, yes, every job situation since then was consciously picked with smokeless in mind. You were just lucky(?)
The Japanese company I used to work for had a 'smoking canteen' and a 'non-smoking canteen'.
At times, you could barely see from one side of the 'smoking canteen' to the other, and it was a pretty small room. I left in the late 1990s, and as far as I know, this facility continued to exist until the ban kicked in during 2007.
No shelter. No seating. Very attractive in the middle of winter.
You have obviously never suffered from nicotine addiction. It's a bastard(*) – took me 11 years from first trying to give up smoking to actually succeeding (which was 30 years ago about now), and that was after a medical emergency. When I was still hooked I'd have been out there smoking even if I was accreting a thick layer of ice from freezing rain or being scorched by a solar flare.
(*) I've been told it's worse than heroin by people who have given up both.
There's a wonderful promotional video from DuPont titled "halon 1301 the instant firefighter" that features a demonstration of a lab tech in a plastic box trying to light a fire in a halon filled environment (that he's breathing in). Given DuPont's track record with worker safety, how long he survived afterwards is another question but it certainly doesn't instantly kill or impair.
The mechanism is ingenious: it reacts with burning compounds to render them non flammable. It's really annoying that it's an ozone killer because having a breathable gaseous fire suppression compound that causes no damage really was a boon. I'd have it in my house if it were possible.
Edit: sorry, this was meant to be a response to Peter2
Many years ago I attended a fascinating demonstration.
The first part of the demonstration was in a dummy room with normal atmosphere. A match was struck and dropped on a sofa. The resulting first was rapid and terrifying.
The second part was an identical chamber with an identical sofa, this time however the atmosphere was different in an unspecified way. Matches were struck repeatedly and dropped on the sofa. The matches would initially flare as normal but go out almost immediately. Even when the match was struck right against the sofa the match would go out without any more than scorching the upholstery. A blowtorch was applied to the sofa, the upholstery started to burn but the flames died as soon as the blowtorch was removed. The mystery atmosphere was revealed to be that oxygen content was sligthly reduced. IIRC the concentration was between 17 and 18%.
This was incredibly effective at preventing fires. Some stuff would burn at these concentrations of oxygen but not nearly as quickly or violently. A system to achieve this would not be difficult to build for a contained environment like a datacentre. An office building or residential space may be more difficult. And you wouldn't even need to use any noxious gasses. Nitrogen would do the job. Another good thing is that this wouldn't be harmful to humans, maybe it wouldn't be comfortable for somebody with low blood oxygen levels but I'd go out on a limb and say that firstly somebody in that condition shouldn't be at work and secondly even if they were it's not something that's suddenly going to kill them. They'd probably get short of breath and need to leave the room.
I was always surprised that there wasn't a wider uptake of this sort of solution for environments like DCs. Sure it may cost a little to run 24/7 but I would have thought preventing fires from starting in the first place was preferable to waiting for them to start and then putting them out. I remember one DC having a minor fire, not only was the power to the DC cut as a precaution but when fire fighters turned up they wouldn't let anybody back into the building until they had completed safety checks. This level of downtime was very costly. There was also the cost of getting the halon tank refiled. And the cost of getting somebody in to repair the damage done by the halon system being triggered. And of course the risk of running without a fire suppressant system for a couple of days until the tank could be refilled.
air exchange and air management systems to keep oxygen levels low I expect would cost a fortune to run! Don't forget air exchanges in buildings are part of building regulations. We are doing some load testing in our DC in a few weeks using rack mounted heaters and one thing we need to keep an eye on are CO2 levels in the DC during the test. And I'm not sure working long term in such an environment would be very healthy for you either!
Not sure if these exist on the market, but why not the same system in a rack? Environment controlled racks do exist, having one adjust oxygen level/pressure/mixture sounds doable. And you don't have to worry about human safety, as even if the system is not turned off and the closet is opened, the presumably small BTU system won't be able to affect the rest of the room.
"air exchange and air management systems to keep oxygen levels low I expect would cost a fortune to run! Don't forget air exchanges in buildings are part of building regulations. We are doing some load testing in our DC in a few weeks using rack mounted heaters and one thing we need to keep an eye on are CO2 levels in the DC during the test. And I'm not sure working long term in such an environment would be very healthy for you either!"
Where does C02 come into the above?
Reducing oxygen content would not increase the CO2 content of the atmosphere. And a healthy human would not be harmed in air with 17% oxygen content.
The air we breathe has approximately 21% of Oxygen. Oxygen is the key factor in sustaining a fire and the key factor in keep us alive too!
By removing the oxygen, we will certainly extinguish a fire, but that comes with obvious problems. How do we sustain life at the same time?
Fires need more than 15% Oxygen to combust. Anything below this level of oxygen will not be enough for a fire to sustain combustion.
Luckily, we only need 12% plus of oxygen to survive and this is where the answer lies. To extinguish a fire and sustain life, we need to reduce oxygen from 21% (assuming were at sea level) to below 15%, but not less than 12%. This will extinguish a fire and sustain life at the same time.
I actually can't believe people are discussing working in areas with reduced oxygen levels like it would be a good idea!!!!
So the issue is you are working in a DC and the Oxygen level is maintained artificially at a low level of less than 15%, and then there's a feck up with the air handling system, and the oxygen level slowly reduces further and due to the effects of oxygen starvation you're not actually aware that its happening! This is why the regulations for working in "enclosed" spaces are so strict the lack of oxygen and build up of CO2 kills people!
Exactly this. If I could upvote it more than once, I could. If you're doing anything to screw with the normal, atmospheric, levels of gases, you need to be monitoring it very well.
I recall a story a while ago, where some physicists working with liquid nitrogen managed to asphyxiate themselves when they were carrying a dewar in a lift and the lift broke down. You won't even be aware of it if you are breathing pure nitrogen, you'll just pass out, continue breathing normally, and die.
I saw a similar demonstration some twenty-five years ago . I can't remember now if it was a match or a lit cigarette that was dropped into a faux leather sofa in a controlled room. Nothing happened at first, but after a minute or two, you could see one of the seats starting to bulge slightly. It expanded a bit, enough for most people to notice that something was going on. Those that WEREN'T paying attention to the seat cushion caught on when the damn thing ruptured and the whole room EXPLODED into fire in three seconds flat; carpet, curtains, furniture, books and all. That demo gave me a whole new respect for fires, I'll tell you that.
This is a really bad idea, as impaired thinking and attention starts kicking in between 15% and 19% O2. While it won't kill you (that's typically <10%), it means you're likely to make significant mistakes in that environment, and have no idea because your thinking is impaired. You literally have no idea you're affected, because the awareness that something is wrong is one of the things being impaired. Most low-oxygen alarms kick in at 19.5% for this reason.
Then there's monitoring and control. If you're pumping nitrogen into a room, there's likely to be some pockets of air with MUCH lower O2 concentrations, potentially lethal ones. Sensor placement and proper airflow becomes a life-and-death matter. And the sensors need routine (think every month or two) accuracy verification. I think I'd rather have the risk of fire, especially given that we're talking about rooms without much in the way of combustibles (no wood, cardboard, very little paper, etc.)
Google is probably your friend as you can select sources likely to be reliable, unlike ChatGPT.
“ the lowest oxygen concentrations noted were about 15.2% ”
Which is probably why airlines and insurers are wary of elder passengers:
I've been told that ICBM Submarines do run at reduced oxygen. Interesting that fire suppression is one of the possible benefits. Poor wound healing is one of the drawbacks. Also, from the same source, that "make and mend days" (when you have to do the stuff you've been putting off, instead of the stuff you'd rather be doing), are also "extra oxygen days" as a bit of reward and compensation.
One company I worked in the for 80's had added a computer room late in the planning stages so the only place available for it was between 2 sets of elevators on the 53rd floor. They had a Halon system which had a don't flood button, which need to be held until the cancel button was activated, on a wall about 20 feet from the phone. Unfortunately the cancel button was in a sister company's computer room 2 floors above. Fortunately we never had a fire in there.
"So I'm relieving myself on the cabinet (with the Power OFF, of course) when the user returns to the office. ..
Slightly shocked by the look of it.
"Thank goodness you're here!" I cry, adlibbing like a daemon, "Quick, get a fire extinguisher - I think I'm holding it at bay!!!"
https://www.theregister.com/2000/08/30/bofh_takes_the_piss
Many years ago I'd gone to a site to commission a small installation. The fitting party was finishing up when the fire alarms went off so we evacuated and dutifully closed and locked the room as it was a very secure area inside a secure site. I'm sure you've guessed it - the soldering fumes had set the alarm off (someone forgot to disable alarms) and it took them 30 mins to find someone with access to the room - fortunately it was a sunny day so standing outside wasn't too much of a chore.
I was involved in a maintenance window where some of the underfloor gear was being replaced. Due to the large number of floor tiles being removed at the same time the fire suppression was turned off. During one of my walks from the work area back to the control room I thought I smelled smoke. I called a couple others over to confirm my sense of smell. They agreed that something was off but we couldn't determine where it was coming from. I felt it was strongest in one are so I put my head behind a storage array and took a deep whiff, nothing. Turned the the AS400 in the next isle and did the same. Just as I was about to say I think it's this one, a huge cloud of black smoke came out. I immediately unplugged the power, ripped the network cable out of the back and we pushed that system out of the data center into the parking lot. We checked and the fire suppression system would have tripped if we hadn't disabled it for our maintenance. That data center has been running for 30 years and has never had to dump it's halon. What are the odds it would have an event in the 6 hours we had it disabled?
My dad reckoned that was standard for nitrocellulose film back in the day. The operator was in a booth hung of the back wall partly for that reason (you can still see the design on old theatre buildings). Anyway, when the film caught fire at his High School, the projectionist pushed the fire out the window, pulled the fire alarm, and called the school office on the other side of the campus.
The school office pulled the fire alarm on the other side of the campus, and the fire dispatcher quickly formed the opinion that the entire three-story school, full of hundreds of school-age children, was on fire from one end to the other.
Massive fire response that moved every fire company in town.
Does that explain the overhang on the old Regent Theatre in Norwich? I've wondered why it was like that. Bit of a problem for the house underneath, if they had had a film fire. Presumably they never did, given that the house is still standing.
mis-wired switches.... and the day of fault finding one of the robots.... follow wire '94' to its relay as the symptoms of failure indicate relay '94' has gone ka-put.
And replace relay '94' , button it all up and hit the power button........ and the fault is still there. red light flashing while your comrades all go "ha-ha"
Open it up again... and do a full trace of the cabling and you find that whoever made the wiring loom labeled the wire '49' and its relay '49' should be replaced, sadly relay '49' do not exist and wire '49' ends in relay '84'.... where upon a deep dark desire comes over you to fly to Kobe, Japan, march into the factory where they make the robots, grab some wire and strangle the guy who makes the wiring looms.
PS I could tell of more wiring nightmares... sadly the official secrets act stops me
In industry, I was trying to trace the wiring to a measurement instrument. The red-and-black wires went into a conduit, up to the ceiling, through a small junction box, then down to the instrument - where they were white-and-black. The junction box did NOT contain a splice or connection; the wires (red-and-black) went in one conduit and out another toward the instrument.
As is common in industry, the red wire had +24V on it, the black was the return, and the signal was the current (4-20 mA = 0-100%) going through them. Typically the instrument would be powered off of that 24V, which would throttle the current to produce the needed signal level. As I eventually discovered, this particular instrument was independently powered and providing its own 24V in addition to what was being provided on the wires, so whoever had wired it years before had done something "clever" and confusing with the wires (I think it was +24V source to negative terminal of instrument, + instrument terminal to 500 ohm resistor to ground, or something like that) to get it to work while double-powered ...
... inside the conduit, a foot from the instrument.
Saw something similarly strange. Ringdown panic phone system in a dollar store. Splices where they shouldn't be.
Three phones: one at the cashwrap, one in the stockroom, one in the office. Each one's supposed to be on its own home run cable. They weren't; the office phone was tapped into the stockroom phone's cable.
The phones have an audio connection, and a connection to the alarm panel that signals when the handset's lifted off the hook. Each phone's supposed to be on its own zone so the mothership can tell which phone's in use, even if the user's not in a position to talk. They were all wired in parallel instead. The switchhook contacts included. For the talk path, they're normally open, so that's OK. But the contacts to the alarm system are normally-closed. So the alarm wasn't getting tripped when a handset was lifted.
Moreover, the wire colors in the blue-jacketed Cat5 cable to the cashwrap phone magically changed colors from one end to the other. I toned and traced the cable in the ceiling. Some dimbulb didn't understand the purpose of twisted-pair wiring (cancelling out common-mode noise, e.g. hum). Instead, they'd doubled the copper at a splice in the middle of the ceiling, with pairs subbing for individual wires. So it was white/blue to blue+white, blue/white to green+white, etc. I put it back to twisted-pair all the way.
I eventually got it sorted. And labeled the snot out of the weird stuff that remained, to help the next person who has to deal with it.... who might just be me.
At an engineering consulting firm, they had a computer room with an extensively-large DECsystem10, and a Halon fire supression system. An insurance company auditor came round with a large briefcase of papers, which he laid on a table and opened, pulled out a clipboard with a checklist, and started walking around, asking some questions, checking some things, and marking up his list. A machine operator saw him standing next to the "Fire Suppression" Big Red Button and reaching out his hand. The operator only got out, "DO NOT" before the auditor pressed the button. WHOOM! The tornado-like effects created a blizzard of paper, the auditor jerked back, stumbled, and fell to the floor. He shakily picked himself back up, searched for his clipboard, hunted up a pen, made a mark on his checklist and said, "Works."
Reminds me of the insurance inspector who came to our works one day. Given how accident prone he turned out to be I think he was probably in the wrong job.
The inspection was for our traveling crane to confirm repairs had been carried out correctly following an insurance claim. The injector managed to take the cover plate off the control for the travelling crane without first isolating the power. And if you guessed that received an impressive shock and that the resultant burns ended in a trip to A&E then you guessed right.
More was to follow however. He'd parked his car right outside the loading bay in an area covered with yellow cross hatching and "No Parking" signs. While he was at A&E somebody swung out of the loading bay in a fork lift full of of castings and clipped the car. This resulted in relatively minor damge to the car. Minor that is until two heavy castings rolled off the forklift onto the car, one landed on the boot lid and the other went through the back window.
Was told about a data centre contract for a high security installation, possibly front line military. Part of the spec was to securely erase *all* the data in 30 seconds. This was probably to prevent it falling into enemy hands. The solution they came up with was to install thermite grenades to melt the disk arrays.
It came time for the Site Acceptance Test and the customer was told: "Yes, we can verify that requirement but it will cost you £100k to do so". The customer took them at their word.
I've told this story before but it is relevant to the topic.
My friend worked for a high security Government department and had managed to create a small piece of security-related hardware that matched all the specs - works with wide range of voltages both AC & DC, not polarity sensitive, not bothered by moisture, heat, vibration or being dropped from several feet onto hard surface. However, he failed on the final item in the spec - had to be easily & quickly destroyed in case of an emergency
Why does the human inhabited area have a raised floor? It sounds like the DC used to be larger or was planned to be larger so they built a raised floor bigger than they needed, then decided to occupy the spare area rather than do the sensible thing and use it for storage or keep it empty.
Since it had a raised floor (which I'm willing to bet big money would allow a person to pull a tile in one area and crawl under the glass window wall into the other area...so much for security!) it couldn't use ordinary sprinklers because the water would get into the datacenter's under floor area and cause big problems.
Something very like this happened when I worked at Scottish TV in Glasgow in the 90s except that the dump was caused by a techie working on the system crossing some contacts with his scope probe. The gas release is achieved by a small explosive device so yes, it would go Booooom!
Ah Halon, so useful yet so bad for us humans.
Came across these systems at a Sony building, had a couple of server rooms. Never heard about it going off.
However, did get given a halon extinguisher for home, and luckily never had to use it, although did have several CO2 extinguishers, they were used many times.
To clear wasps, and to put out the odd accident when someone decided to leave a soldering iron on and it, cough, apparently slipped !
Then when work decided to upgrade the whole fire alarm system that had to be linked to the fire brigade.
Before the new system, a few call outs, when someone forgot to turn the isolate key.
When the new addressable system went in, it went off almost daily for several weeks due to a faulty board, so when the firemen turned up on the third day, they said we would get a fine and if it kept happening, we would be removed from automatic call outs!
So whilst the engineers were trying to sort it out, we had to isolate the auto callout link.
But, every time the system went into fire, there were several automatic things that happened, day or night.
Road barriers opened, magnetic locks released (and wouldn't you know it, every single one needed to be manually secured and reset, and large smoke vents opened around the site. Plus the sirens and external klaxons went off which of course woke the neighbours.
We were never so fit as that month !
The newly-installed fire suppression system in the computer room next to me was test-fired with a cylinder of CO2.
That's when the contractors found the uncoupled pipe in the false ceiling - mostly because the entire ceiling, dust, metalwork and all, was now resting across the floor and the IBM mainframe, while the air in the room was doing a good impression of the smoggiest day in history.
At least all personnel had left the room for the test. The cleanup bill was expensive because as well as replacing the entire ceiling, the mainframe had to be cleaned and all dust filters changed. Fortunately, no asbestos found.
My first job out of college was in a medium sized IBM shop with the raised floors. The 'smoke' sensors in those days were optical and could be triggered by too much dust. Some of the sensors were mounted under the raised floor, where all the cables were. Their locations were marked by a small colored dot on the ceiling immediately above. Needless to say, when we had to use the "floor suckers" (big suction cups) to lift a panel and snake new cables, we were very careful.
I was not there when it was tested, but the machine room supervisor told me the whole building shook. I think they used compressed air for the tests, as real Halon was expensive.
We had the exact same thing happen, except in our case it was during the handover from the supplier. He was demonstrating the yellow "disable" button by pressing it.. and boom off it went.
We had to clear out the 20+ story building and basically it was shut down for the day until the floor was sufficiently vented.
My last place of work wanted to move me and my team into a spare room to give us and them more space. Turned out it was the former data centre room which, whilst very spacious still had the *mahoosive* halon tank and pipework proudly on display.
I asked when this was going to be removed, the facilities manager told me it wasn't as it was still charged, had never been decommissioned and that moving it could be tricky. We could live with it. I was being unreasonable in not considering his generous offer to move us there, apparently....
Anyway I told him to naff off and not long after I walked out with no job to go to before my probation was up (not for this reason, but was indicative of the quality of their leadership and employee care).
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
