back to article OVH founder says UPS fixed up day before blaze is early suspect as source of data centre destruction

French cloud provider OVH has suggested a UPS could be the cause of the fire that destroyed one of its data centres and took another three offline. Company founder and chair Octave Klaba has posted a video offering his apology and, at around the 5:50 mark said that the fire department's thermic camera images suggested two …

  1. A Non e-mouse Silver badge
    Unhappy

    UPS Engineer

    I feel sorry for the UPS engineer who's probably going to be made the scape goat for this.

    1. leadyrob

      Re: UPS Engineer

      There's an implication that the UPS wasn't adequately isolated from the server racks.

      UPSs fail and often quite spectacularly - I've been in the nervous position of having to flick the bypass breakers on one, when there was still smoke coming from the inverter capacitors.

      The big failure here seems to be a design that allowed a failed component to have such a negative effect on the rest of the data centre.

      Yes, a UPS engineer may take some of the blame, but most of it should go on the shoulders of the DC designers.

      1. Captain Scarlet

        Re: UPS Engineer

        I can't find it but when The Planet had an explosion in one of its data centres in 2008 there was a lot of recommendations then. They had an explosion in the electrical room if I recall correctly that weakened 3 walls.

        1. Anonymous Coward
          Anonymous Coward

          Re: UPS Engineer

          " They had an explosion in the electrical room if I recall correctly that weakened 3 walls."

          A structural engineer had two specialities. The first was crawling inside industrial boilers for insurance inspections on hot summer days during factory shutdowns.

          The other was designing buildings for work that had potentially explosive accidents. The trick was to make one wall deliberately weaker - so it would blow out. Then something outside to catch the debris.

      2. Anonymous Coward
        Anonymous Coward

        Re: UPS Engineer

        A customer had a UPS fire about a month ago....it wasn't the batteries. A power module failed and then caught fire and damaged the backplane as well. While there were plenty of other modules that can take the slack up, it still didn't stop the Novec from being sprayed to control the situation. So the fire suppression system did detect a fire and let the Novec go (Halon replacement). Interesting enough, they swapped out the Halon in 2020 to Novec. The UPS did go offline but was able to be brought back online after it was inspected.

        Two days prior UPS maintenance was performed, but not on any of the power modules. In this case the UPS and the batteries are not in the same room as servers, network equipment, etc.

        It really sounds like OVH just had a best setup. They can say that it was created with the standards at that time, but clearly didn't have plans to fix it either. Given that the fire took one DC out and spread to one other, it doesn't seem like a good design. I wonder how many customers were steered at having DR capabilities at the same site and have now gotten bit by that? Then again, some of the other reports said customers that used a totally different OVH site for redundancy still had issues.

  2. redpawn

    300 Cameras

    The probability of any one of them starting a fire is quite small...

    1. Ken Moorhouse Silver badge

      Re: 300 Cameras

      Where were they storing the images?

      A different vendor's cloud account?

      1. Sceptic Tank Silver badge

        Re: 300 Cameras

        Funny how that account became corrupted. What backups?

    2. Roland6 Silver badge

      Re: 300 Cameras

      However, it seems none of them were being used by smoke and fire detection video image processing software. Otherwise, they would have the relevant footage to hand... and probably also still have an operational DC...

      1. ForthIsNotDead

        Re: 300 Cameras

        If it really was a battery fire then any amount of cameras will not have made any difference. You could have been standing right in front of it with a fire extinguisher in your hand. Would have made no difference.

        The only solution to this potential issue is to locate the batteries away from the servers. I mean hundreds of feet away in a separate building. That of course escalates costs enormously, but what will be the cost of this disaster?

        1. 080

          Re: 300 Cameras

          Or at least getting rid of all combustible material from the UPS room and sealing all the wiring ducts, sounds like a cheap and cheerful thrown together bodge.

        2. Anonymous Coward
          Trollface

          Re: 300 Cameras

          > but what will be the cost of this disaster?

          Well, the servers and buildings will all be insured so... not much... maybe a month's service credits or so to each affected customer. Oh, you mean the cost to OVH's customers...

        3. Roland6 Silver badge

          Re: 300 Cameras

          >If it really was a battery fire then any amount of cameras will not have made any difference.

          Agreed, they wouldn't have stopped the original fire. However, a timely warning when things were starting to get hot (thermal imaging camera) and smolder (video camera) might have been sufficient to mitigate the worst effects ie. loss of an entire DC building and contents.

          However, looking at the few pictures of inside one of OVH's DC's it seems the walkways are quite confined with most of the space taken by servers with little room to move, wield any fire fighting equipment or if necessary beat a hasty exit.

          1. Aitor 1

            Re: 300 Cameras

            In 1999 I was an DBA for a Spanish bank.

            In the main datacenter we had the "battery section" separated from the actual servers some 25 meters and reinforce concrete walls, so in case of fire and Halon failure, the resulting explosion would go outwards, and not towards the servers or the archival storage rooms.

    3. Intractable Potsherd

      Re: 300 Cameras

      300 cameras - why do I start to think of SCORPION STARE?

      [An important maguffin in Charlie Stross' "Laundry Files" novels, in case you haven't read them.]

  3. Anonymous South African Coward Bronze badge
    Trollface

    It was the BOFH in the UPS room with the Engineer's spanner (and tampered/edited camera footage)

    1. Anonymous South African Coward Bronze badge

      Correction - deepfake video footage.

      As you were.

  4. Paul Johnson 1
    Coat

    Who, me?

    I look forwards to reading the "who, me?" story on this one.

    1. Blofeld's Cat
      Mushroom

      Re: Who, me?

      ... noticed there was a switch on the back of the UPS labelled "110 / 220", and out of curiosity ...

      1. FlippingGerman

        Re: Who, me?

        I turned on an old PC at work recently. Being young, I had no idea that was a thing, and apparently it was set to 115V. It went bang, and the magic smoke came out. Now I know.

        1. Anonymous Coward
          Anonymous Coward

          Re: Who, me?

          Someone had their PC investigated by the police on a wild goose chase. When it was returned it ran ok for a week or so - before it went flash! bang! The PSU had been switched by the police to 110v setting - presumably the "safe" voltage in their investigation lab.

  5. mark l 2 Silver badge

    Is that the plot for series 2 of Mr Robot, using the UPS to destroy the DCs?

  6. Anonymous Coward Silver badge
    Flame

    "but doesn’t say why that means SBG1 will be powered on before the undamaged SBG3 and SBG4."

    From the site layout and evolution, I'd put money on the power cables hitting SBG1 first and then going through 2 to hit 3 & 4. Presumably they need to check/replace the power feeds to 3 & 4 before risking pulling that much current through potentially damaged cables.

    1. Alan Brown Silver badge

      Yup.

      And I'll guarantee that anyone who pointed out the SPOF vulnerability was blasted with assurances "that will never happen"

      1. A.P. Veening Silver badge

        And besides that, the network cables go in the same way (and back out the same way again).

        1. Roland6 Silver badge

          Interestingly, one of the service restoration tasks was to check that the fibres hadn't been damaged. Which reminded me of the Cable Hut at Porthcurno; lesson: terminate your expensive to replace external cables some distance away from the DC's.

      2. Anonymous Coward
        Anonymous Coward

        "that will never happen""

        Customer wanted diversity routing for their external network. It was pointed out to them that there was an unavoidable national microwave link tower in the design that was a SPOF - with a national diversity upgrade planned in the near future. Came a night of unusually strong winds - and one microwave tower in the country collapsed. Yep - that one.

        Then there was the JCB in a large field that managed to dig a trench long enough to sever both fibres of a national network spine.

        Twin network cables across a stretch of sea - separated by some distance between them. During a storm a ship dragged its anchor - through both of them.

        1. Byham

          Not a lot you can do against a determined JCB (Back Hoe) driver.

          I remember a rewiring exercise at an airport and all the drawings showing all important cables like power cables and those from the radar to the control tower were below 1 metre and most at 1.5 metres. So a cable puller was sent across the airport this was a machine with a blade that could be set at variable depths and a cable attached to the end of the blade. Yes. the drawings were all wrong and the new cable layer took out every important cable crossing the airport. Lots of arm waving and red faces.

          There are times when a few men digging a trench might have been cheaper and in the long run faster.

          1. Ken Moorhouse Silver badge

            Re: all the drawings showing all important cables

            The solution there is to annotate the drawings with a cross-hatch labelled "Medieval Plague Pit".

        2. J. Cook Silver badge

          Ah, yes: the Backhoe of Death.

          reminds self to work on that deck of "network tarot" cards...

  7. sbt
    Alert

    Is there a lesson here about putting your eggs in one basket?

    I get that massive co-location makes it cheaper to provide physical and power security, and aggregate network connections. The savings are somewhat undermined when you still need to duplicate the whole setup some preferably large distance away. An argument for smaller, meshed deployments where each node has less than 10% of the load and loss of one or two nodes is not that noticeable. Then the redundancy might be as little as 20%, or less if resources are readily scalable.

    1. Len
      Flame

      Re: Is there a lesson here about putting your eggs in one basket?

      I understand why you'd put four of these buildings next to each other as you can share some resources such as perimeter fencing, entry control, security etc. I also don't think it should be such a problem as fires of this kind are very rare. What probably makes this the exception is that it looks as if the fire did not start in a cabinet (where most of the fire mitigation tech is geared towards) but in a UPS that will not have been placed on the actual floor among the racks.

      Also interesting is that two of these four building were too far away from the fire to be damaged yet went offline too.

      I think the most important lessons that should be learned from this are:

      a) house your UPSs away from the actual datacentre (minimum 50m?).

      b) house the individual UPSs away from each other (minimum 10m?).

      c) split the incoming power lines (and network lines?) away from the buildings (minimum 100m?) so even if one of the buildings is on fire and the fire brigade has to shut off power to it (or the fire has taken care of that) the other buildings can keep operating.

      1. sbt
        Mushroom

        four of these buildings

        I see your points, but I was challenging the notion of even single bulk data centre buildings with racks and floors for multiple clients. If you can go beyond the requirement for diesel generation (which is too costly unless at scale) now that UPS battery density has improved (thanks LiFePoNo) and power consumption per flop is heading down (thanks, ARM!) and provide suitably isolated battery backup for what you have, I'd be tempted to go back to on-prem if there are multiple sites I can mesh.

        You can still use cloud tools if you really love building these tottering edifices of virtualisation and containerisations, etc. I'd be happy with load balancing front-ends and log shipping; I think the complexity and fragility of cloudy systems has reached the limits and hasn't really gone beyond the 'nines' available from a well designed system with as much redundancy and simplicity as possible. Recent outages at GitHub, MS 365, and a whole pile of banks, for example.

    2. tin 2

      Re: Is there a lesson here about putting your eggs in one basket?

      I agree but I don't think that lesson is for OVH tho. They have another 24 DCs around the world in 10 locations. If you're hosting your stuff in one of their locations, your DR is automatically in another location, possibly not even with OVH, in case they have some kind of business-oriented problem.

      Or... you don't have DR, which in itself is a DR strategy.

      Does make me wonder though about all this "dual power supply this" "UPS that" "multiple internet feeds the other". Maybe that's all really really pointless, and host your stuff in at least two different places with as least commonality as possible is the only strategy.

      Put your money into making the code handle it, and have bits of the back end infrastructure die regularly as a matter of course, because it has and will be affected by single points of failure relatively regularly. So when the big disaster happens, your code is used to it.

      1. Anonymous Coward
        Anonymous Coward

        Re: Is there a lesson here about putting your eggs in one basket?

        > Does make me wonder though about all this "dual power supply this" "UPS that" "multiple internet feeds the other". Maybe that's all really really pointless, and host your stuff in at least two different places with as least commonality as possible is the only strategy.

        Even for cloud, it still has some meaning because a failure of a redundant component gives the cloud vendor the time to gracefully move your workloads onto another server and you may never notice. If the cloud vendor gave up with redundancy within the server then you would see a lot more glitches as in-flight transactions were lost and had to be recovered and re-run.

        I still recommend you host in two separate locations as well, though.

      2. ChipsforBreakfast

        Re: Is there a lesson here about putting your eggs in one basket?

        How badly impacted you were depended largely on what products from OVH you were using. Their cloud products did not seem to be badly impacted at all (at least not that we noticed) but they also provide a number of other services including hosted backup & the typical 'rent a server' product.

        We don't use the backup services so I can't comment on those but we do make extensive use of their rental servers & associated networking. A number of our customers want the benefits of not having to deal with local architecture but also aren't ready to go full public cloud and OVH's rented server/network options are an ideal intermediate step for them but as with anything you have to design it right and use it properly to get the best out of it.

        It is remarkably easy (and surprisingly inexpensive) to build a properly resilient architecture using OVH's services. By properly resilient I mean live replication across multiple DC's in multiple locations so you won't get hurt if even a whole DC goes south. Many probably don't bother, trusting that 'it'll never happen' - well, now they know better and that's not OVH's fault. Hard lesson, but the worthwhile ones usually are.

        We did have some downtime - one of our client's netblocks was announced via SBG so despite them having a functional replica site they couldn't actually reach it. It took 24 hours to get that sorted out but a sizeable part of that was down to the fact I wasn't as aware as I should have been of all the capabilities on offer to us. After OVH support helpfully pointed me to their API & told me that we can use it to control where on the OVH network our blocks are announced in near real time the client was back online in less than 10 minutes.

        We're now adding the OVH API to our config tools so we'll be ready if anything similar happens again....

        In this business the learning never stops :)

  8. Cuddles

    Normal?

    "The device was put back into service and appears to have worked normally."

    Catastrophic failure resulting in the total destruction of the data centre suggests a somewhat idiosyncratic use of the word "normal".

    1. Arthur Daily

      Cloudtrastrophie

      Value Proposition

      Datashredding

      Databurnout

      Network BBQ

      Datacentre BBQ

      Now the Exchequer can really accuse multinationals of cooking the books.

      I also suggest that the fire grew because someone tampered with the firedoors so they could go on cigarette breaks - or it allowed 'free' cooling, meaning someone pocketed an electricity bonus when power consumption was lower than estimated.

  9. steamnut

    No excuse

    The statement "SBG2 was commissioned in 2011, was built to the standards of the day," is a poor excuse. Standards change and, with so much at stake, (reputation, users data, lawsuits etc) you would have thought that every bit-barn owner would have staff whose only job is to make sure the data centres are as robust and safe as they could be. Of course, this costs money but, right now, I'm sure that OVH now wish they had adopted a better policy. Other data centres will also have had a big wake up call and will be sending out lots of internal memos...

    1. Tom Paine

      Re: No excuse

      "...as safe as they could be, consistent with the provider's business model."

      OVH were only starting to become an option last time I looked for hosting, what, a decade or so back but IIRC they were always a budget provider. Low margin business with a huge capital cost equals considerable effort to shave costs where possible. Nothing wrong with that as long as customers are aware that to some extent you get what you pay for.

    2. A Non e-mouse Silver badge
      Mushroom

      Re: No excuse

      Sure standards change, but they're rarely retrospective.

      For example:

      - What about when the standards for electrical wiring in the UK changes from Black/Red to Brown/Blue? Did everyone rip out all their old wiring overnight? Nope.

      - What about when asbestos was banned. Was asbestos removed from everywhere immediately? Nope.

      And I bet the standards for data centers aren't legal standards either, just industry ones.

  10. Barrie Shepherd

    Manager to UPS engineer;

    " You did enable the fire suppression system before your left - didn't you?"

    1. Alan Brown Silver badge

      that would make NO DIFFERENCE WHATSOEVER - when a battery pack goes up nothing will stop it burning and it will outlast any extingushants you can provide

      The only safe way to treat UPSes is to house them separately

      1. Peter2 Silver badge

        Depends.

        Restating the obvious school bits, the fire triangle is "Oxygen, Heat and Fuel" Most of the time in IT applications people use CO2 extinguishers to remove the oxygen, and yes an 8 second CO2 extinguisher will never remove the oxygen for long enough for the thing to cool down enough that it won't reignite the moment the extinguisher runs out as there is still tons of oxygen in the room that can burn.

        A buildling mounted CO2 flood might do it, if you threw enough CO2 into the building.

        Back in ye good old days of lore, there was the holy grail of fire supressents, Halon. This worked by having a funky chemical reaction with a fire and removing the heat from the fire when it's percentage in the air exceeds something like 10%, at which point the fire goes out, and stays out until somebody leaves a door open for long enough for the halon to slowly leak out. It's also not harmful to breath in.

        Flooding the room with Halon and disabling external ventilation would certainly have done it, back in the day. Unfortunately Halon is a CFC and was doing it's bit for burning a hole through the ozone layer, and so has now been discontinued and banned. There are replacements out there that work in the same way (ie Novec 1230) however IIRC they are more expensive and weigh more than Halon did. However, they are viable options for applications like data centres.

        1. Anonymous Coward
          Anonymous Coward

          > Most of the time in IT applications people use CO2 extinguishers to remove the oxygen, and yes an 8 second CO2 extinguisher will never remove the oxygen for long enough for the thing to cool down enough that it won't reignite the moment the extinguisher runs out as there is still tons of oxygen in the room that can burn.

          CO2 extinguishers also work by cooling - this is why the instructions say not to hold the horn bit - because your skin will freeze to it.

          1. DS999 Silver badge

            I recall a datacenter I consulted in about 20 years ago that had a battery room off to the side of the datacenter. The floor, ceiling and walls were all concrete, and the one door to enter the room had an air tight seal.

            I guess that was to insure that in case of fire the oxygen was limited? I assume there must have some sort of ventilation (it was a long time ago I don't recall whether there were any louvers in the walls/ceiling) but presumably that could/would be cut off in the event of a fire.

        2. Arthur Daily

          DC Fire Suppression

          Halon was also highly corrosive - at least on IBM mainframe boards. Hardly any electronics was reliable afterwards. That is why DC's went back to plain old water.

          It is unthinkable to have UPS's in the room. Generators and batteries went into the basement.

          UPS's with lithium - are these people retarded? Every bit of equipment going in has a flammability and heat generation score.

          In the 80's we even had an oscilloscope and current loops around the power cables, as a power supply on the way out, normally has more hash and noise than others. Oh, thats right, what is preventative maintenance - we expect it to be free. Well, your data, and your bits are really free now!

          1. Aitor 1

            Re: DC Fire Suppression

            The problem with acid batteries is that they release hydrogen under heavy load... but at least the fires are not as crazy as the lithium ones.

            Also, the weight is a serious issue.

            Datacentres built considering the weight of lead acid ups had crazy reinforcement on the floors that were designed for that.. and even then, there were hard limits on how many batteries you could put together..

            1. Jellied Eel Silver badge

              Re: DC Fire Suppression

              The problem with acid batteries is that they release hydrogen under heavy load... but at least the fires are not as crazy as the lithium ones.

              Hydrogen isn't really a problem because it loves to escape. So just have the battery room vented. And add some BOPs* for good measure.

              Also, the weight is a serious issue.

              Also volume. Hence why batteries are shifting to bombs.. I mean higher performance/higher energy density lithium based designs that need more TLC to prevent them from doing their own version of 'My Little Runaway'. Then releasing that pent up energy as heat, and HF, which is a lot more FUN! than plain H2. But theory generally goes that you should only need enough battery to keep things happy until the back-up generators start up. Assuming any lithium batteries aren't close enough to the diesel store to ignite that..**

              *Blow-Out Panels. They go BOP! if the venting didn't work. Bonus points if you paint 'Front Towards Sales' on the direction they're intended to blow out.

              **As seen in one DC where the battery room was next to the diesel tanks, seperated by a sheetrock wall.

    2. Anonymous Coward
      Anonymous Coward

      "Manager to UPS engineer;

      " You did enable the fire suppression system before your left - didn't you?""

      In a properly managed DC, the UPS guy doesn't even come close to fire suppression systems. It's the job of the DC security guys.

  11. This post has been deleted by its author

  12. Adam Trickett
    Mushroom

    Fumming

    At previous employer we had the lead acid bath UPS of an antique phone system flood the room with sulphuric acid fumes which cased the fire brigade to attend in haz mat suits and closed the site for a day. The following day the phone engineer came, looked at the UPS and turned it back on with the same result - though this time we turned it off pronto before having to close the whole site...!

    We also had one unrelated UPS explode and when someone opened the containment box it was just goo on the inside - not so nice...!

    Dangerous things batteries if you aren't careful...!

    1. N2
      Mushroom

      Re: Fumming

      Perhaps OVH were using Li-ion powered UPS? if there is such a thing.

      1. Alan Brown Silver badge

        Re: Fumming

        There are, but thanifully most sensible designs are pushing LiFe (which doesn't burn much)

        Even so, don't put em in your equipment racks unless they're very small and have extra containment

    2. Alan Brown Silver badge

      Re: Fumming

      The battery room of one phone exchange I worked in was a 10*10metre double-thick-walled bare concrete cell with direct external ventilation in the form of big "vent block" holes in the wall at ceiling and floor levels (the ones which look like curly decorations)

      Never had any problems with birds or rats in there - or even spider webs

  13. IGotOut Silver badge

    I was going to check...

    ...to see if the number of attacks on my websites had reduced.

    Then I remembered I had a Drop All against their ASN in Cloudflare.

  14. Alan Brown Silver badge

    This doesn't surprise me

    About 30 years ago I repaired a 6kW UPS at a broadcast radio station and put it back into service

    The following morning, the breakfast announcer walked into a building full of smoke (alarms hadn't gone off!), opened all the windows and carried on as usual. About 4 hours later that UPS went up in flames, taking out part of the building with it

    Analysis showed the smoke had come from the transformer insulation having absorbed water from sitting idle/cold for 6 months and developed a shorted turn. The burnup was due to the extra load causing the inverter section to run hot as a result and setting the batteries on fire

    Over the years I've heard many similar stories

    UPSes are nasty, dangerous pieces of kit which must NEVER be housed alongside other equipment and ALWAYS treated like they're flammable

    You're nuking futs if you put any UPS in the SAME ROOM as your other IT equipment, let alone in a rack with other kit - and as my experience showed - even the SAME Building is bad (The UPS was in a closet adjacent to electrical distribution for the building. The fire took out that closet, 3 other UPSes, the electrical distribution, audio switchroom and technician work area. Fire containment was cursory and inadequate, as are most designs around UPSes)

  15. myhandler

    OVH founder says UPS fixed up day before blaze

    **.. fixed up...**

    Ewww, what sort of English is that? Antipodean? Strasbourgian?

    1. Yet Another Anonymous coward Silver badge

      It's France, they probably mean they fixed it up with a cute little Renault car booster called Nicole

      1. WolfFan Silver badge

        Hmm. The only Renault car booster I ever encountered was called Jean le Fou by his acquaintances (he had no friends) and was neither little nor cute. He was very good at boosting cars. Not so good at getting away from the cops, though. There would be a reason why he was called the French equivalent of Crazy John. He once got a Renault Megane to exceed 200 kph in a 50 kph zone. Meganes can’t usually get to 150 kph going downhill with a tailwind.

        1. A.P. Veening Silver badge

          Meganes can’t usually get to 150 kph going downhill with a tailwind.

          They can, but they can't get up that hill themselves with the extra weight necessary.

          1. Anonymous Coward
            Facepalm

            Renaults, like UPSs, have gotten better. The Megane RS will do over 250 mph out of the showroom.

            1. Anonymous Coward
              Facepalm

              Damn US centric autocorrect

              kph not mph

            2. Tom Paine

              At 250mph, it's going to be a struggle to make the turn onto the street outside the showroom...

    2. WolfFan Silver badge
    3. General Purpose

      "what sort of English is that?"

      Well-established English.

      Collins Dictionary "If you fix something up, you do work that is necessary in order to make it more suitable or attractive."

      See also "fixer-upper" and Wallace and Grommit.

    4. Ken Moorhouse Silver badge

      Re: **.. fixed up...** what sort of English is that?

      Surely Mafioso?

  16. RM Myers
    Coat

    Serious Question

    If you had your data stored in a cloud provider's data center, and the data center burned to the ground, would it be more appropriate to say your data "went up in a cloud of smoke", or that your data "went up in the smoke of a cloud".

    Inquiring minds, etc.

  17. charlieboywoof
    Mushroom

    Was Richard Hammond in the building at the time, joking aside, that car took days to put out when it crashed.

  18. CrackedNoggin Bronze badge

    Why individual UPS and not a mega battery of the type used by utilities, e.g., "Vistra Moss Landing: 300 MW/1,200 MWh" ?

    I guess the answer would be that individual UPS cover the case of internal blown fuses - but I don't really think that's are a problem compared the regional power outages - or am I wrong?

  19. Anonymous Coward
    Anonymous Coward

    Having used ovh for dedicated servers for a while I'm not entirely surprised, they are on my list of organisations not to use, I had lots of disk failures and it used to take at least 3 days to get a replacement and that was once you'd worked out you had to email a different department rather than support (somehow you were just supposed to know and they didn't do it for you), it would then take about 6 emails with at least 3 hours between responses (and no overnight responses) before the job would finally get gone.

    It's also odd that anyone would have a catastrophic UPS issue that takes out the dc like this even if it was built in 2011.

    30 years ago (1991) I worked at a company that moved its ups and batteries from the 'dc' room (separated by a partition wall) next to us to a purpose built building and bomb proofed our windows for this very reason. Hard to understand how the risk wasn't identified 20 years later.

    1. Yet Another Anonymous coward Silver badge

      Buying small cheap per-rack UPS is much cheaper than a proper designed full-facility backup.

      Especially if you only sell 24x7 redundant power as an add-on package to some customers

      Not saying that this is what they did here, but ......

  20. Tom Paine

    Fuel

    The bit that slightly puzzles me is that there was enough flammable material to burn, once the source UPS had finished oxidising itself. Presumably the main fuel supply would be the plastic insulation on network and power cables, and obviously a DC will have a lot of both; then there are the little plastic trim panels on the front of servers, the handles of hot-swap PSUs and suchlike... but what else is there? Perhaps there's a tools cabinet with some ABS toolboxes or parts tidies... the wheels and grippy "rubber" mats on trolleys... cyanoacrylic light fittings? Does the material chip packages are formed from burn? Genuinely interested. I guess no-one here's got any personal experience of major DC fires because they seem to be so rare.

    1. A.P. Veening Silver badge

      Re: Fuel

      Going by what I read about it, most of the fuel would have been supplied by the wooden floors.

      1. Tom Paine

        Re: Fuel

        b'dum tish!

    2. David Pearce

      Re: Fuel

      Nothing on most circuit boards supports fire

      1. A.P. Veening Silver badge

        Re: Fuel

        But once they get hot enough, the circuit boards themselves will.

    3. hoola Silver badge

      Re: Fuel

      I think it really depends on the temperature of the fire. If it gets hot enough you don't need flames to bridge a gap, just radiation will do it. The real challenge with these huge data centres (and these are no massive by many now) it that the amount of power that is needed in such a small area is such that when something does go wrong it is catastrophic. The MWs of power required by these is greater than many of the gas-fired power stations that sprang up in the 90s. You are dealing with massive capacities in very dense environments and things go wrong, either through mistakes or failures. Electricity is dangerous stuff at those capacities and the UPS is the worst offender with DC on the batteries. You source current draw is only limited by the internal resistance of the battery. Things can go pear shaped very quickly at those levels so your response times for containment will be a few short minutes or even seconds.

  21. This post has been deleted by its author

  22. Charles Smith

    Flywheels are not flammable

    This is an unpaid advert for an unnamed supplier who produces stored energy flywheel UPS systems. They couldn't have received a better marketing story for free. Did they pay for the matches?

  23. m0th3r

    The DC was built like a furnace... was a matter of time

    I've made some calcs on the fire deparment's response, which would have been 15-20 minutes to site. A properly contained UPS fire, with fire-resistant compartments, could have been held back for at least 30 minutes, if not 60 or 90 depending on the FR standard you design to.

    Thus, the building was going up in flames fast, which seemed odd, until I found a video OVH posted explaining how they went "green", and designed their DCs with passive cooling. This works by letting cold air in via side openings, channeling it through the DC equipment, and exiting hot via a central chimney.

    What this means is the entire DC is a giant furnace, where a small fire in the basement gets fed with air, and rapidly takes over the entire structure, with no way to contain it. The DC pre-fire photos show lots of side vents, if they had no way to close them, or were not closed by staff at the time, would explain how quickly the whole thing went.

    OVH video here https://www.youtube.com/watch?v=uampeRtWa00 around 1:50 mark.

    1. Piro Silver badge

      Re: The DC was built like a furnace... was a matter of time

      Instead of fresh air, if they want to save energy, they should have heat exchangers cooled by outside air, but without the air mixing in the DC.

      Also, they can say it was built by the standards of 2011, but if it was built by the standards of 20 years prior, it would have had a sodding great Halon 1301 system and fresh air intakes would have been considered tree-hugger fantasy nonsense. The fire would have likely been controlled somewhat better...

      Still, battery fires aren't fun. Those big UPSs can be a serious point of failure.

  24. Sceptic Tank Silver badge
    Mushroom

    The UPS is installed

    Goodness, Gracious!

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like