back to article OVH data centre destroyed by fire in Strasbourg – all services unavailable

One of cloud provider OVH’s data centres is on fire and services are severely disrupted. An incident report posted at 2342 UTC on March 9, and updated at 0353 on the 10th, states: We are currently facing a major incident in our Strasbourg datacentre, with a fire declared in the SBG2 building. Firefighters intervened …

  1. sreynolds

    Who knew data centres were tinder boxes?

    In the olden days they used to shut off the power and the oxygen before the precious hardware could burn.

    1. Mark192

      Re: Who knew data centres were tinder boxes?

      In the olden days it was /your/ data you were protecting.

      Hardware is cheap and the data isn't theirs.

      Customers look at price and tickboxes. For many, IT has been outsourced for that long there's no one in charge who even knows what questions to ask.

    2. mark l 2 Silver badge

      Re: Who knew data centres were tinder boxes?

      I believe some DC used to use halon gas to suppress the fire once all the personnel were evacuated, but I think these are now banned in the EU since halon is very bad for the environment.

      1. Barrie Shepherd

        Re: Who knew data centres were tinder boxes?

        "I believe some DC used to use halon gas to suppress the fire .........."

        There are alternatives to halon these days so it will be interesting to see if a fire suppression system had been installed.

        1. Lon24

          Re: Who knew data centres were tinder boxes?

          Fires will happen where there is a lot of electricity about.

          "Losing DC2, Mr. Worthing, may be regarded as a misfortune; to lose part of DC1 and shutting down DC3 & DC4 looks like carelessness"

          The problem is systemic. You try not to have a fire or lose a DC - but containment is the be all and end all in fire situations. I would expect to see impregnable firewalls outside the box as well as within them. Failures will happen in IT and buildings. The trick is to make sure they don't cascade.

          I had OVH on my bucket list as the first stop if I encountered issues at my current DCs. They just dropped off.

          1. Anonymous Coward
            Holmes

            Re: Who knew data centres were tinder boxes?

            Keep in mind that inexpensive cable insulation, walls, suspended ceilings, and elevated floor tiles can all be flammable. Also DCs are built to facilitate air flow.

          2. MrNigel

            Re: Who knew data centres were tinder boxes?

            .....fond memories of the late 70's getting my hands dirty mixing vermiculite to seal the cable holes between the floors in UK telephone exchanges. "SLACK BACK!!"

        2. ItsMeDammit

          Re: Who knew data centres were tinder boxes?

          The problem is of course when the staff decide to turn them off.

          This relates to the Ocado fire in Andover:

          https://www.bbc.co.uk/news/uk-england-hampshire-49071456

          "A fire that destroyed an Ocado warehouse spread because a detection system failed and staff turned off the sprinklers, a report has found. The distribution centre for the online food retailer in Andover, Hampshire, burned for four days in February. Hampshire Fire and Rescue Authority's report found there was an hour's delay in dialling 999 and staff initially tried to tackle the blaze themselves."

          1. Anonymous Coward
            Anonymous Coward

            Re: staff initially tried to tackle the blaze themselves

            I would think that unless you are all by yourself, after a couple of minutes, you have someone call in the professionals. And if you are all by yourself. You get the heck out and call the professional right away.

            1. hoola Silver badge

              Re: staff initially tried to tackle the blaze themselves

              That would suggest there was no fire suppression.

              Maybe the fire was outside that data hall hence the "tackle it themselves" and it quickly escalated. So many modern buildings appear to be death traps when it comes to construction materials these days it is not in conceivable that a simple electrical failure in a kettle could have caused this. Fire suppression using gas only works if the room integrity is maintained and are usually concerned with a fire in the protected space, not one starting out side.

              I am sure there will be much navel-gazing as people try to figure out what happened and went wrong.

        3. Phil O'Sophical Silver badge

          Re: Who knew data centres were tinder boxes?

          it will be interesting to see if a fire suppression system had been installed.

          According to at least one article in the French computer press they just had ordinary water sprinklers every two racks, no misting or gas system. Sounds more like a level of protection suitable for a shop or hotel than for a DC.

          1. CPU

            Re: Who knew data centres were tinder boxes?

            Who puts water in a DC? I mea,n with all those electricals tightly packed in racks what's the worst that can happen. Was the building purpose built as a DC?

            1. Byham

              Re: Who knew data centres were tinder boxes?

              "Was the building purpose built as a DC?"

              If it wasn't why was a large Cloud company using it as a DC?

              Why are all their DC's in the same building on the same power supply?

              Anyone that uses this company should move to another supplier and ask the questions that they should have asked before contracting with the OVH amateurs.

              1. MachDiamond Silver badge

                Re: Who knew data centres were tinder boxes?

                "Why are all their DC's in the same building on the same power supply?"

                Corporate did a study and it was more cost effective to consolidate all DC operations to one facility to allow for reduced staffing costs.

          2. Softsuit

            Re: Who knew data centres were flower boxes?

            Water systems come in different flavors. Dry and Wet. It is hard to test a dry system. When a dry sprinkler system goes off, the pressure in the pipes drops, a valve opens, and water flows through the pipes and out to extinguish the fire. The whole system is dry until an Automatic or manual valve is open. Pretty expensive bauble to have break and possible single point of failure. If you water the servers and the wrong time. You have to plant new Servers. ;-)

      2. Keith Langmead

        Re: Who knew data centres were tinder boxes?

        Not necessarily once personnel were evacuated. While at college I did work experience at a large company with a massive mainframe + couple of minis, the place was like a rabbit's warren of kit. The guy showing us the room explained about the halon system and pointed at the alarm and warning lights, and we were told that if those went off 1) there was a fire, and 2) we have a certain number of seconds to get out of that room before the halon turned on. Not it would turn on once we left, rather it was turning on in n seconds (60 I think) and we REALLY didn't want to be in there when it did!

        1. Anonymous Coward
          Anonymous Coward

          Re: Who knew data centres were tinder boxes?

          If it ever gets discharged though they won't be able to refill it with Halon. Last I looked a few years ago, you could keep existing installed systems, but if they were discharged then they had to be refilled with something else.

          One DC we used to use, used Argon gas. Part of our induction tour was the same, "If you're in here at night and the alarm goes off, you've got 60s before the place is flooded with Argon and you won't be able to breathe". During the day when more people were about, the discharge was on a manual system.

          1. JimboSmith Silver badge

            Re: Who knew data centres were tinder boxes?

            One place I visited many years ago had a limit on the number of people allowed in areas protected with fire suppression systems. This was, according to the person I went to see there a safety measure. They had masks/hoods which were supposed to allow you to exit these areas and then the building safely. There were only a certain amount in each area hence the limits on people numbers in those areas . Don't know/remember what the system was using as a suppressant. My contact said it really chilled the blood to be in one of those areas there when the alarm goes off even if it's just a test. Thankfully we didn't have to go there and there were no alarms during my visit.

            1. 080

              Re: Who knew data centres were tinder boxes?

              I would like to see the risk assessment of anywhere that allowed personnel to enter a confined space with an active fire suppression system. All HV switchrooms that I have entered, and there have been more than a few daily over many years, had an interlock system to ensure you could not enter unless the fire suppression was locked off. A bit of a pain but a lot better than trying to breath CO2.

              This would explain why the system might have been switched off during maintenance.

          2. Sean o' bhaile na gleann

            Re: Who knew data centres were tinder boxes?

            Back when I were a trainee operator, one of the first things to be rammed home to me was to keep the computer room tidy at all times - no loose items left on top of various cabinets, desks, etc.

            Not because of some OCD tendency of the chief operator, but because of the sheer hurricane-level force of the fire suppression gas being released.

            A 2400' tape reel flying across the room and hitting you in the neck would severely hamper any attempt to get out in an emergency

            1. Jellied Eel Silver badge

              Re: Who knew data centres were tinder boxes?

              Not because of some OCD tendency of the chief operator, but because of the sheer hurricane-level force of the fire suppression gas being released.

              Yup. I was working in a comms room once when there was a dump in the computer hall. There were windows between and it was pretty spectacular. I only saw a bit of the dump heading for the fire exit. Fire brigade came, no fire found but apparently triggered by a fault. Interesting part was discovering a slight snag. Hall originally had windows, but those got covered with sheet rock, so there were a few halligan marks where they probed to find windows to break so the hall could be vented. Once declared safe, the hall was in quite a mess with printer paper & manuals blown everywhere, panels blown off some of the kit.

              From chatting with the fire people, one of the biggest risks with being inside a room when that happens is the risk of getting disoriented by the sound & fury of a discharge, making it harder to find a fire exit, especially given the time constraint to find an exit before being suffocated.

              1. adam 40 Silver badge
                Flame

                Re: Who knew data centres were tinder boxes?

                I watched a halon discharge in the server room at Marconi Instruments, from the safety of outside, through the windows.

                Our poor Computer Manager was caught inside, I think he had a discharge of his own, but he did escape. It was quite hilarious though watching him running around!

                1. Roland6 Silver badge

                  Re: Who knew data centres were tinder boxes?

                  >I watched a halon discharge in the server room at Marconi Instruments

                  I assume that was Stevenage, not Longacres or Ballito?

            2. Anonymous Coward
              Anonymous Coward

              Re: Who knew data centres were tinder boxes?

              How about false floor tiles flying around?

              I wasn't working on the computer test floor then, but they tested the new Halon system with some other gas. They said that some of the floor tiles "blew up". I think I'd rather be hit by a tape reel then a floor tile.

              1. David Hicklin Bronze badge

                Re: Who knew data centres were tinder boxes?

                Or false ceiling tiles

                one place I worked at had a broom cupboard sized server room with halon, space above the ceiling tiles was shared with general offices.

          3. Anonymous Coward
            Anonymous Coward

            Re: Who knew data centres were tinder boxes?

            60s is generous. My first job the secure vault had a halon system on a 30s timer and was behind a safe door that took 20s to open. The door was required to stay shut when you were working in the room, so I don't think it would be possible to get to the door, open the door, get out, then close it behind to stop the halon getting out within the time available.

            The same job also had a bomb drill plan that was basically "get everyone in one place so its easier to account for the bodies". They tested it one Friday afternoon at 6pm after 3/4 of the people had gone home and they still couldn't fit everyone into the "safe areas".

          4. Xalran

            Re: Who knew data centres were tinder boxes?

            It's going to depend on the datacenter... and especially the room size.

            but overall if the alarm goes off you generally have between 30s and a minute to get your posterior our of the room before the world explode.

        2. Anonymous Coward
          Anonymous Coward

          Re: Who knew data centres were tinder boxes?

          60 seconds? I recall being told that if the alarm goes off, you hit the floor and cover your head immediately. (I could be wrong, that was 20+ years ago)

          1. Evil Auditor Silver badge

            Re: Who knew data centres were tinder boxes?

            60 seconds does sound like an awfully long time. My dazed memory from a large DC seems to recall 20 - when you hear the alarm, head for the exit and if you don't see the exit, hit the floor. Thinking of a domestic fire demonstration I once witnessed (less dazed memory), there might be not that much left worth extinguishing in the room after 60 seconds. I'm far from being an expert in firefighting or data centre fires but found quite convincing in-cabinet fire suppression systems with early detection: as soon as some component start to emit smoky stuff, the power to the affected cabinet is cut and the cabinet flooded with fire supression agent. I assume that is not what OVH had installed.

          2. Xalran

            Re: Who knew data centres were tinder boxes?

            You don't hit the floor, that's the best way to get killed.

            Modern fire protection systems are some inert gaz that replace the oxygen of the air... by flooding the area explosively. 800psi bottles that gets release gaz from the top and in some cases also under the raised floors and in the hanging ceiling, so laying on the floor will only get you :

            - die from suffocation,

            - die from being hit by stuff falling from the ceiling

            - die from being thrown in the air with the raised floor tiles you were laying on. and falling back with all the debris from both above and below flying with you.

            When the alarm is heard, there's only one thing to do : run to the nearest door, don't think, don't try to pick up stuff, just run.

        3. MrNigel

          Re: Who knew data centres were tinder boxes?

          Back in the early 80's when I worked on the KSA TEP4 contract a halon cylinder 'accidentally' discharged and came adrift from the wall it was securely bolted to. It wrecked a large part of the equipment floor where the new Jeddah International Switching Centre was being installed - an Ericsson AXE-10 derivative if my fading memory is correct.

      3. FatalR

        Re: Who knew data centres were tinder boxes?

        Isn't a data centres burning down worse for the environment though?

        1. Pascal Monett Silver badge

          Burning anything is generally worse for the environment.

          Of course, a chimney fire is less worse than a roomful of plastics and toxic chemicals going up in smoke.

          1. Roland6 Silver badge

            >Of course, a chimney fire is less worse than a roomful of plastics and toxic chemicals going up in smoke.

            The modern house is more flammable than a pre-war house with open coal/wood fire, because of all the

            plastic, that s why you need good smoke alarms - in the home you don't have to get out quick because the Halon, it's the toxic smoke.

            1. Loyal Commenter Silver badge

              Another reason (besides cost) to buy old "brown" furniture rather than modern chipboard and plastic tat. It takes a lot more to ignite a lump of wood than it does plastic, and the porous chipboard / MDF will act like a wick for the flammable melted plastic to add to your problems.

              1. Orv Silver badge

                In both cases your biggest problem is going to be the foam cushions. I've heard them described as "gasoline in solid form." They go up really fast and burn fiercely, in no small measure because they already have air mixed with the fuel.

              2. Claverhouse Silver badge

                Old is Better.

      4. Santa from Exeter

        Re: Who knew data centres were tinder boxes?

        We use Argonite in our IT Halls. Although there is enough oxygen in the mix that you won't actually die from lack of it, there is a 'holdoff' button next to the doors in case anyone can't get to them in the 60 seconds allowed.

        1. Spiz

          Re: Who knew data centres were tinder boxes?

          "there is a 'holdoff' button next to the doors in case anyone can't get to them in the 60 seconds allowed."

          Well if you've managed to get to the button, go through the door?

          (I know, I know - it would hopefully be someone else who presses it for you)

        2. Anonymous Coward
          Anonymous Coward

          Re: Who knew data centres were tinder boxes?

          "there is a 'holdoff' button next to the doors in case anyone can't get to them in the 60 seconds allowed."

          Which is itself overridden by the "deadlock doors and release halon now" button under the BOFH's desk.

          1. MachDiamond Silver badge

            Re: Who knew data centres were tinder boxes?

            "Which is itself overridden by the "deadlock doors and release halon now" button under the BOFH's desk."

            I thought that switch supplied natural gas to the blaze. A BOFH can have all sorts of fun when a new DC goes in. An extra box of cable here, a few spare drives there and the new flight simulator at home starts competing with a Cray.

        3. Jellied Eel Silver badge

          Re: Who knew data centres were tinder boxes?

          ..there is a 'holdoff' button next to the doors in case anyone can't get to them in the 60 seconds allowed.

          Some I've worked in also have a key or control to disarm the system when there's people inside the room. Snag then is making sure people re-arm it when they've finished. One had an interlock system where there was a padlock to lock the switch safe, and that had to be removed to exit the hall.

          1. Vometia Munro Silver badge

            Re: Who knew data centres were tinder boxes?

            Making the escape doors inoperable seems like a fundamentally bad idea. :| A previous employer had what was effectively an anti-thievery system on the fire door which prevented it being opened by not-particularly-obvious means; cue fire alarm and large crowd of people at the bottom of the stairs unable to get it open. I managed to do so (definitely a case of "stand aside, determined idiot on the scene" and lacerated my hand in the process) and complained to the building manager about it not being fit for purpose. She insisted she could see no problem with what I described. Sigh.

            1. Eclectic Man Silver badge

              Re: Who knew data centres were tinder boxes?

              Ah yes, I remember a 'test' emergency evacuation a work once.

              We were on the 2nd floor (3rd floor for USAfolk). And were all set to go. The employee designated to smash the glass tube keeping the emergency exit door shut could not be found (it was considered an honour so nobody else was allowed to touch it) for a while. Then it was decided that the padlock would be unlocked instead to avoid the broken glass and cost of replacement.

              So 5 Minutes after the alarm was sounded, the key was found and we all trooped to the bottom of the stairs to find the external door locked, and no key and no emergency override. (Oh, and no other way out, either.)

              What we learnt was that there was a serious deficiency of management, and that someone had to do something.

              What we didn't learn was whether the metal hammer provided to smash the glass cylinder keeping the door to the emergency exit shut would actually work in an emergency.

              It was the only evacuation test we did. But on the bright side we never had a fire, so that's ok.

              1. Repne Scasb

                Re: Who knew data centres were tinder boxes?

                One place I worked at the fire alarms went off, which we presumed to be a fire drill, but this time there was a chap at the bottom of the stairs blocking the main entrance telling people that they weren't heading for the proper fire exit and couldn't use it, to head for the fire escape door instead. There were a fair number of people milling around unsure of what to do, to the point the stairs were backing up. I headed for the fire exit door, broke the glass tube and opened it, we then filed out.

                There was a brief discussion that I might have done something wrong as it was "only a drill", which indeed it was, but they seemed to miss whole point of a fire drill being to ensure that people do the right thing in a real one, which just might possibly include breaking some glass to save your life.

              2. jtaylor

                Re: Who knew data centres were tinder boxes?

                A data center had a sort of 2-stage fire exit to prevent it from being propped open. Standard fire door with crash bar to unlock it, then a second door a few feet beyond. I assumed the second door worked the same (never needed to use it myself).

                Later, I talked with a fellow who used the fire exit. He forgot to grab his access card on the way out. And he wasn't carrying a cell phone. The halon didn't discharge, and someone eventually tracked down his banging and shouting and let him out of the man trap.

                1. Eclectic Man Silver badge

                  Re: Who knew data centres were tinder boxes?

                  "He forgot to grab his access card on the way out."

                  Umm, when a fire alarm sounds you are only supposed to go back for other people and essential prescription medication. It is not supposed to be worth dying for an access card or mobile phone.

                  A colleague had a heart condition, and was on medication, eventually getting a pacemaker. I did ask him if he could evacuate the building from his 4th floor office were there to be a fire. He said it would be a struggle. I then told him that if he's not fit enough to use the emergency exit in an emergency then he is not fit enough to be at work.

                  1. Stoneshop

                    Re: Who knew data centres were tinder boxes?

                    I then told him that if he's not fit enough to use the emergency exit in an emergency then he is not fit enough to be at work.

                    Depends on the location of the nearest emergency exit in relation to the desk he's usually at. At the office I would be 5 meters away from the door to the emergency stairs, and it's just one floor down to the street level exit. Even with an uncooperative leg I can manage that in less time than a fully fit person starting from the most unfavourable location on the third floor; we timed it. For disabled people on higher floors in another part of the building there are stair-chairs (and cow-orkers).

                    1. Anonymous Coward
                      Anonymous Coward

                      Re: Who knew data centres were tinder boxes?

                      >For disabled people on higher floors in another part of the building there are stair-chairs...

                      These (the stair-chairs) are a real nuisance...

                      For a client with potentially lots of low mobility people, I suggested they investigate installing either a couple of airplane slides (the building is only 2 storeys with a reasonable amount of land around it), or an auto-belay based system. Both of these would be fun and likely to be used at other times and so staff would maintain their skills...

                      1. Eclectic Man Silver badge

                        Re: Who knew data centres were tinder boxes?

                        You have to consider that you might be on the toilet when the alarm sounds, so it is not just your normal desk location that matters. In any case he would clearly have struggled to descend the many flights of stairs from the 4th floor (there were no 'stair chairs' available as far as I know).

                        I use the past tense because he died suddenly, while on holiday abroad, about 5 years ago.

                        I used auto-belay devices at the climbing wall, they are ok, but you need to ensure the harness is properly used or you'll fund yourself descending by free-fall.

                        (I'm curious for the reasons for the downvotes of my post above, but I guess they will remain a mystery.)

                        1. Orv Silver badge

                          Re: Who knew data centres were tinder boxes?

                          My guess is people saw it as discrimination against someone with a disability. (People with heart conditions need to hold down jobs too.)

                          1. Claverhouse Silver badge

                            Re: Who knew data centres were tinder boxes?

                            Possibly not if they are a danger to themselves and others.

                            1. Eclectic Man Silver badge
                              Unhappy

                              R. I. P.

                              He was, and is greatly missed. He was the person I asked when I did not know what to do (and so did quite a lot of other people). He had the knack of being a calm voice of reason and pragmatism when others were panicking. Good sense of humour and knew all the best pubs in London. Many were the times when we had a meeting in London, and afterwards he would usher us all past several perfectly respectable ale houses to the obscure one down some alley that did the best beer or cider in the locality. He once put in an expenses claim for two people for a lunch of six pints of beer and a lump of cheese!

                              When he had his pacemaker he was ok to be in the office.

                              Feeling quite sad now.

                              RIP Stewart.

                  2. jtaylor

                    Re: Who knew data centres were tinder boxes?

                    Umm, when a fire alarm sounds you are only supposed to go back for other people and essential prescription medication. It is not supposed to be worth dying for an access card or mobile phone.

                    Absolutely! I think they installed a crash bar on the exterior fire door after that incident. (The card was only to re-enter the DC after he discovered that exit was locked.)

                  3. EnviableOne

                    Re: Who knew data centres were tinder boxes?

                    In the UK: "Where someone meets the definition of a disabled person in the Equality Act 2010 (the Act) employers are required to make reasonable adjustments to any elements of the job which place a disabled person at a substantial disadvantage compared to non-disabled people"

                    In the US: The Americans with Disabilities Act, and the Rehabilitation Act, provide the same protection.

                    If said colleague was unable to evacuate from the 4th floor, due to his condition, then its the companies responsibility to make an adjustment in his work environment to allow him to continue. So move the team to the ground floor, insert fire boxes (refuge areas) in the escape path, provide assistance ...

                    Hence the downvotes

                    1. Eclectic Man Silver badge

                      Re: Who knew data centres were tinder boxes?

                      Thanks for the ex[planation. He was working in a purpose built secure environment, which had to be approved by the relevant HMG agency, so not practical to move the facility. His condition was being treated, and once he had a pacemaker he was ok. My point was that if you are temporarily so unwell that you cannot use the emergency exit in an emergency then you should not really be at work (as this potentially puts yourself and other people at unreasonable risk).

                2. ICPurvis47
                  Boffin

                  Re: Who knew data centres were tinder boxes?

                  We always knew when a fire drill was imminent, when we came in to work in the morning, all of the glass tubes would mysteriously have vanished during the night, only to re-appear the next day.

                  1. Eclectic Man Silver badge

                    Re: Who knew data centres were tinder boxes?

                    Sorry, but that sounds like cheating. How would anyone know if they could actually smash the cylinder to evacuate the building if they were never tested?

            2. Jellied Eel Silver badge

              Re: Who knew data centres were tinder boxes?

              Making the escape doors inoperable seems like a fundamentally bad idea. :|

              That's what intrigued me. It passed fire safety and worked a bit like the interlocks on some power/hazardous kit, ie being able to lock it in a safe position so someone couldn't wander along and turn the machinery on while people were working on it. Once they're done, they can unlock it and put it back into service. I guess the theory went that whoever disarmed the supression system could re-arm it and still exit, plus there were other fire doors that could still be used.

            3. Claverhouse Silver badge
              Joke

              Re: Who knew data centres were tinder boxes?

              A previous employer had what was effectively an anti-thievery system on the fire door which prevented it being opened by not-particularly-obvious means;

              .

              Same as the Shirtwaist Factory Fire.

              .

              Thank God we learnt and progressed over 100 years

              .

              .

              https://en.wikipedia.org/wiki/Triangle_Shirtwaist_Factory_fire

      5. Vometia Munro Silver badge

        Re: Who knew data centres were tinder boxes?

        Ours was one of them, back in the day. I recall one of our ops had to be rescued from the back of the computer room after a halon dump incident. I admit I rather uncharitably wondered at the time if she was having a sneaky cigarette back there...

      6. sean.fr

        Re: Who knew data centres were tinder boxes?

        Halon is gone - production stopped - Use banned with a few exceptions. There are other gases eg Novec 1230. but you can not just change the gas. You need to replace the installation as the new gasses take more space.

        The space/weight issue is why there are exceptions in transport.

      7. MachDiamond Silver badge

        Re: Who knew data centres were tinder boxes?

        "banned in the EU since halon is very bad for the environment."

        But burning plastic is ok?

      8. AlbertH
        Mushroom

        Re: Who knew data centres were tinder boxes?

        All the truly significant data centres I look after have Halon installations - there's no real substitute. We tried a CO2 flooding system, but its fire suppression properties were significantly worse than Halon.

    3. anothercynic Silver badge

      Re: Who knew data centres were tinder boxes?

      Given the photos of the data centre, I have this strange sense of déjà vu. Grenfell anyone? A pretty DC wrapped in - what appears to be aluminium cladding, possibly with a pretty, flammable, polyethylene wafer in the middle, perhaps? Maybe like that Arconix cladding... you know, the stuff that failed flammability tests... the stuff that killed 78 people while the manufacturer squirmed its way through the inquiry trying to avoid getting the blame...

      1. Anonymous Coward
        Anonymous Coward

        Re: Who knew data centres were tinder boxes?

        "the stuff that killed 78 people while the manufacturer squirmed its way through the inquiry trying to avoid getting the blame..."

        Don't forget "light touch regulation" and the "bonfire of the red tape" (c) 2014-2019, the Cameron/Johnson Chumocracy.

        https://www.architectsjournal.co.uk/archive/cameron-claims-victory-in-bonfire-of-the-building-regulations (2014)

        https://www.theguardian.com/commentisfree/2017/jun/20/brexiteers-red-tape-people-die-boris-johnson-grenfell-tower

        1. anothercynic Silver badge

          Re: Who knew data centres were tinder boxes?

          That's the least of the problems. Arconic (not Arconix) should have withdrawn its product from market, yet they continued to flog the product... for another decade. A highrise in the UAE (the Tamweel Tower) went up in flames in 2012, yet... they kept on reselling that rubbish.

    4. MooseMonkey

      Re: Who knew data centres were tinder boxes?

      I feel that some beancounters may have been involved.

      25 years ago, I used to work in large electricity substations, the large indoor ones were all protected by CO2 systems, all mechanical and a right pain in the arse to disengage when you were working in them.

      Some beancounter worked out that the CO2 systems cost so much, it would be financially beneficial to take them all out, and if a substation burned it burned.

      That bean counter didn't seem to give a shit that it would take over six weeks to rebuild such a big substation, and many electricity customers would be powered off for three or so weeks whilst alternatives were put in place.

  2. Mark192

    "Customers told to activate DR plans"

    Customer has an emergency board meeting:

    'But...but... we're paying them to look after our data so data recovery is their responsibility!' screams an about-to-be-sacked exec.

    Another customer: 'We outsourced all our IT. The backup schedule wasn't defined so our single backup dates to the start of the contract. Responsibility for restoring the backups wasn't defined and so they won't do it. We don't have anyone who even /knows/ how to do it. We're doomed'.

    1. sreynolds

      DRP: Start sending CV out. Fire up shredder. Start shredding to destroy all evidence.

      1. Anonymous Coward
        Anonymous Coward

        DRP: First sell all your shares.

        1. Charlie Clark Silver badge

          You mean once you buy the flame accelerant?

    2. Stoneshop
      Headmaster

      Loss of expertise

      We don't have anyone who even /knows/ how to do it. We're doomed'.

      Do they even have someone who knows how to spell 'backup'? And 'restore'?

      1. Glen 1
        Paris Hilton

        Re: Loss of expertise

        we have loads of backups, but restore? Whats that?

        1. Byham

          Re: Loss of expertise

          "we have loads of backups, but restore? Whats that?"

          In the 1980's worked with a software development group with its own network. Backups were done religiously to TAPE each night and each Friday a week of backups again to TAPE. They were S L O W and a short straw was drawn for who would have to stay late and reset everything after the backup was finished.

          A fumble fingered programmer deleted the wrong file one day - and 'it is OK it is backed up' - however although the tape listed the file as there - it would not restore. Nor would any of the tapes restore. Turns out that nobody had thought to test the restore would work!

          From then on the person doing the backup would be required to choose a file at random (make a safety copy) - delete the file - then restore it.

        2. Vometia Munro Silver badge

          Re: Loss of expertise

          I feel lucky that my first job (Philips, around 30 years back) had an IT department staffed by a bunch of long-time mainframe beards who had a lot of experience with that sort of thing and took it very seriously. 24 hour ops did daily, full backups, the tapes were kept in a fire safe and alternated with another set (I think they kept four sets altogether) in their other nearby data centre on a weekly basis and the recovery procedure was fully tested. An engineer visited regularly to check the tape units were in proper working order and aligned correctly. My own contribution was to change the system to use incremental backups six days a week; given that each of the dozen minis needed 2-3 QIC tapes to do a full backup the ops seemed happy with the reduction in faffing about.

          What's surprised me is that so many places I've been (or encountered) since aren't anywhere near as fastidious with their backup strategy. Considering so many organisations are just one incident away from a crisis I'm surprised disaster doesn't strike more often.

    3. Xalran

      At that level of burnage ( sorry )... DR does not mean data recovery but Disaster Recovery.

      ( and the relevant Business Continuity Plan )

  3. Anonymous Coward
    Anonymous Coward

    So there will be a dip in SPAM then; yay for us.

    1. IGotOut Silver badge

      It'll be interesting. I'll check my Cloudflare account later to see if there is a drop in attacks on my websites.

      Pretty sure there will be, hopefully Digital Ocean, will be next.

      1. Stoneshop
        Boffin

        hopefully Digital Ocean, will be next.

        "Oceans don't burn."

        "Oh really?" <ponders the amount of sodium required>

        1. Norman Nescio Silver badge

          Re: hopefully Digital Ocean, will be next.

          Sodium?

          Nah. FOOF

          See Also: Things I Won’t Work With: Dioxygen Difluoride

          The rest of Derek Lowe' blog: In the Pipeline: Tagged with - Things I Won't Work With is an entertaining read.

          1. ricardian

            Re: hopefully Digital Ocean, will be next.

            An interesting read is "Ignition!" http://www.sciencemadness.org/library/books/ignition.pdf

        2. EnviableOne
          Mushroom

          Re: hopefully Digital Ocean, will be next.

          PTF6 anyone?

    2. Disgusted Of Tunbridge Wells Silver badge

      Why is that?

      1. Potemkine! Silver badge

        OVH services are cheap, so often rented for nefarious usage.

        1. Disgusted Of Tunbridge Wells Silver badge
        2. captain veg Silver badge

          Moreover, in my experience they are completely unresponsive to spam reports.

          -A.

          1. Paul Eagles

            Yeah, they really don't give a shit.

            I reported a fake TV Licensing site hosted on an OVH VPS to them last year and they took nearly 2 weeks and several prods before they did anything about it.

        3. Anonymous Coward
          Anonymous Coward

          You have to wonder if this was some criminal enterprise's exit strategy?

          1. First Light

            For some reason I keep thinking . . . extortion attempt? Time will tell.

          2. Charlie Clark Silver badge

            You mean in response to Hafnium attacks with European IP addresses? Coincidence?

        4. Charlie Clark Silver badge

          Really nefarious people are able to do cost/benefit analyses and know which servers are best for which attack and it's still diifcult to beat the millions of compromised PCs out there…

        5. sean.fr

          And reputable folk

          Used by Stormshield, the firewall people.

          The failover to another centre was not work today.

        6. sean.fr

          And reputable folk

          used by Stormshield - the firewall people. Still down today.

    3. JimboSmith Silver badge

      One thing I've noticed during the lockdowns thankfully is I don't get the usual callers to my front door. So the religious groups, the knife sharpeners, the mobile butchers/fishmonger, political canvassers etc. have all gone. My usual ploy of coughing profusely and saying

      "Don't get too close I think it's contagious."

      Would have been much more believable though.

      1. Eclectic Man Silver badge

        Just say "I'm coughing up blood, do you think I should see a doctor?"

        That usually works.

      2. Stoneshop
        Pirate

        the knife sharpeners,

        Someone I know answered the door to a pair of Jehova's Witlesses, while in the process of turning a pig into freezable chunks, so, bloodied apron, bare chest and still holding the knife.

        They didn't bother him ever again.

        No need for knife sharpeners' services either.

        1. Claverhouse Silver badge

          Re: the knife sharpeners,

          "I'm afraid my wife is out."

    4. Anonymous Coward
      Anonymous Coward

      "So there will be a dip in SPAM then; yay for us."

      I came here to say the same thing.

      Now if we could just firewall off all of Arizona the world would be 99.999% spam/malware free!

      (GoDaddy and NameCheap)

      1. Anonymous Coward
        Anonymous Coward

        GoDaddy is very distant from NameCheap. GoDaddy has more domains under their control and yet NameCheap is almost always ranked #1 for illegal activity by Spamhaus. If NameCheap is number 1 in a quarter, they will be again by the next one.

        NameCheap does nothing about abuse reports. I just quit reporting them and go directly to the gTLD owner and they usually take care of it. ICANN does nothing about NameCheap and the failures of NameCheap to comply with ICANN requirements.

        The world would truly be a better place without NameCheap. Then again their "abuse" team that they like to call their legal team is in Eastern European countries.

        https://www.spamhaus.org/news/article/800/spamhaus-botnet-threat-update-q2-2020

        GoDaddy isn't even on the list, OVH is though.

        namecheap.com Namecheap, as detailed earlier in this report, is the most abused domain registrar when it comes to botnet C&Cs. Sadly, Namecheap also managed to get into the Top 10 list of Networks hosting the most botnet C&Cs in Q2.

  4. Ken Moorhouse Silver badge

    Did we really need all that data anyway?

    Will probably be a discussion in many board rooms later today.

  5. A Non e-mouse Silver badge
    Facepalm

    DR Plans

    This is when you find your DR data center is next door...

    1. Len
      Flame

      Re: DR Plans

      I did wonder about that. But considering OVH has nearly 30 DCs all over the world I bet that having your data in Strasbourg and your back-up DC in Strasbourg too sounds intuitively wrong enough for most people to have selected a different city for disaster recovery. Even if you did not know that all four Strasbourg DCs are effectively next to each other.

      1. Roland6 Silver badge

        Re: DR Plans

        >But considering OVH has nearly 30 DCs all over the world

        But only 11 physical locations/sites...

        >Even if you did not know that all four Strasbourg DCs are effectively next to each other.

        From the reports and pictures, it would seem the data centres are merely separate adjacent halls.

        However, it is not clear whether OVHCloud had configured or sold the cluster of data centres as some form of DR offering. Just goes to show that even with cloud, physical location of your live and DR servers is still a necessary consideration.

        1. ChipsforBreakfast

          Re: DR Plans

          OVH has some very clever networking capabilities that in most situations make life much easier for us poor buggers who have to make it all work without upsetting the bean-counters too much.

          Said capabilities make it easy(ish) to have a primary site in one DC and replica in a completely different country. You'd think that would be enough DR for just about any reasonable situation... except...

          That same clever networking is the Achilles heel. Long and short of it, it's broken. That DR site is now sitting there, up and laughing at my futile attempts to get it to talk to the world.

          OVH support, naturally enough, can't give any sort of clue when this is likely to be fixed. I can't blame them for that - I really wouldn't want to be working for OVH's network engineering team right now.

          Next up, restore from backup - to another provider!

          Lesson : there is NEVER enough DR.

          1. Roland6 Silver badge

            Re: DR Plans

            >That DR site is now sitting there, up and laughing at my futile attempts to get it to talk to the world.

            If you want to scare your colleagues and managers, you could try asking: how long (with no IT and thus no business being conducted) before the company is unable to recover. Depending on the business, the window is to get some form of IT back up and running again can be quite short and may also vary depending on when in the accounting month/period it occurs.

            Useful information to have when trying to get sign-off for your seemingly expensive business continuity plans...

            1. Anonymous Coward
              Anonymous Coward

              Re: DR Plans

              It's the first question I ask. Usually followed by silence.

              1. Eclectic Man Silver badge

                Re: DR Plans

                I understand that the UK's Environment Agency's flood warning system had a DR recovery time of a maximum of 2 hours, because when you need to know where the flood is going to be you cannot wait until tomorrow to warn the emergency services.

            2. The Pi Man

              Re: DR Plans

              Question 2 to follow up is, “Can we test that scenario?”

            3. ChipsforBreakfast

              Re: DR Plans

              We have a (tested) 48-hr RTO for a full bare-metal restore from backup to an alternate provider (Azure in this particular case since these are all Windows VM's). Actually activating that process is expensive so the decision to do it will be made tomorrow depending on whether or not we can get any traction with OVH support.

              To be honest, if we DO have to do it we're unlikely to return to OVH. Accidents happen, what's a lot more important to us is how the aftermath is handled (or not handled...)

        2. Mog_X

          Re: DR Plans

          Our two main DCs were until very recently only a mile or so apart, but if something took out both, it would also mean that a good chunk of west London was also on fire, so some lost data would have been the least of our problems....

          1. Anonymous Coward
            Anonymous Coward

            Re: DR Plans

            "Our two main DCs were until very recently only a mile or so apart, [...]"

            IIRC a customer had two DCs - but both in line with the local airport's approach path.

            Another customer acquired a custom-built DC that was on a large river's flood plain. The exterior low-walled compound had large electric water pumps in the event of a storm surge. It was discovered that the pumps were only on the mains electricity supply - not on the site's emergency diesels.

  6. KittenHuffer Silver badge
    Flame

    What would be really useful ....

    .... is if after this we actually get to learn the stories of some of the businesses that go to the wall due to this fire.

    Then I can put those stories in front of Manglers in the future when the demands to push everything into the Cloud come along.

    Too obvious! ------------->

    1. TimMaher Silver badge
      Facepalm

      Re: What would be really useful ....

      Many years ago, a professional body for the insurance industry, published a heavily reported paper on the outcome of a fire in a business.

      They said that over 75 percent folded.

      Now the fire is in someone else’s business but the outcome may well be the same.

      Just guessing.

      1. Anonymous Coward
        Anonymous Coward

        Re: What would be really useful ....

        They said that over 75 percent folded.

        Well, "busting out the joint" kinda requires one to collect the fire insurance on the way out. The fire can conveniently consume all those goods that has really gone "walkies", and, with luck the accounts too!

    2. Stoneshop
      Mushroom

      "push everything into the Cloud"

      The usual: specify type of cloud.

      1. Loyal Commenter Silver badge

        Re: "push everything into the Cloud"

        In this instance, it looks like a cloud of smoke...

        Apologies if that was what you were getting at and I explained your joke...

  7. Anonymous Coward
    Anonymous Coward

    English clause ordering

    "data centre destroyed by fire in Strasbourg" implies that a fire that was in Strasbourg happened to destroy a data centre.

    "data centre in Strasbourg destroyed by fire" implies that a data centre which happens to be in Strasbourg was destroyed by fire.

    Unless this fire was more widespread (which is not what your report says), you wanted the second of those, not the first.

    -- UK Pedants unite!

    1. Amentheist
      Trollface

      Re: English clause ordering

      I think the important message still got through, that fire was the reason for the destruction of a data centre.

      1. Anonymous Coward
        Anonymous Coward

        Re: English clause ordering

        In Strasbourg.

    2. Graham Cobb Silver badge

      Re: English clause ordering

      Thank goodness you are not editing El Reg!

      Headlines do not follow normal English conventions. Sometimes for brevity, other times (like this) to make it more interesting. You can't deny that "data centre in Strasbourg destroyed by fire" would NOT be a very good headline. Just deleting the "in Strasbourg" would have improved it a little but the chosen headline is much better than either.

      The article then makes it clear the fire was not the whole of Strasbourg.

      Writing isn't about rules: it is about goals and how best to achieve them. Violating rules requires good reasons, but it occurs very frequently for emphasis and other communications reasons. Fortunately El Reg has professional editors who understand that.

      1. General Purpose

        Re: English clause ordering

        Yes. For IT professionals, the most important thing is that an OVH data centre's been destroyed. They may be relying on it, their suppliers, partners or clients may be relying on it. Geography's secondary these days.

      2. MarkSitkowski

        Re: English clause ordering

        Talking of headlines, David Frost had a competition many years ago for the world's most eye-catching.

        Number 1: "Archduke Ferdinand found alive. First World War a Mistake"

        Number 2: "Pope Elopes"

        1. EnviableOne

          Re: English clause ordering

          still doesnt beat "Super Cali go Ballistic Celtic are Atrocious"

    3. Loyal Commenter Silver badge
      Trollface

      Re: English clause ordering

      Unless there's more than one Strasbourg, and you're referring to the one which has been destroyed by fire...

      Guess what, natural language is ambiguous (which is why you'll never get a true natural language parser that doesn't also understand context and cultural references). Move on.

    4. randon8154

      Re: English clause ordering

      Just by curiousity, as english isn't my mothertong

      "data centre destroyed by fire in Strasbourg" or "data centre in Strasbourg destroyed by fire" or "Strasbourg, data center destroyed by fire" or "By fire, a data center destroyed in strasbourg", isn't exactly the same ?

      english is pretty easy, even with Grammatical / conjugated / orthography error, a sentence is most of the time understandable, unlike some other language where strict rules is mandatory to be understood.

  8. Anonymous Coward
    Anonymous Coward

    "Yes sir, I'm downloading the DR plan as we speak..."

    "Oh, it looks like it's not available..."

    1. Eclectic Man Silver badge

      Re: "Yes sir, I'm downloading the DR plan as we speak..." and I'll just phone up ...

      I once visited a sensitive HMG site. One of those where you park your car outside the gate, go to the gatehouse and tell them who you are and who you are visiting and they then check with them and if ok, let you (but not your car) in. Unfortunately their computer system was down at the time. The phones were working, but the directory ... was on the computer system, so unless you knew the extension number if your contact you couldn't get in.

      :o)

      1. Pascal Monett Silver badge

        Then again, if their computer system is down, how are you going to work ?

        1. Eclectic Man Silver badge

          Work

          I was there for a meeting to interview them and discuss the project, so working would not have been problem. Also the really important equipment (coffee machines, microwave ovens) were still operational.

          I am also able to perform the arcane and obscure 'witchcraft' known as 'handwriting', so was even able to take notes on something called 'paper' which I carried with me*.

          *(Yes, I am a bit of a smart-arse, sorry, but it comes in handy sometimes.)

    2. Anonymous Coward
      Anonymous Coward

      Re: "Yes sir, I'm downloading the DR plan as we speak..."

      Ah yes ...

      a long time ago in DEC-land there was a full power outage to the data center and they discovered that all of the backup and emergency procedures were documented... well you can guess where

      Then there was the fire that took out the building in Basingstoke and the machine room with it, though the fire was not connected with anything IT.

      Followed by the data centre sales and planning departments holding their heads in their hands as the IT ops manager, when asked about DR was quoted as saying "we have the no-plan plan" ...

      We'll probably find that it's still popular

      1. Eclectic Man Silver badge

        Re: "Yes sir, I'm downloading the DR plan as we speak..."

        Re: "we have the no-plan plan"

        I once did a BS7799 / ISO 27001 review of a small organisation (Govt. Dept.). They had a floor in an office run by another, much bigger, Govt. Department. When I asked what would happen if the building was permanently unavailable (e.g., a fire) they were of the opinion that the larger Department would find them somewhere. When I asked if that agreement was written down anywhere, they said it was not, they had a 'gentlemans' agreement.

        Strangely, my report was not as complimentary as they were hoping.

        1. Roland6 Silver badge

          Re: "Yes sir, I'm downloading the DR plan as we speak..."

          >When I asked what would happen if the building was permanently unavailable ...

          Yes business continuity is difficult, it also seems to fall between traditional IT and "the business". I expect lockdown has enhanced many companies ability to carry on working whilst the office is unavailable.

          1. Stoneshop
            Facepalm

            Re: "Yes sir, I'm downloading the DR plan as we speak..."

            I expect lockdown has enhanced many companies ability to carry on working whilst the office is unavailable.

            Depends on how much of their infra is inside that office, starting with LDAP/AD servers, VPN endpoints etcetera ...

    3. Jim 15

      Re: "Yes sir, I'm downloading the DR plan as we speak..."

      When I was IT Manager, my business insisted all of us with a role to play in a DR event kept paper copies of the DR plan at our homes. Sometimes the quill and parchment still have relevance.

  9. Hans Neeson-Bumpsadese Silver badge

    The SBG2 data centre in better days

    I was a bit surprised to see such obvious branding on the outside of the building. I thought the practice was to house these things in fairly anonymous-looking buildings so as not to advertise the fact it is what it is.

    1. Anonymous Coward
      Anonymous Coward

      Yeah, I worked at a bank where the new (in 2005ish) data centre deliberately didn't have any company logos on it. It just looks like a warehouse building, although the gated security and barriers give a bit of a hint that it's something more serious. Thousands of people will know where it is (most current and ex IT bods, delivery drivers, 3rd party engineers, etc, etc, etc), so the obscurity is paper thin, but there you go.

      1. rg287

        Yeah, I worked at a bank where the new (in 2005ish) data centre deliberately didn't have any company logos on it. It just looks like a warehouse building, although the gated security and barriers give a bit of a hint that it's something more serious.

        There's a bank DC up the road from me. The casual observer wouldn't know what it was (or at least whose it is if they made a guess from the cooling gear visible on Google Earth - at street level it's all very discreet and screened off with embankments and planting).

        I have no business knowing either, I'm just a nosy git who poked around to see who owned the address!

      2. Claptrap314 Silver badge

        Lambs to slaughter

        Anyone naughty who cares about physical location has as their day job scouting buildings for potential interesting locations. Otherwise nondescript buildings with any sort of heightened security measures scream "Find out what's going on here!"

        Obligatory Schlock

    2. Jim Willsher

      The data centre used, until 3-4 years ago, is housed in a very tatty part of Milton Keynes, on an industrial estate (yeah, industrial estate in MK doesn't exactly narrow it down). To gain access it's a side door, like you'd have to get into your garage, with a letterbox and a doorbell. The little nametag under the doorbell simply says "private morgue".

      1. Stoneshop
        Trollface

        The little nametag under the doorbell simply says "private morgue".

        Where hardware goes to die.

        1. KittenHuffer Silver badge

          Re: The little nametag under the doorbell simply says "private morgue".

          Hardware?!? Don't you mean Wetware?

          1. Eclectic Man Silver badge
            Coat

            Re: The little nametag under the doorbell simply says "private morgue".

            "Silicon Heaven"

            1. The Oncoming Scorn Silver badge
              Pint

              Re: The little nametag under the doorbell simply says "private morgue".

              So that's where all the calculators go!

  10. Security nerd #21

    Website attack levels

    I imagine a lot of websites will now have a substantial decrease in traffic, from all the botnets these guys host ...

    1. Anonymous Coward
      Anonymous Coward

      Re: Website attack levels

      ZING!!!!

  11. Anonymous Coward
    Anonymous Coward

    The Cloud

    It's an easy and quick way to migrate all your data to The Cloud.

    Oh wait you don't want your data go up in smoke? Too bad.

    1. Ochib

      Re: The Cloud

      Offprem ≠ Cloud

      1. Anonymous Coward
        Anonymous Coward

        Re: The Cloud

        All the data is in the cloud now, a black cloud.

        1. A.P. Veening Silver badge

          Re: The Cloud

          And the silver lining is formed by all the botnets that went up in smoke.

  12. Anonymous Coward
    Anonymous Coward

    activate DR

    https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcSyiYX0iObudk_G-6fFPQCrx1DklUgsUhmCLA&usqp=CAU

  13. Anonymous Coward
    Anonymous Coward

    something wrong

    Looks like the building was dedicated to the DCs.

    Something went very wrong there.

    Vesda systems are normally here to trigger alarms way before everything is in flame.

  14. steamnut

    welcome to the new world...

    I have avoided the cloud like the money-raising pandemic that it is. The reason? All of your eggs in someone else's basket is risky.

    This fire will send a lot of ripples into a lot of ponds. As we speak there are board-level questions being asked about this incident such as "does it affect us?", "do we have backups?", "are there alternatives?", "are we insured", "is this what 99.99% availability looks like?" and "what happened to our own data centre - and it's staff?" Sadly, some business may even fail.

    Sooner or later something like this was going to happen. I like my spinning rust and noisy power supply fans where I can see and hear them!

    1. Anonymous Coward
      Anonymous Coward

      Re: welcome to the new world...

      Sounds exactly the same as if you have your own data centre and it goes up in flames? Needing backups and DR sites isn’t a new thing.

      1. Doctor Syntax Silver badge

        Re: welcome to the new world...

        The difference is that the DR centre is only activated when you need it, not all the time. One of the advantages of being prepared in that way is that you should have a regular rehearsal event as part of the contract. That enables you to test your backup-recovery cycle.

        I worked for a client who simply had a hot standby at the opposite end of the factory. In the event of a fire big enough to destroy both ends of the factory losing IT was the least of their worries.

        1. Roland6 Silver badge

          Re: welcome to the new world...

          >The difference is that the DR centre is only activated when you need it, not all the time.

          The only way to ensure the DR centre is there when you need it is to regularly use it.

          I panicked one customer by proposing to run a new system in active-active-standby mode with additional capacity in each active DC to handle the full load, with the standby largely being offline capacity to be spun up if an active DC had failed. They, having mostly come from a batch background (hence had operated a pair of traditional lights on and lights off DC's) hadn't fully appreciated the service level issues associated with real-time processing of large numbers of online customer transactions and having a relatively minor backend outage being made visible to customers.

    2. Anonymous Coward
      Anonymous Coward

      This might have been OVH cloud...

      ...but is more likely to have been colo, or customer datacentres.

      So, your schadenfreude and cloud-phobia is misplaced, as customers (like you) will have rented cages and rooms in those buildings too.

    3. Dave 126 Silver badge

      Re: welcome to the new world...

      > I have avoided the cloud like the money-raising pandemic that it is. The reason? All of your eggs in someone else's basket is risky.

      Er, putting some *clones* of your eggs in some distant baskets doesn't preclude you from keeping your own egg baskets.

      You say you want to see your baskets at all times, but that limits you to keeping all your baskets where they are vulnerable to a single event, such as flooding, fire, stoat invasion, etc etc

      1. FatalR

        Re: welcome to the new world...

        I've always felt that a good setup would be to have most of it local, with "backup" going onto 3rd party services which can provide you DR, and for a bit of flexibility in scalability/high load etc.

        Most people NEED some kind of DC setup - though don't want to have to deal with over and under provisioning problems.

      2. KittenHuffer Silver badge
        Joke

        Re: welcome to the new world...

        We always used to keep a set of trained attack dogs onsite! It was very successful at repelling the stoat invasions! So successful that not once did we have any downtime that was attributed to stoat, ferret or mink!

        1. Roland6 Silver badge

          Re: welcome to the new world...

          >We always used to keep a set of trained attack dogs onsite! It was very successful at repelling the stoat invasions!

          A grain of truth in that. Living in the countryside we have lots of small furry visitors. However, since we've had a Jack Russell, we've not seen any live ones and in recent years only occasionally a dead one. An additional advantage, the neighbours cats only enter the garden in a dire emergency with no intention of lingering...

        2. Loyal Commenter Silver badge

          Re: welcome to the new world...

          Nothing managed to weasel its way in?

    4. Hans Neeson-Bumpsadese Silver badge

      Re: welcome to the new world...

      All of your eggs in someone else's basket is risky.

      It depends on how highly you value your eggs. There are cloud providers who will allow you to replicate your eggs across geographically diverse, or even continentally divers, data centres. They'll charge you more for a greater degree of resilience but if you need that resilience it still likely works out cheaper than setting up multiple datacentres of your own

      1. TheMeerkat

        Re: welcome to the new world...

        There are cloud providers who will allow you to replicate your eggs across geographically diverse

        And then one day the cloud company will decide that for some reason they dislike whatever service you provide and cancel your contract overnight, like AWS did to Parler. What you are going to do then?

    5. FatalR

      Re: welcome to the new world...

      You *could* have your stuff running in 2 separate companies, which covers billing blackmail and stuff.

      But that starts to not look as cheap as the calculations did when people only look at the most basic...

    6. TimMaher Silver badge
      Joke

      What is the most frightening sound in a server room?

      No sound at all.

      1. Stoneshop
        Flame

        Re: What is the most frightening sound in a server room?

        Or just a little less.

        Mid 1980's, and your data centre was just your computer room, although it would have a fire suppressant system, an UPS and offsite tape storage, but no backup DC elsewhere. So I happened to be inside when I notice the soundscape changing, but it wasn't immediately obvious what the cause was. Took a couple of seconds before it clicked that the low rumble emitted by the aircon was gone, immediately followed by the sense of urgency caused by realizing that the systems were still belching close to 100kW total into the hall, but none of that was taken out. Storming into the sysadmin pen I had no problem at all conveying that urgency to the ones present, half of which were dispatched to round up any air moving device from the office areas, and the others shutting down and switching off any system not utterly extremely necessary; I think they left three area routers and their routers for Northwestern Europe.

        It was the only time I saw a thermograph pen move: Ten deg C in as many minutes. Luckily there was very little loss of hardware; three RA81 HDAs out of over a hundred, and two systems started throwing memory errors a couple of days after.

        1. Norman Nescio Silver badge

          Re: What is the most frightening sound in a server room?

          I had a very similar experience at about the same time. The on-site DEC engineer pointed out the RA81s would be out of warranty if their temperatures exceeded a certain value, so the Ops Manager got the loading bay doors open* and 'a number' of large fans running. No loss of service (it was some VAXclusters and an IBM mainframe clone) and no hardware loss.

          NN

          *I'm pretty certain he had the building main entrance open, and the multiple security doors through to the machine room open as well to get airflow through - in through the entrance, out through the loading bay, with people stationed to prevent unwanted visitors.

      2. stiine Silver badge

        Re: What is the most frightening sound in a server room?

        Actually, recovering from 'no sound at all' is much, much easier that recovering from 'crackling and roaring'. I've had to recover from 'no sound at all' twice. The first was because of an unplanned UPS shutdown in th middle of the day. The second time was because I personally shut down every system in the room at 2am because it was over 85F and rising, in the computer room.

      3. Loyal Commenter Silver badge

        Re: What is the most frightening sound in a server room?

        Nah, halon discharge. Not that you'll be hearing much else for a while after.

    7. Roland6 Silver badge

      Re: welcome to the new world...

      >I have avoided the cloud ...All of your eggs in someone else's basket is risky.

      Actually it depends on who 'all' is.

      From a national viewpoint, there is an issue with having a significant number of businesses using the same cloud provider. For an individual business, there is little difference: either you are responsible or you've outsourced that responsibility - typically to a single provider.

      With only a handful of cloud providers whilst we might be using IT more efficiently, we have also increased our dependency on things working.

  15. Tim Brown 1

    Well I think I backed up everything

    Well I think I backed up everything to AWS but now the login page to AWS has sent me an email to login and my email server is in OVH Strasbourg (though in SBG4 so hopefully safe).

    Disaster recovery never works quite right, sigh.

    1. Lon24

      Re: Well I think I backed up everything

      But where is your DNS? Switching email servers is fundamental to disaster recovery these days. In the old days you could just use the blower to get round any local difficulty, in theory.

      Reminds me of the Rackshack incident when their only connection to the power grid blew up. The UPS kicked in just fine - not a blink on the servers. But then one of the generators started overheating. They had been both tested but not together - and the exhaust systems kinda got compromised. No problem, a quick adhoc redesign, lots of plywood would sort it - but where were the carpenters?

      Oh - the switchboard wasn't connected to the UPS and, being the US, there wasn't any mobile coverage in the area. They had to drive up a hill to get a signal and call in the troops. The users weren't told the amazing story until a couple of weeks later when the DC was finally reconnected to the grid. Amazing damage control by the bottom of the bargain server companies.

      Sadly the architect of the first company to break the $100 server price point is no longer with us:

      https://www.i2coalition.com/saying-goodbye-to-web-hosting-icon-head-surfer-robert-marsh/

      1. katrinab Silver badge
        Unhappy

        Re: Well I think I backed up everything

        My DNS is at OVH.

        I can still get in to change things, but I have to go to the billing section first, find the bill for the last renewal of that particular domain, then go into settings from there.

        Going by the obvious route leads to error pages.

        I tried a DNS query for a subdomain I don't frequently use, so shouldn't be cached anywhere, and it did respond reasonably quickly.

      2. The Oncoming Scorn Silver badge
        Mushroom

        Re: Well I think I backed up everything

        I remember when they decided to test the UPS & generator systems at a certain county council.

        First the generator wouldn't fire up due to water contaminated fuel.

        Once drained & resupplied, on the weekend of the rescheduled test the generator caught fire.

  16. J.G.Harston Silver badge

    I used to do occasional deliveries at SYGDC which for years I though was South Yorkshire Gold Deposit Centre. After all, it looks like it. Oh, Global Data Centre? So that's what's behind all the barbed wire and fortifications.

    1. Wenlocke

      I used to work near there. If you know what to look for, the mahoosive cooling pipes sticking out of the mound are a dead giveaway.

  17. Coastal cutie

    Seriously odd

    This is seriously weird - every DC I've ever been in had fire suppression systems to remove the oxygen element from the fire triangle, whether that was halon or one of the newer compounds - which is why of course humans have to get the hell out with extreme rapidity, as it isn't just fires that need oxygen. Be interesting to see how the building was fire compartmented as well, did someone skip on the proofing, like in the Twin Towers? Or do we have a cladding issue that's fuelled it and assisted the spread to the second DC?

    1. Ken Moorhouse Silver badge

      Re: it isn't just fires that need oxygen

      The warning that I have seen in places I've visited that have something like this is along the lines of:-

      "If you smell lemons then exit immediately."

      Is this similar to the function of mercaptan to alert people of gas leaks, or does halon really smell of lemons?

      1. Eclectic Man Silver badge

        Re: it isn't just fires that need oxygen

        It is a lemon scent added to the fire suppressant so that us humans can tell when there is a leak or activation of the system.

        Of course pressurised systems usually also provide a 'whistle' distribution nozzle, so that there is an ear-splitting scream if the fire-suppressant system is activated, so anyone who cannot smell still has a sporting chance of surviving.

        1. Stoneshop
          Boffin

          Re: it isn't just fires that need oxygen

          so anyone who cannot smell still has a sporting chance of surviving.

          Though they may find their eardrums having collided in the centre of their skull.

    2. Eclectic Man Silver badge

      Re: Seriously odd

      Well, you are assuming a lot, like, competent management, spending money on fire detection and suppressant technologies. Testing of said technologies on a regular basis, and of course maintenance of said technology as per the manufacturers' instructions. All of which costs $$$money$$$, and has to come off someone's budget.

      Just remember that HMS Antelope was hit by an Exocet missile in the Falklands / Malvinas war. The warhead did not explode, but the engine and unspent fuel did cause a major fire in a 'fire-sealed' compartment. Unfortunately the engineers installing equipment on HMS Antelope had breached the sealing walls, so the fire spread and the ship sank.

      And then a story: I once was asked to approve, as an IT Security consultant, a very large data centre for use by a govt Department. They claimed to have lots of other Departments' equipment there so it must be ok. The could not, however, provide any certificate from any HMG Accreditor as to the security status of the site, or the details of any Accreditor or Departmental Security Officer who would independently vouch for the datacentre. After about there weeks of me pestering the supplier, I had to formally advise my boss not to go with them as it was likely they did not have adequate security, despite having lots of HMG contracts and equipment there already.

      1. Ozumo

        Re: Seriously odd

        Antelope was not hit by an Exocet but by aircraft-delivered bombs, and was sunk as a result of an explosion caused during an attempt to defuse one of the unexploded bombs.

        I was a schoolfriend of the steward killed by the initial bomb impact.

        1. Eclectic Man Silver badge

          Re: Seriously odd

          Hi, Ozumo, you are right, the Antelope was hit by bombs. Very sorry to hear about your school-friend being killed.

          The ship I was thinking of was HMS Sheffield:

          https://en.wikipedia.org/wiki/HMS_Sheffield_(D80)

          "She was struck and heavily damaged by an Exocet air-launched anti-ship missile from an Argentinian Super Étendard aircraft on 4 May 1982 and foundered while under tow on 10 May 1982."

          1. Anonymous Coward
            Anonymous Coward

            Re: Seriously odd

            The Sheffield incident also lead to a change in naval uniforms. For cost reasons, uniforms had changed from cotton and wool to include man made fibres. Plastic in other words, so they stuck to the skin when they caught fire.

      2. Coastal cutie

        Re: Seriously odd

        Hence my question marks - I would not be the slightest bit surprised if so many corners were cut they ended up with a circle.

        1. Eclectic Man Silver badge
          Happy

          Re: Seriously odd

          Sorry, missed that. Have an upvote :o)

  18. Unicornpiss
    Flame

    Backups..

    "Yes sir, the fire's finally out. More bad news, our entire backup on celluloid film was utterly destroyed."

  19. trollied

    It's not looking brilliant ... https://twitter.com/xgarreau/status/1369559995491172354?s=20

    1. Brewster's Angle Grinder Silver badge

      According to the tweet, "Mais si vous utilisez le Backup FTP #OVH, vos sauvegardes sont dans un autre datacenter." Which seems relevant to the discussion.

    2. Dave Pickles

      What on earth is burning in there? I wonder if it was a 'sustainable' construction building, with wooden walls and wool insulation. It certainly sustained combustion...

      1. Anonymous Coward
        Anonymous Coward

        Looks like it was a steel framework with wooden floors....

        https://twitter.com/MedecineLibre/status/1369618937063817219

        1. First Light

          Exactement.

          Wooden floors. So says LeMonde.

          1. Anonymous Coward
            Anonymous Coward

            Exactement.

            Wooden floors. So says LeMonde.

            Thick, heavy hardwood floors are among the least likely to combust or weaken in an intense fire. Steel beams, however, deform and sag like warm taffy in a sufficiently hot fire. FPSE (Fire Prevention and Safety Engineering) people are interesting to talk with - you would much rather have oak beams and floors, and wool insulation than glass and steel with fiberglass insulation after a chat with them.

            1. EnviableOne

              I'm building exclusivley in compacted straw nowadays.

              it wont burn, is self insulating, and strong as owt.

            2. Norman Nescio Silver badge

              Hardwood in fires

              Well, yes and no. With sufficient ventilation hardwood burns quite nicely: just ask any wood-burning stove owner. With insufficient ventilation, it takes a while, but will make hardwood charcoal, which is 'rather brittle', so not something I'd like to trust to structurally.

              But the FPSE people do have a point: hardwood chars nicely, and char is rather a good insulator, so a structural element will form a layer of char on the outside while maintaining good strength inside for a long time. Which is why the Chinese used (impregnated) white oak re-entry shields for some of their recoverable surveillance satellites (Fanhui Shi Weixing).

              Astonautix: Fanhui Shei Weixing;Recoverable Test Satellite.

              The capsule for the FSW, like that of the US Discoverer/KH-1 spy satellite, was mounted heat shield-forward on top of the launch vehicle. The ablative impregnated-oak nose cap covered electrical equipment. The spherical aft dome contained the recovery parachute. The film reels for the camera were located in an intermediate compartment.

              StackExchange: Puzzler - which spacecraft(s) incorporated real wood structural elements?

              The FSW did have an oaken heatshield. I saw one on display at the bicentennial airshow (Richmond, Sydney, 1988). This craft was launched on 1987 August 5, and was previously known in the west as China 20, then variously called FSW 0-9 or FSW 0-10, depending on whose chronology you refer to. Cospar 1987 067A, NORAD 18306. The oak was charred, and some broke off when display attendants moved the spacecraft. They simply vacuumed most of it up so that the charcoal didn't mess the carpet! I was able to photograph both the exterior and interior (equipment had been removed).

      2. Claverhouse Silver badge

        I wonder if it was a 'sustainable' construction building, with wooden walls and wool insulation

        Doesn't look like a traditionalist dream to me.

        Rather more like a modernist's delight, with plenty of plastics, cladding, foam ceiling tiles and exciting new materials recommended in Bauhaus Weekly.

    3. Pascal Monett Silver badge

      I love the remark in the tweet that says "if you used OVH backup, you're backups are in another DC".

      SGB3 ?

      1. EnviableOne

        I feel sorry for those whos systems were in SBG2, and backups in the quater of SBG1 that isnt anymore.

    4. Camilla Smythe

      Looks good to me....

      https://twitter.com/xgarreau/status/1369559995491172354/photo/1

      Says OVH Cloud. Looks like a cloud. What's the problem?

  20. DaemonProcess

    all bets off

    When the fire brigade turn up, all your previous bets around proximity to other local DCs are off. I bet some companies decided to back up to another DC on site in order to save on comms costs, so will be completely unavailable until the fire marshalls agree to power up again. Of course if the power supply wasn't sufficiently compartmentalised then it would be OVH's fault.

    Another potential issue is if the any local safe holding backup tapes is operated electronically. Yes some orgs are still glued to tape backups.

    What if they also located the backup catalogs on line in one of the DCs - even cross connected makes no difference if it's all down.

    However, in the defence of colo DCs such as OVH, some of the FUBARs from cloud providers with respect to DNS outages, storage configuration admin errors and so forth have caused entire regions to go down with poor/slow back-out options when it happened - an even bigger blast radius.

    1. TheBorg

      Re: all bets off

      Tape backups are fine if the copies are remote (and not in the building next to the one on fire).

      Reason people still use tape is because its the cheapest medium and can be taken out of that expensive tape library and stored (in the correct environmental conditions) for years if needed.

      1. Anonymous Coward
        Anonymous Coward

        Re: all bets off

        A van full of tapes driving 60 miles down the road has an awful lot of bandwidth as well.

        1. katrinab Silver badge
          Happy

          Re: all bets off

          A van full of 14TB USB hard drives also has pretty good bandwidth. The 5TB 2.5" versions might even have better bandwidth, because I think you can get about 6 of them in the same space a 2 x 3.5" drives

          1. Stoneshop

            Re: all bets off

            I'd rate tape as more robust, especially compared against the data density of those high-capacity disks.

            And for specific applications there are WORM tapes.

          2. EnviableOne

            Re: all bets off

            Amazon have a Data waggon, 40ft artic with a server farm, has multi TB/s connectivity and serious capacity.

            I was considering a service similar with a Backblaze storage Vault in a truck ... 12PB traveling at 60 mph anywhere you need ....

      2. Eclectic Man Silver badge

        Re: all bets off

        In the St Marys Axe devastation, one of the problems was that tape backups were in a fire safe in the building. Although perfectly ok, the Fire Brigade refused entry until they were sure the building was structurally sound, which took several days, during which time, of course service could not be resumed.

        1. Anonymous Coward
          Anonymous Coward

          Re: all bets off

          The same thing happened to ICL after the Manchester bombing. Backups in the Firesafe turned out to be fine, but they weren't allowed to retrieve them for some time!

  21. sitta_europea Silver badge

    If all the other OVH facilities burn down, I'll be very glad.

  22. JacobZ
    Joke

    Turns out there was a hole in their firewall

    Also, many geeks are just now learning the physical origin of the term "firewall"

    1. Potemkine! Silver badge
      Coffee/keyboard

      Re: Turns out there was a hole in their firewall

      ROTFLMAO!

      1. Eclectic Man Silver badge
        Alert

        Re: Turns out there was a hole in their firewall

        Along with the warning icons available to warn for Trolls, pedants and jokes, maybe there should be one for total lunacy:

        There was a military site, when the Internet had just been invented, that had an internal network connected to the 'outside' via a firewall (just the one, it was before dual homing and DMZs). There was also a connection to the outside world bypassing the firewall "in case the firewall breaks".

    2. Jamie Jones Silver badge
      Facepalm

      Re: Turns out there was a hole in their firewall

      I always thought it was a reference to a wall of fire that is impenetrable by mere mortals...

      The real meaning is far too boring!

      1. Anonymous Coward
        Anonymous Coward

        Re: Turns out there was a hole in their firewall

        The level 5 spell?

        1. Jamie Jones Silver badge
          Thumb Up

          Re: Turns out there was a hole in their firewall

          I was thinking more of a later episode of stargate-sg1, but yeah, that sort of thing!

  23. Anonymous South African Coward Bronze badge

    This begs the question - was there any space between the buildings to prevent the spread of a fire?

    Surely if the buildings was designed, planned and built far apart enough, the fire would have been confined to one building only.

    And why did they put so many eggs in one basket?

    1. Anonymous Coward
      Anonymous Coward

      That only works if they don't share any services or single points of failure.

    2. Anonymous Coward
      Anonymous Coward

      "This begs the question - was there any space between the buildings to prevent the spread of a fire?"

      My bet is they screwed up this part. Anti-fire compartimentation is very costly, particularly in the cabling columns of a building. And of course, you need to redo them every time you add cabling.

      1. The First Dave

        The DC provider that I am most familiar with, has about 50m of concrete / barely used car park between each DC building, and any one else's building.

    3. Anonymous Coward
      Anonymous Coward

      > Surely if the buildings was designed, planned and built far apart enough, the fire would have been confined to one building only.

      They're data centre operators, not property developers.

    4. Anonymous Coward
      Anonymous Coward

      Looking on Google Maps it's in a very odd location

      1. Victor Ludorum

        Cheap land

        I expect Octave got a good deal on the land as it looks like it's in the old docks area. Also really close to the German border for a resilient fibre connection or two. At least there was a plentiful supply of water to suppress the fire...

        I've got a cheap VPS with them, fortunately at Erith in Kent. It gets more activity from Digitalocean than other OVH IP addresses, although the subnet it's on occasionally gets blocklisted...

  24. yas1

    Bad Timing

    "The fire came just two days after OVHcloud announced it is taking the first steps toward an IPO, floating on the Paris stock market according to Reuters. "

    Talk about bad timing!

    1. Tim Brown 1

      Re: Bad Timing

      The eventual cause and response to the fire will be interesting but I expect it will be several months down the line.

      1. First Light

        Re: Bad Timing

        Yeah, I "smell" something fishy here also.

        The fire started at 1am.

        1. Eclectic Man Silver badge
          Coat

          Re: Bad Timing

          OK, I give in. "Bad Timing", Nick Roeg film starring Art Garfunkel, excellent, but a bit noir.

  25. Sceptic Tank Silver badge
    Flame

    IEEE 1394

    All the servers and DC's are connected via FireWire.

  26. Anonymous Coward
    Anonymous Coward

    It has to be said.

    Your data has gone up into the cloud, unfortunately it's a cloud of smoke.

  27. Ravencroft
    Trollface

    Burger King Orders

    Well it looks like Burger King used OVH for delivery order system. I cannot order in BK at all!

    1. Anonymous Coward
      Anonymous Coward

      Re: Burger King Orders

      How is that a problem?

  28. wyatt

    One of the main Birmingham Universities found that one of it's Computer Rooms fire suppression cylinders were empty recently, there wasn't anyone maintaining the system and topping them up as required. It took quite a period to source the required gas in the quantity required and get them charged!

  29. Anonymous Coward
    Anonymous Coward

    Mandatory XKCD reference

    https://xkcd.com/1737/

    "We don't have sprinklers or inert gas systems. When a datacenter catches fire, we just rope it off and rebuild one town over."

  30. Ken Hagan Gold badge

    It's not too late, yet!

    Good news about everybody being safe, but...

    Update 5:20pm. Everybody is safe.

    Fire destroyed SBG2. A part of SBG1 is destroyed. Firefighters are protecting SBG3. no impact SBG4.

    — Octave Klaba (@olesovhcom) March 10, 2021

    We appear to be reading this tweet before it has been posted. If people move quickly enough, they may be able to migrate away from SBG2 before the fire starts.

  31. captain veg Silver badge

    shame the servers weren't running BeOS

    https://www.everything2.com/title/is_computer_on_fire%2528%2529

    -A.

  32. Anonymous Coward
    Anonymous Coward

    bloody impressive ...

    pictures in the press !

    The thing burnt to the ground !

  33. KittenHuffer Silver badge

    I'm surprised that nobody ....

    .... has made an HCF comment yet!

  34. Eclectic Man Silver badge
    Joke

    Construction?

    I'm not an architect, or construction engineer for that matter, but the video and pictures of the DC make it look like it was built out of shipping containers. Shirly that cannot be correct? Does anyone know how the building was constructed?

    On a personal level I have received literally 0's of messages of concern and I can confirm that my local backup disk, sat on my desk just under my computer is perfectly safe and well, and my collected digital legacy is perfectly safe (although I should probably get around to checking that it is actually storing backup files some time, but hey, it's only been 5 years since I last checked, so it must be ok).

    1. Victor Ludorum

      Re: Construction?

      They had a power outage at that DC back in 2017. SBG4 was built from containers.

      See http://travaux.ovh.net/?do=details&id=28247 for details

      1. Roland6 Silver badge

        Re: Construction?

        I wonder if they upgraded their power supply after that incident:

        "The SBG site is powered by a 20kV power line consisting of 2 cables each delivering 10MVA. The 2 cables work together, and are connected to the same source and on the same circuit breaker at ELD (Strasbourg Electricity Networks)."

  35. Anonymous Coward
    Anonymous Coward

    Peculiar shape of building SBG2

    Looking from above at SBG2 the building looks like number 8 as displayed with a classic 7 segment display. Only these segments are the building.

    There are two voids in this building as in the number 8. These voids go down from top floor to almost ground floor. The height of the floors are lower than expected.

    Looking at pictures of the fire these inner voids acted as chimneys.

    The building seems to be a metal structure mostly. Were the data halls only on ground floor level? Why this kind of design? Was it done this way to be energy efficient/ green?

    The adjacent massive building looks like a typical data center building.

    1. Eclectic Man Silver badge

      Re: Peculiar shape of building SBG2

      The building that burned down definitely appears to be of a modular metal construction:

      https://www.google.com/search?rls=en&source=univ&tbm=isch&q=News+Fire+at+SBG2+picture&safe=strict&client=safari&sa=X&ved=2ahUKEwiDjvCvjqbvAhUNQxUIHSZYCyYQjJkEegQIFBAB&biw=1807&bih=987#imgrc=FW38xe3tWWBcJM

      Not your usual concrete floored 'old telephone exchange' concrete and brick cladding type building at all.

      1. Anonymous Coward
        Anonymous Coward

        Re: Peculiar shape of building SBG2

        It seems to be a metal framework, with wooden floors, and then clad on the outside.

        I imagine that the wooden floors provided very little fire containment once it was established - and looking at pictures, at least each side of that 8 was one long corridor.

        In short... I don't think they had a chance once this fire got established.

        https://twitter.com/MedecineLibre/status/1369618937063817219

        1. Eclectic Man Silver badge
          Facepalm

          Re: Peculiar shape of building SBG2 - possibly fire risk

          I asked a friend of mine who worked in supporting physical infrastructure for a tech company. His assessment is interesting:

          "...they seem to have stacked a number of UPS / generator containers right up tight to the building (give away is the chiller radiator on the outside). You would normally place these some distance from the build.......as they are a serious fire risk!!!!!! Either diesel or chance of combustable fumes from the batteries"

  36. Chris Collins 1

    I imagine even those with backups most of them are either on same site (or even same server) or using OVH backup addon service (which may well be hosted in same DC).

    Personally I implement a local backup which is also copied to a remote location, the remote location been an entire different company, so I can be sure its a different physical location.

    1. Anonymous Coward
      Anonymous Coward

      Just make sure that the connection between them is via diverse paths. A long time ago, in a gaxaxy far, far away (yes, that company), we had to get representatives of both ISPs on a call so they could confirm that their fiber paths didn't share the same interduct or conduit or trenches between our facility and each of their COs.

      1. jtaylor

        confirm that their fiber paths didn't share the same interduct or conduit or trenches

        That reminds me of some conversations with $Major Investments when I worked for a vendor. They asked questions at a level of detail that my company hadn't considered. We delivered our service over leased lines with 2 separate carriers who used separate trenches, but we didn't know whether the lines might share a trench somewhere in the middle. And when other clients specified "redundant feed" they hadn't expected that the second feed came from a data center in a different geographic region.

        To be fair, we all worked to a high standard. It's just that $Major's standards were exceptional. I learned a lot about operational rigor. Also about conflict management (PFY style).

  37. Eclectic Man Silver badge

    Top Priority

    How long will it be before they state that their 'Top Priority' is the resumption of services, or the safety of their staff, or their customers' data, or smoother such statement?

  38. David Cotton

    We sure this was an accident?

    Best to wait for the investigation but this shouldn't be able to happen in any modern data centre.

    1. Eclectic Man Silver badge
      Mushroom

      Accidents

      The largest explosion in the UK since the second world war was at Buncefield, and that was an accident.

      https://www.bbc.co.uk/news/uk-england-beds-bucks-herts-34919922

      1. Ken Moorhouse Silver badge

        Re: Buncefield

        I was woken up by that explosion... as far away as West London!

  39. Anonymous Coward
    Anonymous Coward

    It's quite a small DC.

    Correction, it was a small DC.

  40. Ken Moorhouse Silver badge

    OVH

    Now more frequently known by the tla OFH.

  41. Anonymous Coward
    Anonymous Coward

    simpsons reference here

    I worked somewhere and the fire exit door was painted on.

    Actually I dont recall being shown any escape procedure anywhere I worked. And that was in they days they actually cared about you.

  42. First Light

    From France

    LeMonde says the fire started at 1am.

    "The fire quickly spread through the building. 'We put in place a large hydraulic device, using a high-powered pump-boat [which drew water from the Rhine], to prevent propagation to adjoining buildings,' Damien Harroué, commander of relief operations, told AFP. 'The floors are wood, and well-heated computer equipment will burn. These are plastics, it generates significant smoke and flames,' he added, to explain the large amount of smoke and the rapid spread of the fire." LeMonde also said it was not a Seveso-classified site meaning no dangerous chemicals were present and air-quality testing showed no pollutants had been released.

    https://www.lemonde.fr/societe/article/2021/03/10/a-strasbourg-un-important-incendie-sur-le-site-de-l-entreprise-ovh-classe-seveso_6072548_3224.html

  43. Xalran

    Pictures

    Pictures of the fire/damages can be found here :

    https://www.dna.fr/faits-divers-justice/2021/03/10/strasbourg-important-incendie-dans-une-entreprise-situee-sur-un-site-seveso-au-port-du-rhin

  44. Jim 15

    No geographical separation of 1, 2, 3, 4?

    How come they have 4 centres on the same estate? Shouldn't they be in different towns on different grids?

    1. Roland6 Silver badge

      Re: No geographical separation of 1, 2, 3, 4?

      Depends on the rationale. I've had clients who have a number of "datacentres" adjacent to each other on the same 'production' site. In this instance they were referred to as "halls" and each had a different security designation and purpose. The DR versions of these halls being located across various locations around the country.

      So I expect each OVHCloud DC is actually a specific size to allow for modular manufacture and easy kit out etc. The mistake (other than whatever caused the fire) seems to have been insufficient separation (in all its forms) between DCs/halls.

  45. Anonymous Coward
    Anonymous Coward

    That doesn't always help.

    google: fire extinguisher causes disk failure

    You may save the site, but it can still be a long time before all the services are up, especially if all the roof tiles fell off.

    Don't try this at home: https://www.youtube.com/watch?v=tDacjrSCeq4

    1. EnviableOne

      power failure tens to send Hypervisors into panic, we lost power to the DC for 20 mins, the networking kit came up fine, but it took them days to get the VM hosts to decide which servers where where and which copy they were going to keep....

  46. streaky

    BRB..

    Making backups..

  47. Richard Pennington 1
    Flame

    Quis obscuraret ipsas res obscuras

    (Latin:) Who is to cloud the cloudy things?

    Off-site backup is good, even for off-site backup sites.

  48. Richard Pennington 1
    Flame

    What was the building construction?

    At the risk of an inflammatory comment ...

    Were they using French Grenfell-spec cladding?

  49. Arthur Daily

    CLOUDTASTROPHIE

    Are there other language translations of cloudtrastrophie?

    My bet is 50% of big customers will have no backup, or only backups and DR that is at least one year out of date. Every DR exercise I saw, failed, but they told management all was good anyway. Meh, with 'agile' configuration done by waves of contractors and no documentation, there will be gnashing of teeth.

    Your data will NOT get any 'priority'. The customers who will, will be the big ones - carsales, realestatesales will be at the front.

    In addition INSURANCE will not pay out either. In a legal spat they will subpoena the company documents and minutes that had signoff that backups and DR were in place.

  50. Anonymous Coward
    Anonymous Coward

    Once upon a time......

    ....in a universe far away, large companies with mission critical systems and data used to build their own data centres (plural):

    - DC1 - A main data centre somewhere near the head office

    - DC2 - A secondary data centre at least a time zone distant

    Network connections from both DC locations had connections to a local POP and a backhauled connection to a remote POP.

    In these archaic times there would be an annual disaster recovery trial every year, likely taking at least a week to complete and document.

    Ah....but all this engineering on two sites COST FAR TOO MUCH MONEY!

    So........we move everything to a CLOUD PROVIDER where the staff knew nothing about the business requirements, and had only a tenuous connection (if any) with staff within the business. BUT IT WAS CHEAP..........UNTIL.....................................

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like