back to article Electrical explosion at Google datacenter injures three

Google's consumer-facing and advertising services have faltered after an incident at one of its major datacenters. KETV Omaha reports that Google's datacenter in nearby Council Bluffs, Iowa experienced an electrical incident just before noon Monday local time and that three electricians were critically injured. A Twitter …

  1. Alan J. Wylie

    Arc flash?

    Perhaps an Arc flash ?

    The potash mine in Cleveland was fined £3.6million recently after such an accident

    Update: arc flash confirmed

    1. Cederic Silver badge

      Re: Arc flash?

      That link also tells us it's at the electrical substation, rather than in the data halls, which I find useful context.

      Also: Google have substations attached to their data centres!?

      1. Anonymous Coward Silver badge
        Big Brother

        Re: Arc flash?

        The datacentre looks to have been built next to a fairly major electrical plant... https://goo.gl/maps/TwbzespEtDxRfodj8

      2. daflibble

        Re: Arc flash?

        I'd be amazed if they didn't have their own sub-stations.

        1. Claptrap314 Silver badge

          Re: Arc flash?

          They do. Source: I worked for G as an SRE.

      3. Anonymous Coward
        Anonymous Coward

        Re: Arc flash?

        It's common for large companies to have their own substations. At least two of my workplaces have had them.

        (No downvotes for the question, it's valid and something a lot of folks wouldn't know.)

        1. Martin-73 Silver badge

          Re: Arc flash?

          Indeed, electrical infrastructure is a 'hidden service' by and large

      4. Ommerson

        Re: Arc flash?

        In the UK any larger commercial premises will have an 11kV supply and a small substation on the premises managed and owned by the distribution network operator. There are some in central London with dual-fed 33kV supplies - typically larger, newer developments.

        Will be similar state-side, although the supply potential may be different.

        1. Martin-73 Silver badge

          Re: Arc flash?

          where i trained, (an academic institution) we had our own 11kv ring main, which was usually part of the DNO's ring main, (by judicious use of 11kv breakers and labels, and ...communication). Metering was quite clever, there were CT's on both intake breakers, so connected that current could flow entirely round the institution's own ring, without registering on the meter. All done in 70s ct/actual trivector meter tech. 33 yrs ago now, but i miss that place

    2. chivo243 Silver badge

      Re: Arc flash?

      That's some nasty stuff, hope all the victims recover...

    3. Martin-73 Silver badge

      Re: Arc flash?

      The Potash mine was because they'd had a similar event earlier, been told to smarten their act up, and utterly failed to do so. But I do hope the sparkies recover soon, and the injuries aren't too bad :(

  2. Evil Auditor Silver badge

    Well, years ago, in a different life, we had some Google appliances in our data centre. The Google draft contract had a disclaimer that they are not liable for any deaths resulting from the use of their appliances. We read, sniggered, and had it removed from the agreement. But now I start to understand why they wanted it in the first place...

  3. stevebp

    I'm amazed at the descriptions of the injuries sustained - a serious procedural error must have taken place for such critical injuries to have been sustained. Anyone operating or maintaining LV equipment, where there is the remote possibility of an arc flash should be kitted out in full PPE - although the blast impact would still have been considerable, the PPE would prevent major burns

    1. Martin-73 Silver badge

      Yes, pure speculation here but familiarity breeds contempt, as the old saw goes. LV doesn't tend to hurt much if you incidentally touch it, even 277 if stateside, wouldn't want to mess with 347/600 in the great white north tho..... if you're wearing rubber soled shoes etc, so despite KNOWING the amount of energy involved, it's way easy to get complacent.

      I know this, i've done it myself. Fortunately due to fast acting HRC fuses on the intakes in the UK, i've only ever had a ringing in the ears and an inability to see for a few seconds. Plus that panicked thought 'feck, now i have to call the DNO'

  4. Claptrap314 Silver badge

    Seems unlikely

    I worked for Google as an SRE when it became Alphabet.

    Our systems were designed so that something like this would not cause disruption to the customer. One data center unexpectedly going offline was a core scenario, and one that we regularly exercised. (And by "regularly", I mean "more than once a month".)

    Three options come to mind. The first is that this outage was deemed to be non-transient, which means that additional capacity for the affected services needs to be brought online. If someone fat-fingered the change, you might see an outage.

    The next option is that the bean counters might have pushed to reduce the resilience of the systems. Of course, they don't admit that's what happens, but it does. Our PM's response to one such initiative was brilliant, "It is in Google's best interest that you pretend to believe these calculations."

    Third is, just a coincidental fat-fingering of an unrelated change that happened to be on that day.

    1. Martin-73 Silver badge

      Re: Seems unlikely

      Given the way fate works, I'd go for the latter option, plus locusts

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like