back to article Google's newest cloud region taken out by 'transient voltage' that rebooted network kit

On July 25th, Google cloud launched a new region with all sorts of fanfare about how the new facility – australia-southeast2 in Melbourne – would accelerate the nation's digital transformation and make the world a better place in myriad ways. And on August 24th, the region went down quite hard. Late in the afternoon, local …

  1. JamesTGrant

    transient voltage at the feeder to the network equipment

    RCA1: ‘problem with electric supply, voltage spikes made our routers unhappy.’

    Or RCA2:… there’s a zoo keeper who has to keep the network beasties fed and watered and they got tazed/electrocuted so they couldn’t tend to said beasts which then got grumpy and wouldn’t perform at the datacentre zoo. Hopefully they’ll be ok with some burns treatment and some rest and the zoo will arrange cover for them to prevent the network animals getting too hungry again in future.

  2. Kevin McMurtrie Silver badge

    Harden the F up

    Being in a big city doesn't seem to be much of a defense against everything trying to kill you in Australia.

    1. Anonymous Coward
      Anonymous Coward

      Re: Harden the F up

      I agree there are more fuckwit low IQ Aussies in Aussie cities than rural towns.

      1. RegGuy1 Silver badge
        Joke

        Re: Harden the F up

        What do they say about New Zealand? If someone emigrates from there to Australia they increase the average IQ of both countries.

    2. cookieMonster Silver badge
      Joke

      Re: Harden the F up

      I read on this forum not too long ago that the only things in Australia that won’t try to kill you “are some of the sheep”

      1. W.S.Gosset

        Re: Harden the F up

        Depends how you cook them.

      2. hittitezombie

        Re: Harden the F up

        New Zealand, on the other hand...

        Just watch Black Sheep.

        1. Kane
          Joke

          Re: Harden the F up

          "Just watch Black Sheep."

          "What about the sheep?"

          "Fuck the sheep!"

          "No time for that bro. Go go go!"

  3. Rob F

    Australia does seem to have some brown out issues

    Which is why the Tesla battery farm in South Australia.

    I wonder if there was a surge due to non-consistent supply?

    1. Chris G

      Re: Australia does seem to have some brown out issues

      The surge was due to a server being bitten by one of Lu Tze's experimental electric spiders.

    2. TRT Silver badge

      Re: Australia does seem to have some brown out issues

      The problem is the kangaroo gin gang. Individually they might have more power than horses but in terms of smoothness of output...

      Now if they could work out a way of hooking up those fuck off humongous spiders to the generator instead ...

  4. Anonymous IV
    FAIL

    Both Sides Now

    Joni Mitchell's song seems oh so prophetic:

    So many things I would have done

    But clouds got in my way

    I've looked at clouds from both sides now

    From up and down, and still somehow

    It's cloud illusions I recall

    I really don't know clouds at all

  5. John 104

    Sutpid

    Are we to believe that Google, one of the most wealthy companies in the world, failed to implement proper generator, ATS, and UPS infrastructure? I mean, come on! That is basic data center 101 stuff. The only scenario that would explain otherwise would be if the, no, forget it, there is no logical scenario. There should have been redundant switches with redundant power supplies, plugged into redundant two different circuits, supplied by a rack sized PDU tied directly into the UPS, which in turn, should have been tied into an ATS and generator. What a fuck up.

    1. Anonymous Coward
      Anonymous Coward

      Re: Sutpid

      I bet they have a spelling checker though.

    2. Lars Silver badge
      Coat

      Re: Sutpid

      The article says this "Google hasn't said if the networking equipment that rebooted belonged to it, or a supplier.".

      1. DS999 Silver badge

        Re: Sutpid

        Same thing. If they are relying on someone else's infrastructure they need to perform due diligence to insure it is up to the standard required for an always on service.

        Blocking voltage spikes is one of the easier tasks in a datacenter. If you can't do that right, what else can't you do right?

    3. Paul Hovnanian Silver badge

      Re: Sutpid

      They lease server capacity from the cloud.

    4. jockmcthingiemibobb

      Re: Sutpid

      She'll be right!

    5. Man inna barrel

      Re: Sutpid

      You can't eliminate failures like this, only reduce the probability to an acceptable level. I remember a vendor of surge suppression devices demonstrating how much energy could be delivered by a direct lightning strike to comms or power cables. A large ocean liner, three feet out of the water. I never bought the biggest suppressors, that could handle that much energy, but none of my kit suffered a direct lightning strike, so I guess I my statistics were about right.

      I did some work on a power supply, which had to work off what might be called "dodgy" mains, from a diesel generator on site. So I added something to voltage ratings, to allow for surges. But how much is enough safety margin? You have to draw the line somewhere. Put it this way, the old power supplies popped often enough to cost money on site callouts, and the new ones generally keep going, so I guess that is a job well done.

  6. Throatwarbler Mangrove Silver badge
    Devil

    obBOFH

    *flips pages*

    ”It looks like your issue is ... transient voltage, you know, like when a homeless man steals the copper out of your electrical wires?"

    *dummy mode on*

    "So what do I do to fix it?"

    "Just run down to the main power transformer, and you see that big red toggle switch?"

    "Sure."

    "Just flip that thing up and down a good twenty times."

    "Won't that take out all the power to the site?"

    "No, no, it will normalize the power by forcing the volts to flow over a non-transient circuit."

    From the other end, I hear *chunk* *chunk* *chunk* *blam* as a million volts explosively surge through the transformer and, incidentally, the poor luser. I swear, you can't pay for job satisfaction like this.

    With apologies to Simon Travaglia.

  7. Anonymous Coward
    WTF?

    My personal computer could survive this

    I would never think of running feral electrons directly into my kit. Everything goes through a UPS which blocks transients and handles low and high voltage situations. All my computer ever sees is a sine wave of proper voltage. It's held up through nearby lightning strikes that took down the local power grid.

    I have to think that Google is covering up something more serious.

    Either that or some Aussie bigwig spilled his Fosters

    1. elip

      Re: My personal computer could survive this

      Nah...I was involved in helping google figure out basic data center power, DC cooling, Linux, UNIX, storage issues after they acquired a company I worked at, and proceeded to screw the pooch on a thoughtlessly executed data center migration (that was wholly based on marketing of the new location [in their words - "to enable us to hire younger engineers"], *NOT* on any technical requirements post-acquisition). They truly don't know what they're doing when it comes to data center ops compared with companies who had been doing it 6+ decades. Their engineers I worked with believed all workloads were easy to understand and troubleshoot. It was beautiful watching them scramble as they failed to understand one protocol after another. :-P

    2. rcxb1

      Re: My personal computer could survive this

      <blockquote>Everything goes through a UPS which blocks transients and handles low and high voltage situations.</blockquote>

      And when your battery is getting weak, your UPS informs you by rebooting everything connected to it, even while grid power is working fine...

      After managing a few hundred of them, you'll be less enamoured of UPSes.

      1. jockmcthingiemibobb

        Re: My personal computer could survive this

        Which is why we're all DC these days.

    3. Kevin McMurtrie Silver badge

      Re: My personal computer could survive this

      You've never had the high voltage wire fall onto the low voltage wire outside your house. I was a service tech sent out to a small Mom & Pop store that had it happen. The system I was repairing had fuses, RF suppressors, MOV suppressors, and ground paths to catch arcing. Had. All of that and another centimetre of PCB was gone. Anything sensitive to being licked by high current plasma needed replacement. The store was having a really dark, smokey, bad day but at least nobody was hurt.

      It wasn't in Australia so it probably wasn't as bad as it could be.

      1. John H Woods Silver badge

        Re: My personal computer could survive this

        "You've never had the high voltage wire fall onto the low voltage wire outside your house"

        USA, I'm guessing? Opportunities for high tension wires to touch domestic voltage in the UK are extremely specific and accidents are rare. Even in rural areas I have never seen the final stepdown on a pole anywhere a road-going vehicle could hit it. Infrastructure in the USA looks almost Third World by comparison.

  8. Stumpy

    Obligatory XKCD

    https://xkcd.com/908

  9. W.S.Gosset

    Victoria...

    They didn't say spike, they just said voltage. Given Victoria routinely has to shut down industry *cough* "Demand Management is the bold new power industry tool for a renewable future!" as a result of leaning hard into renewable energy, I personally suspect a voltage DROP rather than a spike. Sustained for longer than Google's setup anticipated in a First-World country, or just not caught/detected for the same reason.

    Pity they didn't run South Australia's tactic. After the last stuffup there, SA loudly announced a Tesla battery and decent backup despatchable power. The battery had IIRC 2.5mins power supply (deliverable in a dribble over 75mins). The despatchable power was a trick taken from EUK windfarms* : they built a dirty great diesel park. A$415m to burn 80,000 litres of diesel an hour and sustain the Green Revolution.

    .

    * (The EUK grant-farms tricked the bureaucrats by saying they wanted their despatchable power reqts to be technology agnostic. The govt assumed they meant not cutting off future innovation from the flow of subsidy and cheered+implemented the changed wording ; "imagine the surprise" when they just turned round and whacked in diesel.)

  10. Kane
    Trollface

    clouds are far from infallible

    A cloud struck by lightning?

  11. JonGodfrey

    Clouds and lightning don't mix... oh! no, wait

  12. Aussie Doc
    Megaphone

    Optional sensible title here

    Meanwhile the Aussie googletwat said "We're not here to fuck spiders - let's just get on with it" and all was forgiven.

  13. Man inna barrel

    Bloody rabbits ate the RF cables

    My father worked as a radio astronomer, and his regular place of work was an observatory built on an old airfield in rural Worcestershire. It had a couple of steerable dishes, on railway tracks. You have to get the RF signals back to the lab, so the data can be analysed, and this meant long cables. In the early days, cables used to insulated with rubber, and this is apparently quite attractive to rabbits. I can't think why, but then I am not a rabbit, even though I admit to being vegetarian.

    Dad's radio observations often had to take place at unsocial hours. One of the things I learned about astronomy is that it does not work according to office hours, synchronised to GMT. Anyway, the data would come streaming in, duly recorded, and then the signal would falter, and drop altogether, so ruining hours of work. Bloody rabbits.

    I suppose it would be possible to coat the cable in poison, to deter the rabbits. From my knowledge of lethal chemistry, I would recommend cyanide, because it is very quick acting. You have kill the nibblers before they cut through the insulation. Slow acting poisons like arsenic or warfarin are no good, because what is the point of killing the rabbits days or weeks after the damage is done? I think chilli powder might have worked.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like