back to article World+dog share in collective panic attack as Google slides off the face of the internet

Google services such as YouTube and Gmail started the week with an almighty bang as the Chocolate Factory's cloud came crashing to the ground. Despite an insistence from the company's various health dashboards that all was fine and dandy, it most definitely was not. Those seeking distraction in video form were treated to …

  1. Winkypop Silver badge

    “So there was no need to resort to something like Bing”

    Steady on now Ted!

    1. Lomax
      Thumb Up

      Re: “So there was no need to resort to something like Bing”

      I've been using Qwant as my default search engine for months now, and have found it to be really quite good. Previously tried to use DuckDuckGo as the default but found myself constantly repeating the search in Google - not so with Qwant!

    2. MatthewF

      Re: “So there was no need to resort to something like Bing”

      Careful Now

      1. Anonymous Coward
        Anonymous Coward

        Re: “So there was no need to resort to something like Bing”

        Down with this sort of thing

        1. David 132 Silver badge

          Re: “So there was no need to resort to something like Bing”

          That would be an ecumenical matter.

          1. Korev Silver badge
            Windows

            Re: “So there was no need to resort to something like Bing”

            Yes!

    3. Imhotep

      Re: “So there was no need to resort to something like Bing”

      Does anyone have a suggestion for a good alternative to Google search? None that I've used measure up.

      1. Lomax

        Re: “So there was no need to resort to something like Bing”

        > a good alternative to Google search

        Did you try Qwant?

        1. Anonymous Coward
          Anonymous Coward

          Re: “So there was no need to resort to something like Bing”

          What's wrong with "[web] search" then?

        2. low_resolution_foxxes

          Re: “So there was no need to resort to something like Bing”

          Ecosia is surprisingly good fir search, EU too!

  2. Victor Ludorum

    That explains it.

    Thunderbird asked me to re-authorise access to GMail - I thought it had just expired or something, but I was then getting a 500 error. All seems normal now (well, as normal as 2020 can be).

    1. Anonymous South African Coward Bronze badge

      Re: That explains it.

      Same here - but I needed to re-authenticate myself after the charlie foxtrot event have passed.

      Everything's fine and dandy now.

  3. sorry, what?
    Unhappy

    The entire g string fell off...

    Exposing friends and foe alike to naked robots with their innards on show. Not just gmail and YouTube etc. (my son found he could still access YouTube through a desktop browser, just not on his device, so could continue to chortle at those kittens and their antics); gsuite was borked too. Looked like a serious authentication infrastructure boo-boo. I was told that my gsuite email address wasn't recognized at all when attempting to login. Ooof.

  4. Brewster's Angle Grinder Silver badge
    Coat

    THIS WAS SO SERIOUS I HAD TO GO ROUND TO MY EIGHT-FOUR YEAR OLD NEIGHBOUR AND SEE IF SHE COULD COPE WITHOUT GMAIL.

    YES, I DO NEED A COAT - IT'S RAINING AGAIN.

    1. TRT Silver badge

      Raining again?

      That’ll be the one cloud service that you can guarantee will be working.

      1. richardcox13

        Re: Raining again?

        Except when it comes to summer... and all the gardeners start moaning after 30min of hot weather.

      2. Ken Moorhouse Silver badge

        Re: the one cloud service that you can guarantee will be working.

        Comment of the week and it is only Monday.

    2. David 132 Silver badge
      Happy

      THIS WAS SO SERIOUS I HAD TO GO ROUND TO MY EIGHT-FOUR YEAR OLD NEIGHBOUR

      It's alright, you're not talking to your neighbor any more, there's no need to speak so loudly while enunciating carefully.

      Unfortunately, we don't have Rich Tea biscuits or Werther's Originals, sorry.

  5. JonW
    FAIL

    Impressive

    Last Month

    AWS: Here's a major outage for you

    This Month:

    Google: Hold my beer...

    1. Anonymous Coward
      Anonymous Coward

      "Google: Hold my beer..."

      "Oooops, I poured it on the Master Console and Main Power Controller"....

    2. Captain Scarlet
      Trollface

      Re: Impressive

      Office 362: Been there done that, oh no not again

  6. chivo243 Silver badge
    Facepalm

    Enterprise stuff

    G-Suite\ GAffE went offline... my personal gmail was up the whole time. I received an error message trying to authenticate into the google chat app stating my domain didn't have access to the service???

    Things are back to normal for the time being?

  7. Fábio Rabelo de Deus

    You can add Brazil in the list of affected Countries .

    1. TRT Silver badge

      I hear things were left alone there, but only in a narrow strip down the middle of the country.

      1. Anonymous Coward
        Anonymous Coward

        It's been too humid here, that's probably related.

        1. Sgt_Oddball
          Trollface

          I think you'll find...

          The word is 'moist'...

  8. Warm Braw

    The Register contacted Google

    Given we're not talking about Apple, you might be able to infer from their silence that they eat their own dog, er, food.

    1. EnviableOne

      Re: The Register contacted Google

      its not Samsung or LG

    2. DavCrav

      Re: The Register contacted Google

      Or that they werw a bit busy?

  9. andy 103
    FAIL

    Try again *later*

    It always makes me chuckle when you get error messages that say

    "Try again in five minutes"

    Why five? Where has that figure come from? They don't even know what the error is! So how can you reliably determine when someone can "try again"?

    But it's somehow better than "try again later". When is later? 10 mins, tomorrow, next year?

    Either way we'll all keep madly hitting Ctrl+F5 until we get the response we want.

    1. TRT Silver badge

      Re: Try again *later*

      How long does it take YOU to make a cup of tea?

      1. andy 103

        Re: Try again *later*

        How long does it take YOU to make a cup of tea?

        Well I made at least 2 in the time it took Google to fix this.

        So, considerably longer than 5 mins.

        1. TRT Silver badge

          Re: Try again *later*

          I’ve always been fond of HandBrake’s policy on timing user activity.

          1. Anonymous Custard
            Boffin

            Re: Try again *later*

            Or the Windows file transfer dialogue box, where the last 10s often seem to take at least 10 minutes...

            Never a finer example of quantum time dilation!

            1. bigphil9009

              Re: Try again *later*

              Obligatory XKCD:

              https://xkcd.com/612/

        2. EnviableOne

          Re: Try again *later*

          too long untill i installed the HTCPCP -TEA extension

          1. Strahd Ivarius Silver badge
            Joke

            Re: Try again *later*

            did you get a response code 418 ?

    2. Pierre 1970

      Re: Try again *later*

      I'm pretty sure this could apply as a new official Vulture unit....the Cloudwatch* is the time between retries in order to see if the cloud troubles were fixed...

      (*) Ok, ok, I already know that this name is already taken by AWS :-(

      1. Antonius_Prime

        Re: Try again *later*

        Could call it the "Shower". I mean, that's what happens when clouds fall over in meatspace, right?

        MTBS - Mean times between showers...

        Could catch on...

        1. zuckzuckgo Silver badge

          Re: Try again *later*

          Cloudtwatch?

    3. Doctor Syntax Silver badge

      Re: Try again *later*

      "Either way we'll all keep madly hitting Ctrl+F5 until we get the response we want."

      I think you answered your own question. The worst possible alternative would be to say try again at $TIME" where the saem time is given to everybody.

    4. Charlie Clark Silver badge

      Re: Try again *later*

      Depending on the system, it's a reasonable approach because it acknowledges there is a problem and sets an expectation that the problem can be fixed soon — this is a much worse word to use — and that you should do something different for a couple of minutes rather than pressing refresh all the time.

      A major outage like this can never be ruled out but should have some form of recovery plan. Individual plans for each system should also exist and are easier to test.

    5. Claptrap314 Silver badge

      Re: Try again *later*

      Makes me wonder how technical you actually are. What should they have said? "Our systems are currently overloaded. Please do your part by refreshing immediately."?

      I understood that the "please retry in 5 minutes" was an attempt to limit the incoming DDOS by their legitimate users long before I was an SRE.

  10. thondwe

    Fail-over/Fall-over

    What's the betting that's another "transparent fail-over" fail in the ultra-clever routing domain then?

    1. TRT Silver badge

      Re: Fail-over/Fall-over

      It’s good that the search engine is on a different system though. I mean how else do they find a fix for a problem?

      1. Anonymous Coward
        Anonymous Coward

        Re: Fail-over/Fall-over

        Well, they'll have a job logging in to StackExchange sites to find solutions, since those rely on JavaScripts hosted by googleapis (much to the annoyance of many).

        (Makes you wonder how many sites will also be borked if Google's captcha service is also down: even if you don't use sign in with Google, they still try to ensnare you...)

      2. David 132 Silver badge
        Happy

        Re: Fail-over/Fall-over

        Thank you, I now have a lovely mental image of nervously-sweating Google engineers frantically Bing/Qwant'ing for things like

        - "how to reboot a server"

        - "help I forgot my login password"

        - "what does read error mean on floppy disk"

        - "does anyone have a backup of google infrastructure"

        ...while managers scream at them :)

        1. TRT Silver badge

          Re: Fail-over/Fall-over

          Similarly how do you find out if DownDetector is working or not?

          1. Rob Daglish

            Re: Fail-over/Fall-over

            Was it Red Dwarf where they were unable to tell if there was any damage as the damage report machine had been damaged?

            1. TRT Silver badge

              Re: Fail-over/Fall-over

              I think so. Also The Grebulon space ship in HitchHikers.

  11. Anonymous Coward
    Anonymous Coward

    The Register contacted Google to find out

    not until they get their gmail sorted, I suppose... We're all (?) DOOMED!!! :)

  12. Anonymous Coward
    Anonymous Coward

    Ain't those Cloud services supposed to be up 100% of the time?

    Asking for a friend...

    1. TRT Silver badge

      Re: Ain't those Cloud services supposed to be up 100% of the time?

      To be fair the services were probably all up and working just fine. It’s just that no one could get to them to use them.

      1. Anonymous Coward
        Anonymous Coward

        Re: Ain't those Cloud services supposed to be up 100% of the time?

        None of our GCP hosted instances went down even if we couldn't load anything in the GCP console

    2. Antonius_Prime
      Joke

      Re: Ain't those Cloud services supposed to be up 100% of the time?

      Global Warming. It's getting hotter, so the Cloud is evapourating a little...

    3. storner

      Re: Ain't those Cloud services supposed to be up 100% of the time?

      Supposed to? Yes.

      Did Google/AWS/Microsoft/Oracle/RedHat promise that they would be? No.

      Are they? Apparently not.

  13. DJV Silver badge

    As reported by the BBC, a guy on Twitter said: "I’m sitting here in the dark in my toddler’s room because the light is controlled by @Google Home. Rethinking... a lot right now."

    And THAT is why I don't have anything essential in my house controlled via the internet in any form or fashion. But you just can't tell people that - they have to experience the downsides for themselves before they get the clue!

    1. TRT Silver badge

      Mine all have a local controller on the inside of my network as well as being RF controlled without IP so I don’t need internet or network. The light bulbs I have (those that are smart) rather sensibly also turn on if the supply is interrupted and restored so I reckon Twitterman is spouting bullshit.

    2. Cederic Silver badge

      IoT was crushed by corporation demands to control all the data, resulting in cloud services the users don't control. It's lovely to see the people that didn't realise this get a helpful education.

      My favourite quote on the BBC though was "Google has been contacted for comment, but one spokesperson said they were unable to access their email."

      1. Anonymous Custard
        Headmaster

        If you really must use the things, then Home Assistant is your man (or Pi)...

      2. Bitsminer Silver badge

        Use the competition?

        Most mobile phone companies around here have repair crews equipped with....wait for it....phones linked to their competitor's networks. Because, obviously. Repair crews, obviously.

        I wonder how long it takes Microsoft to switch to posting their status on GCP and Google to set their email backup plan to Office 362.

        Until they're both down. Then, what?

        1. Claptrap314 Silver badge

          Re: Use the competition?

          I supported Hangouts. The rest of the Google SREs used us as their primary channel for communication during outages. (I'm pretty sure that they were all on the pre-prod servers.)

          WE used a private irc channel.

      3. tip pc Silver badge

        “ IoT was crushed by corporation demands to control all the data, resulting in cloud services the users don't control. It's lovely to see the people that didn't realise this get a helpful education.”

        Not sure about that.

        In the late 80’s we had a variety of remote controlled plugs, different packs all incompatible with each other.

        IoT means a common controller being Amazon, Google or Apple.

        Guess who has the most privacy conscious IoT implementation.

        If you want more private IoT you need the device to do most of the processing, and the cloud do what is commanded by you and your IoT.

        Case in point is cctv on device person recognition, not cloud recognition.

        Buy cheap pay twice.

        1. Jellied Eel Silver badge

          In the late 80’s we had a variety of remote controlled plugs, different packs all incompatible with each other.

          We had those in the 1580s and compatibility was ensured by a bit of beating & heating. HR keeps those in a locked cupboard. Although clothing may differ in the 21st Century, process is much the same. Once escorted away by security, you don't see them return. Just an occasional reminder from HR to return their pressure washer immediately.

          IoT means a common controller being Amazon, Google or Apple.

          I suspect some may be reconsidering that world view when all their Io Things became inaccessible via their Apple or Google phone, or Amazon app. Just as well that in the good'ol days, T meant 'thumbs', which most people still have. Despite best efforts of IoT vendors to disable those via shrink-plastic packaging.

        2. Doctor Syntax Silver badge

          "Guess who has the most privacy conscious IoT implementation."

          Somebody who keeps it all in-house?

        3. Cederic Silver badge

          You seem to exclude the 'server under my control' option, which is my preference.

          After all, I'm not going to point-to-point every 'thing' in my house. I'll automate and remote control/access them via a server.

          I just don't want a large data slurping company to host it.

        4. Down not across

          I still have bunch of old X10 stuff. Some still controlled by CM10 since it still works.

          Plenty of commercial or open source solutions to control those (and many others like Zigbee etc) all perfectly on-prem on your own hardware with no external network access required.

      4. Doctor Syntax Silver badge

        "one spokesperson said they were unable to access their email."

        How? By email?

    3. Anonymous Coward
      Anonymous Coward

      When I'm unable to control my Philips Hue bulbs via apps or Alexa I use this cunning workaround device called a "light switch", or did he remove them?

    4. KarMann Silver badge
      FAIL

      The hard part…

      …is figuring out whether 'failsafe' for a door lock means it's locked or unlocked if the Internet or cloud or power fails. Either way can be pretty bad. Which is why, while I'm not totally opposed to such 'smart' IoT things*, door locks will be about the last thing I'd 'upgrade' that way.

      * Brought to you by the Redundant Department of Redundancy Department. Ask us about our PIN number for the ATM machine!

      1. bigphil9009

        Re: The hard part…

        "Redundant Department of Redundancy Department". Congratulations! You have totally melted my brain with that one. It feels like I'm one re-read away from entering an infinite loop...

      2. Lazlo Woodbine

        Re: The hard part…

        Within the realm of Access Control, 'Fail Safe' means the lock opens when power is lost, 'Fail Secure' means the lock closes when power is lost.

        1. DJV Silver badge

          Re: The hard part…

          A school local to me found out the hard way that their electronic door locks (the only type they had on them at first) were "fail safe". When they did have a power outage, they discovered (to their surprise), that all and sundry could just walk in and out of the school with no problem. They rapidly "downgraded" to proper old fashioned locks after that.

          1. tip pc Silver badge

            Re: The hard part…

            How often do they have power cuts there?

            1. Doctor Syntax Silver badge

              Re: The hard part…

              One too many.

          2. Nifty Silver badge

            Re: The hard part…

            It stops the doors from getting damaged during the Zombie Apocalypse.

    5. Anonymous South African Coward Bronze badge

      All your IoT devices are belong to us

    6. Martin Summers Silver badge

      Normally smart lights have an override. When you switch the power on to them first time they come on and they function just like any other. This is what happens with my TP-Link bulbs and I've never had any issues if there's been Internet connectivity problems. In fact the app looks for them locally. Just because I can control my lights via Google home or Alexa, it doesn't mean I can't independently control them. Either he's got crap smart bulbs or he doesn't know how to use them (or possibly the switch for that matter).

    7. Natalie Gritpants Jr

      Given the outage was at lunchtime, why not open the curtains? Also, why remote control the light in a toddler's room? Does the toddler have a smartphone? If it's for the adults, why not use the conventional light switch when you walk in the room? Also^2, why haven't you fitted a smart switch with a local button instead of a stupid 'smart' bulb that requires the normal light switch to be left on?

      If you really really want to gaslight your toddler reliably try one of these: https://www.home-assistant.io/blue no internet required as long as you have the correct bulbs/switches.

      1. Anonymous Coward
        Anonymous Coward

        It's not lunchtime all over the world when it's lunchtime in the UK.

        1. Cuddles

          Time is an illusion, lunchtime doubly so.

    8. richardcox13
    9. Anonymous Coward
      Anonymous Coward

      Did he not think of walking over to the light switch to turn on the light.........

      Its called manual override

      1. Claptrap314 Silver badge
        Alien

        You know that never works.

    10. DasWezel
      Happy

      Local OpenHAB install unaffected by Google outage. Engage smug mode.

      That being said, depending on the setup, the light switches at least don't rely on wifi to perform basic functions, they'll still operate as regular light switches even if your whole network dies.

      Not sure how you'd build that sort of redundancy into a light bulb, but that's the price of convenience I guess.

    11. tip pc Silver badge

      TwitterTwat just couldn’t be bothered to go and turn the light on manually, probably thought he’d get a ton of likes and retweet’s if he posted what he did.

      Clearly a feed back loop in progress, rewarding stupid behaviour.

      I saw a trio of leccy company vans outside the other week from ~6pm, bang on 8 the power went out for ~40 mins. My lightwave & smart plug lights went out too. No end of tweeting would have resolved that. I didn’t even have the option of getting off my bottom to manually turn lights back on!!!!!

  14. Wily Veteran
    Happy

    And for a few minutes....

    The world was a better place.

  15. Anonymous Coward
    Anonymous Coward

    Youtube usable via incognito mode

    YouTube is still accessible using incognito mode suggesting something up with Google Accounts, although I can get into the accounts admin page fine so maybe the problem is further down the line.

    So back to looking for amusing cat videos everyone :)

    1. 's water music
      Coat

      Re: Youtube usable via incognito mode

      So back to looking for amusing cat videos everyone :)

      Pussy Galore!

      1. Fruit and Nutcase Silver badge

        Re: Youtube usable via incognito mode

        Mrs Slocombe is not at all amused by all this innuendo about cats and would like it be known that both she and her Pussy is unanimous on that

  16. Queeg

    Eggs and Basket come to mind.

    Googling "How to make the perfect Omelette"...opps

    really missing that sarcasm icon right now

  17. Stoneshop
    Holmes

    they have to experience the downsides for themselves before they get the clue!

    s/sides/times/

    Indeed. Not even referring to the Revolver fiasco, the Otto lock firmware update fiasco and the hundreds of other similar clod-related fiasci will get through to dent the "must have because shiniez" attitude.

    1. Jan 0 Silver badge

      Re: they have to experience the downsides for themselves before they get the clue!

      > Otto lock firmware update fiasco

      Pardon, what firmware? My Ottolock (Google doesn't seem to find any Otto locks) is entirely hardware. Aren't they all? It's only a fancy café stop lock, I don't expect it to resist much more than penknives or nail clippers!

      1. Stoneshop
        Big Brother

        Re: they have to experience the downsides for themselves before they get the clue!

        Ah sorry, mixed up two IdioT lock mishaps. The firmware bricking was by Lockstate, while Otto was developing a finger-scanning door lock but locked its own doors before even delivering a product.

  18. Anonymous Coward
    Anonymous Coward

    The Cloud

    Somebody elses computers you have no control over.

    Stop whining anyway - you sold your soul and private data in order to pay for these free services so you could be harvested, monetized and resold.

  19. Microchip

    On the plus side...

    ... this finally gave me the kick up the arse I needed to pull my mail back onto my own servers, instead of hoping and praying it didn't "accidentally vanish", or at the very least, losing access to it. Free grandfathered G-suite is great, but it does mean absolute trust in the system holding it.

    Hopefully roll-your-own-antispam is a lot better than I remember it being with manual SpamAssassin and DNSBL setups.

    1. anthonyhegedus Silver badge

      Re: On the plus side...

      Roll your own antispam is really hard to do - nearly as hard as making sure all your outgoing mail is actually received!

      Don't bother.

  20. Anonymous Coward
    Anonymous Coward

    TITSUP

    Total Inability To Surveil User Processes

    1. Jan 0 Silver badge

      Re: TITSUP

      Downvoted for nuspeak.

      S is for See or Survey.

      1. Anonymous Coward
        Anonymous Coward

        Re: TITSUP

        wtf is "nuspeak"?

        S is for Surveil

  21. BenDwire Silver badge
    Facepalm

    Same here, but had to have a few tries at re-authentication before it finally stuck.

    In the intervening period I was going to email El Reg to tip them off about the outage, and realised I couldn't easily. (It was too much hassle to find my login credentials to my little-used Outlook account).

    The last memorable time I was faced with a similar dilemma was when I got a replacement soldering iron element ... that had to be soldered on.

  22. Dave 126 Silver badge

    Google Maps

    Google Maps was misbehaving, around 11.40 GMT 'Sorry something went wrong, please try again' and Your phone is offline' upon hitting g the 'Directions' button. Que me staring puzzled at 4G status bar, restarting Maps, toggling mobile data, and turning off WiFi in case it had latched onto a weak signal.

    Haven't seen any report of Google Maps failing, but the timing fits, and no reason to think it was immune from Google's outage

  23. lglethal Silver badge
    Angel

    To think I never even noticed, and if it wasnt for this article, I still wouldnt have known.

    I wonder if that worries Google more (that there are people out there that dont use their services daily) or less than them annoying all their current customers?

  24. Mister Dubious
    Black Helicopters

    Headgear?

    It seems appropriate that the "something went wrong" monkey wears a MAGA hat.

    1. Fruit and Nutcase Silver badge

      Re: Headgear?

      Make America Google Again

  25. Version 1.0 Silver badge

    It must be a local issue?

    I'm have not seen any problems on the Gulf coast in the US, everything is working normally (working most of the time).

    1. Stoneshop
      Headmaster

      Re: It must be a local issue?

      With the outage being reported from the Phillippines to Europe, this clearly is a definition of 'local' of which I was not previously aware.

      1. Bill Gray
        Headmaster

        Re: It must be a local issue?

        I presume, then, you are unfamiliar with the Local Group of Galaxies?

        (As an astronomer, I run into this sort of thing frequently. An event ten million years ago is "recent". Something ten light-years away is "nearby".)

        1. Stoneshop
          Thumb Up

          Re: It must be a local issue?

          I presume, then, you are unfamiliar with the Local Group of Galaxies?

          Now that you mention it, but no, it wasn't on my mind when I wrote that.

          We'll hear more of them once they're connected to the Galactic Area Network, and Google figures out what to punt to these new advertising targets.

  26. romulusnr

    Services that depended on Google Authentication were unusable. Youtube worked fine if you were not logged in. This affected Docs, GMail, Youtube, Photos, etc. Also Nest. Any open documents or data was okay, but new docs were not possible, and opening existing docs after the outage would fail or show old versions.

  27. tip pc Silver badge

    As it took out typically most operations in the impacted regions competitors on the same platform would have had the same treatment.

  28. Anonymous Coward
    Anonymous Coward

    Enterprise application suite...

    So like we lost all access to our 'enterprise grade application suite' which we are sooo super-excited to be working with. Truly, today, for once, I did not have my head in email. Thanks to Google.

    I wonder if Google use Solarwinds, and were remediating *that* issue

    anon because criticism of the decision to migrate to g-suite results in no longer being on the payroll the following week.

    1. Anonymous Coward
      Anonymous Coward

      Re: Enterprise application suite...

      Super-excited sounds very "swedish". If that makes sense for you, I share your pain. It was not the best decision.

  29. John Brown (no body) Silver badge

    Cloud on the ground?

    "Chocolate Factory's cloud came crashing to the ground."

    That would be FOG, yes? Fucked Over Goodly.

  30. RyokuMas
    Joke

    "... due to an internal storage quota issue."

    Translation: "we didn't realise just how much space tracking 7.85 billion people would take up."

    "This was resolved at 4:32AM PT, and all services are now restored."

    Translation: "we've bunged in some more hard drives to tide us over"

    1. Stoneshop
      Big Brother

      "we didn't realise just how much space tracking 7.85 billion people would take up"

      So they should be glad I'm trying to keep as much data away from ChocFac as feasible.

  31. Palpy

    Dec 15, 7 AM Pacific Standard...

    ...GMail was still glitchy on my machine. *shrug*

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like