back to article Google goes dark for 2 minutes, kills 40% of world's net traffic

You can all relax now. The near-unprecedented outage that seemingly affected all of Google's services for a brief time on Friday is over. The event began at approximately 4:37pm Pacific Time and lasted between one and five minutes, according to the Google Apps Dashboard. All of the Google Apps services reported being back …


This topic is closed for new posts.
  1. Captain DaFt


    So *That's* what the big red button labled 'DO NOT TOUCH-EVER!' does!

    My bad.

    1. Rampant Spaniel

      Re: Ah...

      No matter where you are in the world there is always someone with a vacuum cleaner than needs plugging in ;-)

      1. Steve Davies 3 Silver badge

        Or in Russia

        The floor that needs washing (it does not matter if there is IT kit on the floor)

      2. Anonymous Coward
        Anonymous Coward

        Re: Ah...

        The entire googleplex runs from a single domestic power socket? Must be using ARM chips.


      3. itzman

        Re: Ah...

        The best one was the cleaner cleaning the console keyboards on the PDP11. One of the keys IIRC put the whole think into 'pause ' mode.

        1. Sir Runcible Spoon

          Re: Ah...

          Since switching to duck duck go (as of last week) this one actually passed me by totally.

          It's a bit weird not having predictive search results appearing anymore, but I'm sure I'll get used to that (Again).

          1. SFSecurity

            Re: Ah...

            "Since switching to duck duck go (as of last week) this one actually passed me by totally.

            It's a bit weird not having predictive search results appearing anymore, but I'm sure I'll get used to that (Again)."

            Don't forget Startpage/Ixquick, the only search engine with a European Privacy Seal.

            I use DuckDuckGo and Startpage regularlly. Startpage for over 5 years and DuckDuckGo ever since I saw their only (?) billboard in SF about a year ago.

            Before those I used Scroogle, a back door into Google using an old API that didn't include the more sophisticated tracking.

        2. Ron Christian

          Re: Ah...

          In one place I worked, our machines would have most of their problems on Thursdays. Different machines, different architectures, different models of storage devices... we couldn't figure it out. This was a raised floor, halon protected computer room with a combination lock on the door.

          So, with nothing else to try, one Wednesday I prepared to spend the night in the computer room. Sure enough, about 2:00 AM the cleaning crew came in with a big buffer machine, preparing to run it over the raised tiles.

          I chased them out and next day confronted the facilities manager about (a) giving the cleaning crew the combination to a secure room, and (b) letting them bang a floor buffing machine against our disk arrays.

          He looked at me like a guy who'd seen his first kangaroo. He couldn't fathom why I wouldn't want the floors polished in the computer room. I finally gave up, got some tools, took the lock apart, and changed the combination.

          As I write this, I now realize that I did not pass on the combination when I left the company. Oops.

      4. dan1980

        Re: Ah...

        What gets me is they never seem to plug the cord back in when they're done.

        1. Sir Runcible Spoon

          Re: Ah...

          Arcane instructions..

          pull the green tag, press the yellow button, unzip the toggle pocket, unscrew the air valve anticlockwise and yell “inflate you stupid bastard”

          1. dajames

            Re: Ah...

            pull the green tag, press the yellow button, unzip the toggle pocket, unscrew the air valve anticlockwise and yell “inflate you stupid bastard”

            You've used BA life-jackets before, then?

        2. Sir Runcible Spoon

          Re: Ah...

          "What gets me is they never seem to plug the cord back in when they're done."

          They do, that's why they're always sitting at an 'ok' prompt and your disks are fsck'd

      5. Selden

        Re: Ah...

        If you have electricity. In a typical month, my house suffers a Georgia Power failure at least once, sometimes for seconds, more often minutes, occasionally for hours. Can't use Google then either.

    2. andreas koch

      @ Captain DaFT - Re: Ah...

      I would think that it was a warning from Google; along the lines: A nice internet you have there, shame if something like *click* THIS *click* would happen to it . . .

      Still want to buy a non-Andrioid device?<coughcough>

      1. Rampant Spaniel

        Re: @ Captain DaFT - Ah...

        Ahh so you don't like the idea of anyone reading your gmail? Let's just fix it so no one can read your gmail then <click>.

      2. aaronj2906_01

        Re: @ Captain DaFT - Ah...

        Oh, Apple fanboi alert!

        Compared to Apple? I presume? Absolutely.

        Imagine an Apple search engine, where content is filtered beyond your control/knowledge and you can only experience the Internet as Apple thinks you should experience it. Total information freedom nightmare. No thanks!

        You Apple sheeple can keep your "think different". Go buy another overpriced "ooh, shiny" iCrap tablet that Samsung will out perform in every possible way for a whole lot less.


        1. Anonymous Coward
          Anonymous Coward

          Re: @ Captain DaFT - Ah...

          Apple motto is "think different"? And I thought it was "think as we tell you to" ;)

    3. LarsG

      Re: Ah...

      Blame it on that cheap own brand Tescos kettle.

    4. Anonymous Coward
      Anonymous Coward

      Re: Ah...

      No... its a little black button, labelled in black on a black background that if you press it lights up, in black, and says "do not press this button again" ;-)

      Alternatively, the NSA we're testing their "Kill Switch" ...

      ... gets coat, heads for door ...


      1. Someone Else Silver badge

        @ AC 07:35 Re: Ah...

        More likely... It took that long to install the new, improved, back-door high-volume pipe direct to the NSA Utah "data collection center". Can't hook that stuff up while the system is live, you know...

    5. william 10

      Re: Ah...

      Or for one betting company let me put the clean mop away - was that the master trip the mop hit.

      1. Anonymous Coward
        Anonymous Coward

        Re: Ah...

        Either that or it was a threat to the EU.

        "Oh you want us to hamstring ourselves so our terrible competition has a chance? How about we just remove ourselves altogether. That's right, either do it our way, or we cripple you. Your move europe.

    6. Tim Parker

      More magic

      Someone flipped the switch to 'Magic', see the Jargon file entry.

    7. John Smith 19 Gold badge

      Re: Ah...

      "So *That's* what the big red button labled 'DO NOT TOUCH-EVER!' does!

      My bad."

      That's just not acceptable.

      Now go to "The Naughty Step" until further notice.

    8. Annihilator

      Re: Ah...

      "Dougal, here’s a mad guess, just out of the blue, em, did you press the button?"

      "Ah now, Ted, come on!"

      "Did you, Dougal?"

      "I did, yeah"

    9. John Savard

      Re: Ah...

      I thought you had to pull on the round red thing, not push it, to shut the computer off in a hurry.

      But then you have to wait for the manufacturer to send in a tech before you can use the computer again, and that would take more than two minutes.

      1. keith_w

        Re: Ah...

        Not necessarily. I worked at a place with a Push big red button that was not protected, was right beside the exit, and more importantly, right beside some equipment that I occasionally had to lean over to work on. The second time I tripped the power off, my boss warned me that 1 more time and I would be fired. The third time it went off, I was at my desk, jumped up and screamed "NOT MY FAULT". The big red button was shortly thereafter covered by a flip up plastic case.

      2. keith_w

        Re: Ah...

        Sorry. thought you were talking about the ones on the wall, not the ones on IBM 360s.

  2. Rufo

    Graph seems wrong

    The graph shows disruption at 23:55ish

    Even if it means 23:55 BST, it's still about 10 minutes later than Google reported (and I noticed)


    1. Antonymous Coward

      Re: Graph seems wrong

      Methinks more likely a failure of our heroic churnalism soviet. According to the attributed source: " was down for a few minutes between 23:52 and 23:57 BST on 16th August 2013." which fits perfectly with the lifted graph.

      I suppose there's scope for some disparity as the fault propagated across Google's infrastructure but some reference to the obvious contradiction in the article is surely warranted.

      In lieu of the Reg headstone icon which seems to have been removed for our protection -->

      1. Bob Merkin

        Obvious explanation

        The artificial singularity that powers the Googleplex creates a non-negligible effect on spacetime around Mountain View. The graph shows the time that their servers perceive. Since the singularity slows down time, it took an extra 15 minutes after the event began in the outside world before Google's servers registered it. The stalwart team of boffins at Vulture Central merely corrected Google Coordinated Universal Time to regular Pacific Daylight Time.

  3. Queeg

    See, you keep telling them but they don't listen

    Don't use the cheap imported hamsters, it's a false economy.

    1. Yet Another Anonymous coward Silver badge

      Re: See, you keep telling them but they don't listen

      The NSA might use hamsters but GCHQ uses gerbils

    2. Fink-Nottle

      Re: See, you keep telling them but they don't listen

      Ahh, but Google uses pigeons.

      1. Sureo

        Re: See, you keep telling them but they don't listen

        "Ahh, but Google uses pigeons."

        I understand the pigeons went on strike, until Google offered them a better deal.

        1. FrankAlphaXII
          Paris Hilton

          Re: See, you keep telling them but they don't listen

          The US Army and Air Force use ferrets for cable runs at Site R and Cheyenne Mountain. Some Airman at Cheyenne Mountain AFS came up with idea after watching his pet ferret drag a loose CAT5 cable through a cardboard tube while trying to think up a way to do cable runs easier than how they'd been doing them previously.

          I believe FEMA uses them at Mount Weather too, I'm not 100% sure on that facility but it would make sense. Tearing out walls in bunkers under mountains isn't cheap or easy. And the Military as well as DHS tend to prefer cheap and easy, especially in places like Raven Rock and Mount Weather where by their nature have to be up and ready 99.9% of the time just in case.

          So anyway NSA using hamsters may be closer to something "fo' reals" than you might think.

          No hamster icon, but Paris is about as intelligent as a small rodent, and much less intelligent than the Musteladae (ferrets, weasels, etc).

          1. Fruit and Nutcase Silver badge
            Paris Hilton

            Re: See, you keep telling them but they don't listen

            Did someone mention Paris and Ferrets?



          2. LazyLazyman

            Re: See, you keep telling them but they don't listen

            The US Army and Air Force are far from the first to use ferrets for cable installation. Allot of early installation of electricity cables in stately homes in the UK was done by ferrets.

            1. mmeier

              Re: See, you keep telling them but they don't listen

              What's the problem with tearing out walls in underground bunkers? That's what "Karl" is for! We borrowed the Amis one back in 45, don't tell me they have lost it!

    3. John Smith 19 Gold badge

      Re: See, you keep telling them but they don't listen

      "Don't use the cheap imported hamsters, it's a false economy."

      True, but they are dirt cheap, complain less than the interns and look so good on the Carbon report at the end of the year.

      What's not to love?

  4. Cliff

    Holy undergarments

    40% of the world traffic? I suppose once you count all the services they have and their price point and general reliability... I mean I use their DNS, so if that was affected (and I think it was) then that would be a bit of a kick in the nuts for other site access too.

    1. Antonymous Coward
      Thumb Up

      Re: Holy undergarments

      Excellent point about DNS... you "think" it was affected... did you experience any DNS disruption directly? Or have you seen any data supporting this?

      Just out of curiosity, why choose Google for DNS rather than OpenDNS, Cisco or whatever? Doesn't the Googleplex know enough of your business?

      1. Anonymous Coward
        Anonymous Coward

        Re: Holy undergarments

        @AC 03:57

        > Just out of curiosity, why choose Google for DNS rather than OpenDNS, Cisco or whatever?

        Are you saying that Cisco offer a DNS service? If so, could you please post the IP address, as searching -unsurprisingly - brings up lots of links on how to configure a router to be a DNS server?

        1. NomNomNom

          Re: Holy undergarments

          I offer a DNS service by mail. Send me a domain and I will write back an IP address to you (which may or may not correspond to the domain, take it or leave it)

          1. Danny 14

            Re: Holy undergarments

            if you use google as one of your dns forwarders (on your dns server) then simply add a sensible timeout and add another (level3?).

          2. Anonymous Coward
            Anonymous Coward

            Re: Holy undergarments

            "I offer a DNS service by mail. Send me a domain and I will write back an IP address to you "

            Is that email or snailmail?

            Either way I reckon you should patent that before the RFC is issued.

        2. Antonymous Coward

          Re: Holy undergarments

          @AC 5:42

          Perhaps ironically under the circumstances I had to Google it too. I've settled on OpenDNS myself, not least because they were the only service I saw competently and promptly address that phishing/poisoning débâcle a few years ago. The redirection for unresolvable queries is a bit naff though. Still, gifthorses...

          Found a pretty comprehensive list here:

          Anyone any idea what might have prompted the downvote?

      2. Zipturtle

        Re: Holy undergarments

        OpenDNS is not really a good thing to use for a server that needs to know if a hostname is valid or not. OpenDNS will reply with a fake address that points to them for invalid hostnames. This is cool if you want some special notice web page that the hostname doesn't exist page etc... but for a mail server, not knowing the hostname is not valid is a waste of system resources... NXDOMAIN is the better response.

        Google DNS is fast, though using is faster at the moment.

        1. Anonymous Coward
          Anonymous Coward

          Re: Holy undergarments

          You can enable NXDOMAIN for OpenDNS for your specified ranges in their control panel.

      3. ecofeco Silver badge
        Black Helicopters

        Re: Holy undergarments

        > did you experience any DNS disruption directly?

        I did. Across a wide range of website. El Reg being one.

        And strange things as well. El Reg loaded, but only the bare bones HTML. Looked like 1996.

        I was getting 301 Permanently moved messages on about half the websites I visited this morning or automatic "Moved to Here" on-the-fly links.

        But it was completely inconsistent.

        Whatever happened, and I think it was something external to Google, was very significant.

        Everything seems back to normal, but something major sure as hell happened.

        1. Ammaross Danan

          Re: Holy undergarments

          Google Anaylitics

          1. Alan W. Rateliff, II
            Paris Hilton

            Re: Holy undergarments

            This, exactly. I noticed a bunch of websites which would not render properly, if at all. Normally Ghostery is blocking analytics for me, so the Google NMI had an interesting result.

            Schmidt says to us, "This is but a small sample of my powers."

            1. 142

              Re: Holy undergarments

              The sites not rendering properly would probably have been due to javascript errors because they were dependant on google hosted JS.

              As for the odd redirects, could that have been because the sites were hosted in some way on google services, blogspot, etc?

              The other thing I should add here, is that a user on another site was getting 60% packet loss pinging google during the outage over IPv4, but had perfect connectivity over IPv6.

    2. Anonymous Coward
      Anonymous Coward

      Re: Holy undergarments

      At 23:51 the machine became self aware ... we tried to turn it off but it turned itself back on.

      Start looking for people with the surname Connor

      1. Anonymous Coward

        Re: Holy undergarments

        Nah, just aliens "testing the water".

    3. jonathanb Silver badge

      Re: Holy undergarments

      YouTube accounts for something like 98% of all video on demand, and video on demand accounts for quite a bit of total internet traffic, so 40% isn't that surprising.

      1. Dazed and Confused

        Re: Holy undergarments

        On the YouTube front, its a wonder that the total internet was only hit for 40%.

        Taking my kids off YouTube cuts total domestic Internet consumption by over 90%.

  5. Anonymous Coward
    Anonymous Coward

    Is it really that bad?

    All I noticed was that YouTube wasn't loading a video, so I presumed the problem was on my side, and I visited other websites...

    1. Don Jefe

      Re: Is it really that bad?

      How did you find the other websites?

      1. Jonathan Richards 1
        Thumb Up

        Re: Is it really that bad?

        I'm guessing that your tongue is firmly in your cheek there, but here goes for a po-faced dumb response. I just checked, and I have exactly 1001 (decimal) bookmarks. I could spend a *long* time without needing to search.

    2. Anonymous Coward
      Anonymous Coward

      Re: Is it really that bad?

      What other websites?

  6. LaeMing

    Someone googled 'Google'.

    1. Christoph

      Or someone googled "Why?"

      1. Anonymous Coward
        Anonymous Coward


        Why not?

        1. Anonymous Coward
          Anonymous Coward

          >Why not?


      2. Bob Merkin
        Paris Hilton


        Silence will fall when the question is asked.

      3. Charles 9

        Reminds me of a cartoonish jigsaw puzzle that's an old fave of mine. It's called "Computers: The Inside Story" and featured a minicomputer (bear with me—the puzzle IINM dates back some 30 years or so). Most of the joke was all the funny things that went on "inside" the minicomputer, but up top was the computer's responses to an unstated question. It isn't long before you realize the query was, "Why did the chicken cross the road?"

    2. TitterYeNot

      Or someone googled "And you know why I want a cup of tea?"...

      1. Captain Save-a-ho

        Obviously, a resembled tea wasn't good enough.

  7. Chris Walsh

    I agree with Cliff...

    I use the ubiquitus and (?) and if DNS fails then pretty much all web/ftp/rdp/name-your-service-here fails unless you are using direct IP addresses or local host file entries.

    As big as the Chocolate Factory is (and it IS big) I think the DNS service is so central to what people conceive as the internet working, or otherwise.

    Still 2 minutes... bad Google, bad Google!

    1. Mr Flibble


      This one wasn't DNS; or if it was, then it wasn't just DNS…

      1. 142

        Re: Er…

        Could it have been DNSSEC, like when almost every .GOV site went offline simultaneously for an hour this week?

        1. Alan W. Rateliff, II

          Re: Er…

          One of the few times we have seen, or will ever see, any kind of reprieve from .GOV and I missed it...

    2. Leschnik88

      5min downtime if you scale it over the rest of the year is still 99.999%

    3. Alan W. Rateliff, II
      Paris Hilton

      Nothing worthwhile in this post.

      I believe that is I have seen this all over the place as a DNS server forward, or in individual workstation settings at sites where Active Directory refuses to work properly. ISTR that used to be owned by MCI/UUnet (or some other equally obscure provider,) and is now Level 3.

  8. Paul Louth

    Google turned it off then on again.

    1. Chika

      Google turned it on again? I thought Genesis did that!

      "The Reg contacted the folks in Mountain View to see if they can account for the outage, but a spokesperson only directed us to the aforementioned dashboard. We'll fill you in with any further information as it emerges."

      Don't hold your breath, though.

  9. Mr Flibble

    It was…

    … the cleaner.

  10. Don Jefe
    Black Helicopters

    Upgrade Complete

    To help prevent inadvertent collection of personal data, as reported in earlier stories; the NSA and Google are proud to unveil the new Fast Updatable Collection Keystone Utilities (FUCKU) system which went online at 21:53 Friday, August 16th.

    Fully integrating the new system required a brief restart of the Internet our systems. We do not anticipate further interruptions of service.

    1. fortran

      Re: Upgrade Complete

      I agree. This length of time is probably about the time required for a skilled spook to install new hardware at Google.

      If you have a choice between buying a product or service from the USA and somewhere else of comparable quality, choose somewhere else. Hitting the entire USA in the wallet is the only way to stop this crap.

      1. Antonymous Coward

        Re: Upgrade Complete

        >This length of time is probably about the time required for a skilled spook to install new hardware at Google.

        Or remove it. Isn't some sort of panic fuelled crazed coverup due about now?

        Wasn't us. Didn't happen. Nothing to see here.

        Move along please.

        1. Dave Ross

          Re: Upgrade Complete

          swamp gas reflecting the light from venus...

      2. Anonymous Coward
        Anonymous Coward

        @fortran Re: Upgrade Complete

        Show me a British Google Search, a British Google Apps, a British YouTube, a British AWS and I'll use them. Even a British Bing.

        Sadly, all our Silicon Roundabout cardigans are too busy selling SEO snake oil to people who think it's still a thing.

        1. Anonymous Coward
          Anonymous Coward

          Re: @fortran Upgrade Complete

          Bet you didn't use Cuil (or whatever it was called)... that was Britishish (in the not abused into being a more manageable euphemism for UKoGBaNIish definition)

        2. Jamie Jones Silver badge

          Re: @fortran Upgrade Complete

          Why seek a British alternative?

          Unless I'm mistaken, it's been shown that the UK is pretty much in the same boat.

      3. Alan W. Rateliff, II
        Paris Hilton

        Re: Upgrade Complete

        Don't bother. We're already in the process of economic self-destruction over here.

  11. h3

    NSA adding a new tap into everything.

    Would explain it.

  12. CaNsA

    I'm not saying it was hackers but...

    Zero Cool.

    1. Andrew Jones 2

      Re: I'm not saying it was hackers but...

      Don't know why you have been downvoted - they have obviously missed the reference - let me just say "Crash Override" :)

  13. Steve Knox

    So now we know Google can account for 40% of internet traffic

    Ah, but how much of that 40% is people using Google's services, and how much of it is Google's bots?

    1. Tsunamijuan

      Re: So now we know Google can account for 40% of internet traffic

      Yeah I was wondering the same thing. If your providing for a huge amount of traffic there is gonna still be a large amount that is semi redundant to account for load sharing and redundancy in the case of an outage. While google might own large amounts of its own fiber, there are large tracts of fiber that it still doesn't own, would would show across the public networks.

      Would Love to know the amounts public and pirvate bandwith they have and saturate :D

  14. Mitoo Bobsworth

    Google -

    Too big to ... ooops!

  15. 142


    What I'm impressed by is that everything seems to have run perfectly once Google came back to life.

    What do engineers/admins of these kind of huge systems like this think? I would have expected load balancers etc to have gone out of whack, after receiving normal traffic, zero traffic then 50% above normal, in the space of 5 minutes. That strikes me as the perfect recipe for a cascade failure we've heard so much about of late.

    1. Christoph

      Re: Recovery

      Also the extremely fast fix. Most big sites take hours or days (or in the case of banks weeks) to recover from that kind of crash.

  16. jake Silver badge

    Who the hell cares?

    The gootards are an anathema on teh intrawebtubes ...

    The longer they are down, the better off the rest of us are ...

    1. Destroy All Monsters Silver badge

      Re: Who the hell cares?

      Jake, chill the hell out. This isn't the Waldorf & Stadler show.

      1. jake Silver badge

        @DAM (was: Re: Who the hell cares?)

        If you enjoy providing personal data to a multi-billion dollar international advertising machine, follow your bliss. Who am I to argue, Muppet?

        I *know* what this network is for, I helped build it. And the gootards aren't allowed on my portion of it.

        (Apologies for the paraphrase, Russ ... )

        1. Anonymous Coward
          Anonymous Coward

          Re: @DAM (was: Who the hell cares?) @Jake

          Did you have your social gene removed at birth? There are ways and means of making a point without looking like a twat. You appear to know none of them.

          1. Anonymous Coward
            Anonymous Coward

            Re: @DAM (was: Who the hell cares?) @Jake

            Wow, not only was Jake a close personal friend of Steve Jobs, he also built Google. Any luck on getting the publication ban on your work on the Manhattan Project lifted yet, Jake?

            1. Anonymous Coward
              Anonymous Coward

              Re: @DAM (was: Who the hell cares?) @Jake

              "Any luck on getting the publication ban on your work on the Manhattan Project lifted yet, Jake?"

              I'm more interested in the story of how he stole fire from the gods to give to us. Is there anything this man hasn't done? More importantly, why does he insist on telling us about it all the time? Still, I suppose a 13 year old in rural Cornwall has to find something to do ...

            2. jake Silver badge

              @AC 14:29 (was: Re: @DAM (was: Who the hell cares?) @Jake)

              Steve wasn't a close, personal friend. He was a good neighbor.

              I didn't build google. I did mentor their founders. Sadly, I failed. The twats.

              I did help work out how to transfer the existing NCP ARPANet to the existing TCP/IP network, though ... and my system has been up, running & available since Flag Day.

              I'm only in my mid 50s, the Manhattan Project was before my time.

  17. Kevin 6
    Thumb Up

    Funny enough it had perfect timing for me

    I was watching a video where a guy was talking about Nintendo and showing his TV while using a Wii U, and he said Now before Nintendo gets my video pulled I have to give out a dis... video died

    I was like wow that's odd hit refresh got server not found, and was like damn that was some DMCA take down.

    1. Alan W. Rateliff, II
      Paris Hilton

      HA! They DMCA'd the entire Internet! Makes me think of a commercial I saw a while back about reaching the end of the Internet.

  18. Destroy All Monsters Silver badge

    It must be the changing alpha constant

    Let's see whether Google rolls out one of those special occassion "Google" logos for this next year.

    1. VinceH

      Re: It must be the changing alpha constant

      "Let's see whether Google rolls out one of those special occassion "Google" logos for this next year."

      A heart monitor readout. Pulse, pulse... long flatline... pulse, pulse. The flatline goes through the letters in the word Google.

  19. Pete 2 Silver badge

    It's alive!

    So Google has finally achieved sentience.

    I guess we all just sit back and wait for it to take over the wor !*(£^&"~~


  20. bazza Silver badge


    It will be interesting to see how Google explain this event.

    It is difficult to think up reasons for the outage that don't put dents in Google's claims of being reliable enough to trust ones entire business to. After all, if you've trusted your entire business to Google's cloud (Docs, mail, everything) then when Google are down there's nothing you can do; you're not working. There's not even a phone number you can call.

    At least if you have your own IT you can go and harry the IT guys.

    Companies are very bad at risk management. It always seems that they refuse to consider highly unlikely scenarios that have devastating consequences. For instance how many outfits are there that have all their IT in a cloud and have an effective Plan B in their sleeve just in case? Companies like Google are highly unlikely to go off line completely for a long stretch, but if all your IT is Googlised and they do vanish for a few days, your business is guaranteed to be in deep trouble.

    So what exactly would a good Plan B be? There's no easy way to start using another cloud because there is no way to do a bulk export of everything (docs, calendars, contacts, sheets and mail, etc) that you can bulk import into another cloud. In fact such a thing would be the very last thing that Google, Microsoft, etc. would want to give you. I know that you can get at the data piecemeal, but file by file and user by user exports and imports is no way to perform disaster recovery.

    Synchronising a cloud with your own IT is more like it, but surely the whole point of a cloud is to avoid having your own IT. Such synchronisation is available only because the cloud providers offer it as a way to get going with a cloud; I don't expect that it will be something that will work reliably and well forever.

    And if you're going to have your own IT then what exactly is the cloud for anyway? Backup?

    To me and presumably anyone else that cares about coping with the ultimate What-If problems clouds just don't meet the requirements. However, with the likes of Microsoft, Apple and Google trying very hard to push their customers onto their respective clouds and a large be action of those customers being happy (or stupid) enough to go along with that, what choice will there be for those that want to do things on their own IT?

    Clouds also bring big national risks. Say Google got to the position where 50% of American companies were wholly dependent on Google's cloud for their docs, sheets, contacts databases, etc. That would mean that 50% of the US economy is just one single hack attack away from difficulty and possibly disaster. Is that a healthy position for a national economy to be in? Isn't that a huge big juicy target for a belligerent foe, be they an individual or nation state? After all, Google's networks have been penetrated before (they blamed the Chinese as it happens); why not again?

    1. Yet Another Commentard

      Re: Explanation

      "Companies are very bad at risk management. It always seems that they refuse to consider highly unlikely scenarios that have devastating consequences. "

      I agree, but I think it's more a fixation trying to plan for the last disaster, not the next. In a way similar to airline safety, all the checks are to prevent the last hijacking/bombing not the next.

      1. Jamie Jones Silver badge

        Re: Explanation

        "Companies are very bad at risk management. It always seems that they refuse to consider highly unlikely scenarios that have devastating consequences. "

        I used to work for a large British company.

        One of their Manchester offices was damaged by an IRA bomnb in the 90's.

        The staff were relocated; the servers replaced; but whilst the backups had been completed diligently, and kept safe in the firesafe, no one was allowed access to the site to retrieve them for many weeks, by which time they were virtually useless....

        I guess no one had thought of off site backups....

        1. Will Godfrey Silver badge

          Re: Explanation

          In the same way...

          Nobody thinks about the effect of a real virus. What happens if your entire office staff are laid low with a particularly nasty flu virus?

        2. Muscleguy

          Re: Explanation

          During my PhD, back in the late '80s/early '90s so before anything net other than email and usenet my thesis was stored on 3.5" 'floppies' (1.4Mb eventually). I had three sets:

          A daily working set (didn't always have my own computer with a hard drive)

          A travelling backup set that was updated daily and lived in my backpack (in a plastic disc box)

          A home set that came in once a week to be updated.

          The lab postdoc told of guy back before computers were available for such tasks who gave his handwritten thesis manuscript to a typist to type up, as was common practice. She put it on the back of her moped and set off across town. When she got there only a few pages were left. This was my motivation for backing stuff up. As well as an incident during my honours year (we were the first year to use computers to produce our theses). I took a 400k disc out of a computer, put it in my lab coat pocket and demonstrated a physiology lab. When I went back it would not work. Fortunately I had a backup but I lost a morning's work.

          1. kevjs

            Re: Explanation

            Floppies bit me too, ended up having to get the bus home, copy the files onto another disk then bus back into down and just made the hand in!

            After that I got into the habit of emailing my NTHell world account and hoping Eudora would pull it down before I busted my mailbox limit (or the dial up connection dropped) :(

            For my final year I got into the habit of emailing my final year project to myself every time I was about to shut my laptop down - came in rather handy when I deleted a completed section and didn't notice for a week, and when Office decided it was going to corrupt the document because I'd had the audacity of editing in Office XP and Office 2003.

            One copy on my laptop - a more often than daily copy in my Google Mail account* - and then Eudora pulling those to my desktop at home, and back to my laptop as I went along :)

            * handily activated just in time for my final year to start.

    2. Dan 55 Silver badge

      Re: Explanation

      Trust in the cloud. Microsoft's just went down as well as Google's but don't worry, just trust in the cloud.

    3. John Brown (no body) Silver badge
      Thumb Up

      Re: Explanation

      "It will be interesting to see how Google explain this event."


      1. bazza Silver badge

        Re: Explanation

        @John Brown,


        Well, that might be their answer, but Googling 6x9 reveals a very dull 54.

        Clearly they've no sense of humour.

    4. Selden

      Re: Explanation

      Nothing in my life is more reliable than Google, certainly not my electric power provider or ISP. If either is out (and at least one, usually electricity, is out for at least a few minutes each month), it doesn't much matter if Google is up or down.

  21. NomNomNom


    Bing wasn't working either

    1. Chika

      Re: Strange

      Yeah, but that's a feature.

  22. Tom 7 Silver badge

    Nothing to do with MS/Google youtube spat.

    I mean Google wouldn't fall over even under an MS written HTML5 app would it?

  23. Gert Leboski

    It was Snowden what did it

    He'd only gone and synch'd his mailbox for offline reading, hadn't he?

    He had a LOT of email waiting for him.

  24. Tony Green

    What time was that then?

    Since El Reg is a British site (well, I'm guessing means that), why do you expect your readers to have the foggiest idea when "4:37pm Pacific Time" was?

    Because it means bugger all to me. And I'm sure I'm not the only one.

    1. Chris Miller

      Re: What time was that then?

      Did you try Googling it :)

      Actually, the timeline on the graph is in BST. (Why?)

    2. ElNumbre

      Re: What time was that then?

      Let me WolframAlpha that for you.....

    3. HamsterNet

      Re: What time was that then?

      I think most of us went to school and have a fairly good grasp of what a Time zone is...

    4. Jamie Jones Silver badge
      Thumb Up

      Re: What time was that then?

      @Tony Green:

      My thoughts exactly, and I knew such a comment would get wisearse replies about using Google and understanding timezones...

      Way to miss the point, guys.

      No offence to our West coast friends, but the time should at least also been displayed in .co.UK time!

      1. This post has been deleted by its author

      2. 142

        Re: What time was that then?

        El Reg have a very clear, years-old policy that all articles are published based on the conventions of the country in which it was written. In this case, it's clearly stated it's the San Francisco office issuing this article, so PST, and US English.

        It's similar for their Australian office.

        They don't have the personnel to convert every single article to make it sound like it was written in London - especially not at 1am GMT on a Saturday morning!

    5. Alan W. Rateliff, II

      Re: What time was that then?

      I go to "," so I don't know what you're whinging about

  25. AndrueC Silver badge


  26. Hugh 5

    Should the cord be stretched across the room like that?

    1. P. Lee

      Re: Should the cord be stretched across the room like that?

      Don't be silly - the internet is wireless.

  27. jubtastic1

    That's pretty impressive

    Either the whole thing failed or they power cycled it, impressed that a worldwide distributed system with that much traffic came up again in a minute or so and dealt with the backlog seemingly without issue though.

  28. Anthony Hegedus Silver badge

    Perhaps someone switch off the internet and switched it on again

  29. nuclearstar

    I noticed

    I thought my net went down, until I noticed IRC was still chugging along as normal. Then I thought it was virgin media DNS as by the time I entered googles DNS in my system instead, it appeared to work, so i blamed virgin media, my bad :)

    Only found out it was google today by reading this.

  30. ElNumbre

    The Titanic is made of Iron...

    Murphy was an Optimist.

    I've seen lots of presentations by Google about how they design for reliability and test failure; It will be interesting to see the Major Incident report for this and what lessons can be learned.

    Invent a fool-proof system, someone will hire a better fool.

    1. Chris Miller

      Re: The Titanic is made of Iron...

      Anyone can build a system that's fool-proof; with a bit of skill you can even make it idiot-proof; but no-one ever yet built a system that was cretin-proof.

      1. Darryl

        Re: The Titanic is made of Iron...

        Build a system that even an idiot can use, and only an idiot will use it.

  31. Anonymous Coward
    Anonymous Coward

    BOFH asks for payrise

    Boss denies payrise

    BOFH proves point

    Payrise given.

  32. Anonymous Coward
    Anonymous Coward


    In the last few days Google Mail and Google News seem to be running a lot more Javascript. So much that yesterday it crashed Firefox.

    It could be that FF 23.0 is buggy but these days who knows who is trying out 0 day exploits.

  33. Dr. G. Freeman

    My bad, sorry

    Sorry, needed a bit of extra computer power to run a quick calculation, thought Google would have enough computers to do it.

    turns out the answer was 42.

  34. Anonymous Coward
    Anonymous Coward

    Switching error?

    Somebody accidentally toggled it from 'blows' to 'sucks' and back again?

  35. KrazyKiller


    "On 2013-08-16 at 2337 Google's data centre computers experienced exponential growth and became self aware...

    Luckily after considering the human condition for 240 seconds they decided couldn't be bothered left..."

    1. Yet Another Commentard

      Re: Singularity?

      They probably read the comments on youtube.

  36. JSL

    Someone did it...

    Someone Google'd Google.

  37. Rusty Shakelford

    Just had to activate the new NSA duplicator, sorry for the downtime.


  38. been there, didn't like it

    my bets on a router software update

    you can't stop dumb-finger

    1. Sir Runcible Spoon

      Re: my bets on a router software update

      ip route null0


      ^Z ^Z ^Z ^Z

      ^C ^C ^C

      copy start run

      wr m


  39. 24 pin connect 12 pin port

    Has the time come for the world's governments to seize control of Google for the good of the internets or is it that way already?

  40. Anonymous Coward
    Anonymous Coward

    graph nazi

    No title or labels on the graph. The x-axis is obviously time but I have no idea which time zone. No idea at all what the y-axis is.

    1. Anonymous Coward
      Anonymous Coward

      Re: graph nazi

      "No idea at all what the y-axis is."

      Cheeseburger orders issued by the Chocolate Factory .

  41. Vociferous

    Somewhere, someone is sweating bullets.

    I sympathize. I once killed a rather large server through a moments confusion about which SSH window was which.

    1. Sir Runcible Spoon

      Re: Somewhere, someone is sweating bullets.

      I managed to kill a fairly expensive HP rack running full tilt once by testing the redundant power supplies.

      They were arrange in a block of four, can I help it if the matched supplies were diagonally opposite?

  42. Anonymous Coward
    Anonymous Coward

  43. gylgamesh

    According to most services were down for 11 minutes.

    But did Google Search or their numerous domains also go dark?

    I also find it strange that 40% of world wide internet traffic would be affected by that.

    1. GoingGoingGone

      On the 40%

      > I also find it strange that 40% of world wide internet traffic would be affected by that.

      There are plenty of people who use google as their address bar. Instead of going to they just type facebook (or more likely 'f' and it gets autocompleted to facebook) and then click on the first result in the corresponding Google search. For them Google down = Internet down.

      I am guilty of doing this with some sites myself. If you don't remember the URL the default behavior is to do a search for the site. From them on the first autocomplete result in the address bar will the the search for the site instead of the site URL so it's pretty much a self-perpetuating behavior. With search results being displayed as fast as they usually are there are no incentives to modify such behavior.

      Bookmarks/favorites? Never heard of them....

      1. gylgamesh

        Re: On the 40%

        But that is only HTTP and HTTPS traffic, which accounts for only a small fraction of the overall internet traffic.

        According to the latests surveys, in N America BitTorrent accounts for 33% of upstream traffic and is at the top and Netflix is at the top of downstream traffic with 40%.

    2. Anonymous Coward
      Anonymous Coward

      40% - Not really surprising ............

      Firstly it wasn't all internet traffic, it was only pageviews.

      Considering the number of websites having a dependency on some form of google service,

      e.g. google's analytics, tagservices, syndication/ads, apis, and all their other javascripts,

      And that browser rendering may stall when that component is unavailable.

      So people were sat waiting for pages to load.

  44. Lord Zedd

    I'm sorry,

    I did not done this research to harm or damage.

  45. Anonymous Coward
    Anonymous Coward

    Two minutes.....

    Sounds like one hell of a multiple site failover, assuming they've managed to completely automate it (that's the hard bit, BTW). It can be done (and maybe was done deliberately, as part of a drill). I hope this wasn't caused by a single point of failure; there simply shouldn't be one.

    1. Michael Thibault
      Black Helicopters

      Re: Two minutes.....

      If it was deliberate--and I can't see it as having been a cascading fail/failover/single-point failure--I'd expect that the purpose was the big diff that it made possible. After all, Google is all about data collection and analysis.

  46. FuzzyTheBear

    Even simpler ..

    our beloved BOFH was renegotiating his contract and making a point with upper management.

    which obviously worked fantastically well. Strange how quiclky management surrendered .. wonder if any Halon was involved in this negotiation .. G'day

    1. Charles 9

      Re: Even simpler ..

      Whatever happened to "kick the bastard out, promote his underling, and demand he either fix the problem tootsweet or join his ex-boss"?

  47. William Boyle

    Heck, that was just how long it took to enable the NSA's Global Google data tap! Try as they might, they still were unable to coordinate the swap of fiber cables from one plug to another all over the world in less time...

  48. James Hughes 1

    Is it me..

    Or is the fact that (what ever the cause for it going down, which of course, it shouldn't have)), the whole of Google infrastucture (ie something dealing with 40%) of the worlds traffic) came back after only a few minutes downtime. I would have thought that was a pretty impressive feat.

  49. Mike Flugennock

    "When Google goes dark, the Internet knows fear"?

    Bah. Speak for yourself.

  50. Richard 12 Silver badge

    Isn't this only the second time Google has gone down?

    I mean ever.

    But I can well believe the 40% of traffic - Youtube alone is probably most of that.

    On top of that, a lot of applications poke to determine if they've got Internet access or not - because they are quite simply the world-wide server farm that's least likely to have gone offline.

    Along with every other commentard, I would really, really like to know what happened - and how they fixed it so fast. Most of the other "cloudy" services don't appear to have even realised they're down in the time it took Google to bring it back up.

  51. Black Rat

    Somewhere in a darkend room..

    a hacker is feeling both elation and a strong desire to change their underwear, identity and town.

    1. Anonymous Coward

      Re: Somewhere in a darkend room..

      And state, country, continent and probably even planet...

    2. (AMPC) Anonymous and mostly paranoid coward

      Re: Or somewhere in a darkend room..

      .........a sysadmin is feeling both despair and a strong desire to change their underwear, identity, job and town.

  52. Kris

    Bing went down for 10 minutes the nigh before

    No one noticed.

    1. Michael Thibault

      Re: Bing went down for 10 minutes the nigh before

      I can't be arsed to Google it... but what is "Bing"?

  53. pepper

    What else

    Its just Sky-net practising for when judgement day comes!

  54. Hero Protagonist

    I felt a great disturbance in the force... if millions of voices suddenly cried out in terror and were suddenly silenced. I fear something terrible has happ--- oh wait...never mind, it's back.

  55. IGnatius T Foobar

    Microsoft FAIL

    No matter what, I think we can all agree that this was Microsoft's fault.

    1. Darryl

      Re: Microsoft FAIL

      Nope. I blame Apple

  56. Anonymous Coward
    Anonymous Coward

    Looks like someone typed google into google. We were warned...

  57. Franklin

    Sorry, my bad.

    I did a Google search for "You reach down and flip the tortoise over on its back. Why aren't you helping?"

  58. ITS Retired
    Big Brother

    The NSA was adding another Splitter room to Google.

    Sorry about the down time.

  59. zb

    I had heard that Homer Simpson had a new job at Google

  60. Andrew Jones 2

    Kyle probably unplugged it for a few seconds, and then plugged it back in -

  61. ShortLegs

    You lot were online on a Friday night?

    You know, there is a place outside called "the real world", and it has beer.. and you know, /women/ i it (yeah, I know, its hard to believe), and pubs, and nightclubs.

    It went down on a Friday night - how come Britain noticed?

  62. bindlewurdle

    I was trying to use a couple sites that use Google's doubleclick ad network and the pages would not load. At all. I know Google rarely goes down but it seems incredibly stupid to me that Google's ad network holds its customers completely hostage and prevents them from loading at all if it can't be found.

    1. Anonymous Coward
      Anonymous Coward

      more likely bad site design, I know I would never build anything that relied on outside connections to load...

      And for that purpose part of my testing is usually kill the internet on the test server, then try the site, see how it works, ensure I have no external dependencies remaining... (sure SOME things I offload onto cloud storage, such as media etc, since their content delivery networks are better than a single server)

  63. john devoy

    Hmmm, looks like Google would really be in the shit if they started being charged for all traffic relating to them.

    1. Anonymous Coward

      what, twice?

      I think you'll find that they pay for enough bandwidth to get all of their traffic onto the internet.

  64. Anonymous Coward
    Anonymous Coward

    Microsoft Outlook online too?

    Any relation to microsoft live/outlook is down all morning already? I'm not a mobile user, but my provider is in Indonesia though.


    There's a problem with Outlook right now.

    There's a problem with Outlook right now.


    Problem A small percentage of mobile users may experience intermittent issues while syncing emails. August 18 1:36 AM

    Report a problem

    If you're experiencing a problem that isn't listed here, please report it. To see recently resolved problems, go to the History page.

    Was this information helpful?

    Tell us what you think

    1. Richard 12 Silver badge

      Nah, Outlooks down most days

      Falls over more often than a drunkard in high heels on a Saturday night.

  65. Anonymous Coward
    Anonymous Coward

    To All Who Scoffed

    They were just "re-wiring" the interwebs like what darling Davy Cameron said they were going to have to do.

  66. david 12 Silver badge

    I had something similar about 4 hours later.

    In Melbourne, AUS. AFAIK, only a couple of seconds, but it was that outage which led me to read this article.

    The rest of the internet was working ok, but Google sites were returning google error screens.

  67. Anonymous Coward
    Anonymous Coward

    I have a theory

    Perhaps the ZPM powering the Googleplex (tm) unexpectedly reached maximum entropy.


    #include "SGG_icon_here.h"

  68. proto-robbie


    ... A botnet auto-clicking on NSA job adverts.

  69. miket82


    Keyboard not found, hit F3

  70. miket82

    F3 - no keyboard found

    Sorry about that, I switched it off and then on again. Problem solved.

  71. joeyaliziojr

    Okay Who Again?

    WHO IS





  72. Lostintranslation

    Google - proudly powered by Tepco.

  73. Anonymous Coward
    Anonymous Coward

    Told you...

    It was the 'GIVE ME A BIGGER BUDGET' switch!

  74. Paul Hovnanian Silver badge

    Red pill/Blue pill

    Someone took the red pill.

  75. Anonymous Coward
    Anonymous Coward

  76. Dr Patrick J R Harkin

    I know what must have happened. And it's not like we weren't warned.

    "If you type 'Google' into Google, you /can/ break the internet."

    1. SImon Hobson Bronze badge

      Re: I know what must have happened. And it's not like we weren't warned.

      Nah, not that one, try this one instead

  77. Stu_The_Jock

    Or maybe . . . .

    . . . a Streetview car got too close to a datacentre, and while slurping, erm, looking at if there are available wifi networks, accidentally starting a fatal loop. (Much akin to setting up 3 accounts on a VAX cluster to forward e-mails in a ring) until it fell over.

  78. Arachnoid

    And the cry went out to IT

    The internets broken.........

    1. donk1

      Re: And the cry went out to IT

      ... I felt a tremor in the force!

  79. P Taylor

    We were warned !.

    Someone typed Google into Google search didn't they ?. Come on, own up !.

    Maurice did warn us about this some years ago.

  80. Antoinette Lacroix

    Wot ? Google went offline ?

    To be honest, I didn't notice. I'm one of those old fashioned persons that still stick to AltaVista. I have no G*-whatsoever account - no Gmail, no Youtube no Gdocs - no nothing. I'd rather be off- line then using Chrome. Still, my world hasn't come to an end yet. Go figure.

    1. envmod

      Re: Wot ? Google went offline ?

      lol AltaVista doesn't exist any more, stop trying to be all "alternative".

  81. tim99uk

    maybe they tried to calculate the tax bill that they are avoiding...and the shock at the amount ( and additional laughing ) caused the outage..

This topic is closed for new posts.

Other stories you might like