back to article Code crash? Russian hackers? Nope. Good ol' broken fiber cables borked Google Cloud's networking today

Fiber-optic cables linking Google Cloud servers in its us-east1 region physically broke today, slowing down or effectively cutting off connectivity with the outside world. For at least the past nine hours, and counting, netizens and applications have struggled to connect to systems and services hosted in the region, located on …

  1. Andy The Hat Silver badge

    Multiple independent fibers, single physical cable in a duct, single digger bucket ... oops!

    1. Martin Gregorie Silver badge

      The problem is often due to 'economies' made by the cable provider. Client orders dual cables in separate ducts. Cable provider's beancounters change this to dual cables in the same duct because "its the same connectivity but 50% cheaper, so PROFIT". Never mind that they've just negated the reason for requiring dual cables. Pricks.

      Or they use existing ducting because it has room in it, which can lead to very non-intuitive routing and expose a link to unexpected hazards. Example: in the early '80s I was working for the BBC in offices at Cavendish Square, programing their mainframes which were near the Goldhawk Road railway station, just west of Shepherds Bush, and had the cable cut by a digger in Acton - on the far side of Shepherd's Bush from Cavendish Square.

      1. Alister Silver badge

        The trouble is that even if, at the site, there are multiple cables in multiple duct routes, invariably they all go back to the same telephone exchange, which often has only one cable entry duct, so at some point all your multiple redundant routes end up in a single digger target...

        1. Giovani Tapini

          old problem, new implementation

          "multiple diverse routes, provided by multiple service providers" were all broken as a result of the attacks on the WTC years ago. Turns out they all used the same comms route there.

          Also raised questions about the secrecy of the physical routes used by the different providers as there was no way of actually assuring diverse routing (and may still not be, its not my thing now)

          If however the break is on Google's premises then my sympathetic stance will transform rapidly into the WTF category...

  2. John70

    Joys of Cloud

    Cloudflare and now Google Cloud...

    Coin toss on AWS or Azure being next...

  3. Confuciousmobil

    A single point of failure? WTF?

  4. Pascal Monett Silver badge

    "Mitigation work is currently underway [..]"

    It's called REPAIRS. Mitigation is what you do to get around a problem, but there is no getting around a broken fiber link. It just has to be repaired.

    Why is it that today's PR people can't just say it as it is ?

    1. Alister Silver badge

      Re: "Mitigation work is currently underway [..]"

      To be fair, in this instance Google are indeed trying to mitigate the problem, as I doubt that they are directly involved in the cable repairs, so their efforts will be in trying to route traffic round the break.

  5. Anonymous Coward
    Anonymous Coward

    Ordering diverse links

    Ah yes, reminds me of the BT Manchester tunnels fire causing unexpected outages. Ordering seemingly diverse links from a provider doesn’t guarantee they’re diverse, even if they are ordered from different providers. Once mapped to physicals, they often turn out to be in the same duct.

    The only way to know is to demand exact routing information or pay a premium for diversity and pray they actually provide it

    https://www.theguardian.com/uk/2004/mar/30/simonjeffery

  6. Jonjonz

    Local cities do the repairs

    It's a closely guarded secret, but your local municipality does all the repair work on underground cabling. That's why it takes over 24 hours to fix.

    1. RedCardinal

      Re: Local cities do the repairs

      They may do in the US? They certainly don't in the UK.

    2. Daedalus Silver badge

      Re: Local cities do the repairs

      You'll have to be a bit more specific. In which country? If in the US, in which state? If, say, NY, which county/city/township/village?

      The fragmentation of some authorities in the US is beyond what Europeans, for instance, are used to, and highly variable. Schools may be run by the municipality, or as a district under county or state authority, or as in a few cases, the entire state is one district. You may find yourself paying town taxes and school taxes to different authorities. Your trash collection may be county, municipal, or private with four or five different company trucks rumbling around on trash day. It has been known for certain religious groups to incorporate an area as a town so they can do what they want in their own little enclave.

      1. Claptrap314 Silver badge

        Re: Local cities do the repairs

        I have three different utility providers (electricity, gas, water&sewer) plus a separate trash service. My property tax is city, school, transit district, and county. (Fire is provided by city.)

        While this is the record for me personally, it is only by one.

  7. Anonymous Coward
    Anonymous Coward

    Local cities do the repairs

    It's a closely guarded fact, but your local municipality does all the repair work on underground cabling. That's why it takes over 24 hours to fix.

  8. Alan Sharkey

    It could be coincidence

    I was working for a large telecoms company some time ago, They had links from the UK to Ireland. One in Scotland, and one in Wales. One night there was a huge storm in the Irish sea. A tanker dragged its anchor over the Scotland cable and a wall fell down in Wales, severing the cable there.

    Oops!

    Alan

  9. Anonymous Coward
    Anonymous Coward

    remember London Telehouse and their duplicate power supply feeds... both severed at their single point of entry by mistimed JCB bucket

    I remember a story of a US finance company paying megabucks for 'diverse' E1 circuits (plus ISDN backup) from their main site and DR site down to Goonhilly, where they terminated in separate comms rooms to go over different sat links, with a switching arrangement between the 2 rooms. At some point there was a failure that meant that for a while the only circuit capacity to Goonhilly for the E1s was via Plymouth and as the customer was screaming blue murder someone reset the 'diversity' flag to allow both E1s to temporarily route over the same link. Unfortunately once the fault had cleared, because the 'diversity' flag had been reset, the circuits didn't bother to reroute themselves... and nobody noticed until after there had been a couple of simultaneous outages of both the main and DR circuits

  10. Daedalus Silver badge

    Everything old is new again

    If you look at the history of the 19th century monopolies, such as Standard Oil, you see parallels to Google. Standard Oil needed transportation as well as refining capacity, so the company cut deals with rail companies and eventually owned them. Before Standard came along the industry was fragmented with a lot of companies taking their profits and not doing much to make things more efficient. Standard Oil, at least in the beginning, made everything more efficient with vertical integration and use of all available resources to get oil and gasoline to the customer. Prices plunged and companies that couldn't compete went under.

    So now we have internet companies like Google and Amazon depending on telecom companies to "transport" their services. Telecoms are of course FD&H** companies who don't have the same interests and motivations. Guess where the future will find us.

    **Fat Dumb and Happy

  11. Elgreppo
    Linux

    Holy moly

    So now we know the latest secret project from Jeff of the rainforest - an army of stealth google-cable-seeking demolition moles.

    I wonder if he trained penguins for the undersea cables?

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2021