Code crash? Russian hackers? Nope. Good ol' broken fiber cables borked Google Cloud's networking today Fiber-optic cables linking Google Cloud servers in its us-east1 region physically broke today, slowing down or effectively cutting off connectivity with the outside world. For at least the past nine hours, and counting, netizens and applications have struggled to connect to systems and services hosted in the region, located on …
The problem is often due to 'economies' made by the cable provider. Client orders dual cables in separate ducts. Cable provider's beancounters change this to dual cables in the same duct because "its the same connectivity but 50% cheaper, so PROFIT". Never mind that they've just negated the reason for requiring dual cables. Pricks.
Or they use existing ducting because it has room in it, which can lead to very non-intuitive routing and expose a link to unexpected hazards. Example: in the early '80s I was working for the BBC in offices at Cavendish Square, programing their mainframes which were near the Goldhawk Road railway station, just west of Shepherds Bush, and had the cable cut by a digger in Acton - on the far side of Shepherd's Bush from Cavendish Square.
"multiple diverse routes, provided by multiple service providers" were all broken as a result of the attacks on the WTC years ago. Turns out they all used the same comms route there.
Also raised questions about the secrecy of the physical routes used by the different providers as there was no way of actually assuring diverse routing (and may still not be, its not my thing now)
If however the break is on Google's premises then my sympathetic stance will transform rapidly into the WTF category...
Ah yes, reminds me of the BT Manchester tunnels fire causing unexpected outages. Ordering seemingly diverse links from a provider doesn’t guarantee they’re diverse, even if they are ordered from different providers. Once mapped to physicals, they often turn out to be in the same duct.
The only way to know is to demand exact routing information or pay a premium for diversity and pray they actually provide it
https://www.theguardian.com/uk/2004/mar/30/simonjeffery
You'll have to be a bit more specific. In which country? If in the US, in which state? If, say, NY, which county/city/township/village?
The fragmentation of some authorities in the US is beyond what Europeans, for instance, are used to, and highly variable. Schools may be run by the municipality, or as a district under county or state authority, or as in a few cases, the entire state is one district. You may find yourself paying town taxes and school taxes to different authorities. Your trash collection may be county, municipal, or private with four or five different company trucks rumbling around on trash day. It has been known for certain religious groups to incorporate an area as a town so they can do what they want in their own little enclave.
I was working for a large telecoms company some time ago, They had links from the UK to Ireland. One in Scotland, and one in Wales. One night there was a huge storm in the Irish sea. A tanker dragged its anchor over the Scotland cable and a wall fell down in Wales, severing the cable there.
Oops!
Alan
remember London Telehouse and their duplicate power supply feeds... both severed at their single point of entry by mistimed JCB bucket
I remember a story of a US finance company paying megabucks for 'diverse' E1 circuits (plus ISDN backup) from their main site and DR site down to Goonhilly, where they terminated in separate comms rooms to go over different sat links, with a switching arrangement between the 2 rooms. At some point there was a failure that meant that for a while the only circuit capacity to Goonhilly for the E1s was via Plymouth and as the customer was screaming blue murder someone reset the 'diversity' flag to allow both E1s to temporarily route over the same link. Unfortunately once the fault had cleared, because the 'diversity' flag had been reset, the circuits didn't bother to reroute themselves... and nobody noticed until after there had been a couple of simultaneous outages of both the main and DR circuits
If you look at the history of the 19th century monopolies, such as Standard Oil, you see parallels to Google. Standard Oil needed transportation as well as refining capacity, so the company cut deals with rail companies and eventually owned them. Before Standard came along the industry was fragmented with a lot of companies taking their profits and not doing much to make things more efficient. Standard Oil, at least in the beginning, made everything more efficient with vertical integration and use of all available resources to get oil and gasoline to the customer. Prices plunged and companies that couldn't compete went under.
So now we have internet companies like Google and Amazon depending on telecom companies to "transport" their services. Telecoms are of course FD&H** companies who don't have the same interests and motivations. Guess where the future will find us.
**Fat Dumb and Happy
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
