What knocked out Brit cloud slinger Memset for the night? A busted fibre cable upstream of its data centre, apparently UK cloud hosting outfit Memset has blamed yesterday's lengthy outage on some iffy fibre infrastructure. The company acknowledged that all was not well at 13:45 BST (12:45 UTC) on 21 June with a terse note among the sea of green on its status page to the effect that it was aware of some connectivity issues at its data centre in …
Knocked out my comms to sites in Kent & Hampshire.
not sure if it was this or a different incident, it was some time ago
https://www.theregister.com/2004/03/29/bt_fire_disrupts_emergency_services/
when i had my monthly review i insisted they provide an outage statement, apparently the national ring our services in the south used traversed the pit that was alight, other services in and around Manchester remained operational throughout.
This was in the days of horrendously expensive 2Mb/s frame relay connections.
Don't ever think that comms lines follow anything that even faintly resembles the shortest route from A to B. They don't.
Back in 1980, when I was on contract at the BBC, our development team was in an office in Cavendish Square and the main BBC computer Centre was about 800m west of Shepherds Bush. We lost all contact with our mainframe for a couple of days when some berk with a digger took out the cable - at Acton!
For those who don't know London, Acton and Shepherds Bush are both west of Cavendish Square - but Acton is about twice as far away from Cavendish Square as Shepherds Bush and almost directly in line with it, so that cable not only appeared to double back on itself, but was at least three times as long as it needed to be.
Getting geographically separate routes is difficult and time consuming as there are often layers and layers of companies renting from one another.
I have two internet connections. They are theoretically distinct as each is bought from one of the three largest companies in the market. Company A's service is provided on a leased fibre from Company C who then muxes it and backhauls it on their larger fibre to the next large city. Company B theoretically has their own microwave links and separate Northbound and Southbound copper backhaul. FOUR independent ways to get my data to the Internet!
Here's the surprise: in a cost saving measure Company B has thrown away their Northbound and Southbound copper and now leases a single fibre from Company C. Yes, the very same Company C that provides my primary connection.
I know this because some joker put a spade through the fibre they both use. And no, I checked, and I'm not one of the half dozen almost identical stories also in this comment section.
' “We identified faults with two separate lines which impacted connectivity ..."'
There was a case over a decade back in Manchester, where a commercial ISP with banking customers had contracted for a primary and a backup upstream connection frrom separate providers to provide resilience. Unfortunately, when vandals dropped a wheely bin of burning petrol down a street manhole, they discovered that their two independent upstream providers had leased dark fibres that ran through the same duct.
The very nature of the pyramid of contractors renders this kind of incident likely as the right questions are hard to ask, let alone answer, and we have to remember that the internet becomes intrinsically less and less resilient as we approach its its periphery (where we all are as users). For all the hype about the wonders of "digital", the underlying infrastructure is frighteningly fragile considering the reliance we are increasingly placing on it.
This post has been deleted by its author
There was a 2 day Virgin outage last week that knocked out our fibre leased line in Birmingham. Someone went through a fibre cable on a building site in Canary Wharf. Apparently the cable wasn't ducted and no-one would let them on the site until 10am the next morning. I'd love to have seen that reported on.
Only a couple of years ago a fibre cut took out a data centre in St Asaph, somewhere between Manchester and the DC. It was a shock because there were supposed to be 2 redundant routes coming into opposite sides of the building and travelling separate paths arriving on the opposite sides of Manchester.
The DC owner (also an iomart company) found out that the fibre cables were separately routed… until the last couple of miles or so where they traversed the same farmer’s field who had decided to do some ploughing that night…
Even more shocking was they had found this out years before the incident, had asked for it to be re-routed, and believed it had been done.
https://hostinguk.net/blog/seriously/
This reminds me of an incident back in the 1990's IIRC. I had arranged our 3 main sites to have Megastream2 links in a triangle and BT sales assured me that the links were diverse. But a fire in a tunnel between Manchester and Salford was upstream of our links and they failed. We were OOS for about a week, again IIRC.
Not just us, either! Lots of businesses including banks were offline. So no cash machines, card payments..... (cue public unrest).
The situation was so serious they drafted in plods from other areas to reassure the public. I had to issue them with local guide maps (thanks to the Tourism centre) so they didn't come across as gormless when interacting with the tourists and other plebs.
Lesson learned: If your provider says the links they serve are diverse:
* Don't believe a word they say!
* Insist on them providing guarantees with significant penalties for failure!
* BT/Openreach sales lie!
aka Bernard
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
