Virgin Media biz service goes TITSUP* across London A London-wide Virgin Media outage caused by a "fibre break" has left business customers across the UK capital without broadband - including a number of borough councils. Bromley Council apologised for its phones and online systems being out of action, blaming the "London wide technical issue" caused by the outage. Virgin …
We had that here, in Germany, a while back. New building site, JCB dug through the cable BUNDLE and we were without telephone and internet for 2 days, then a re-routed cable with limited bandwidth (2mbps instead of 100mbps) for a few days, until they could lay a new fibre bundle.
Are you sure that was not regular copper ? Regular fiber is much easier to splice. I remember when they were doing a BART extension they hit a bundle of 500 pair copper lines from pac bell. The contractor and BART had to pay. Pac bell handed out free cell phones till it was fixed.
Bet it was the same 16 yr old that dug up Portsmouths cable connection a few years ago.....
What we really need to do is route 33Kv underground power feeds next to the critical internet connections
Then when the moron in the JCB gets into action, we're one moron less.....
One single fibre break takes down a bunch of councils. Normal broadband at the access layer I can understand, the entire London Grid having a single fibre point of failure not so much.
So, so bad. I hope that at very least this was a case of redundancy not working as expected rather than that it wasn't built with any.
When one of VM's predecessors sold us a fibre connection, the story was that the fibre formed a ring so that a single break would not result in an outage. It quickly transpired that the ring had some rather significant single-fibre branches and that the local youths were entertaining themselves by pouring petrol into the ducts, igniting it and watching the pavements erupt...
Surely councils would have a backup link(s) with another provider that would seamlessly fail over...
That level of resilience costs money - money which councils have precious little of. The majority of people aren't familiar with IT infrastructure redundancy models and the like, but they are familiar with council services.
That leaves the council with two options...
Option The First - close a few libraries and buy some redundant connections from another ISP. If something goes wrong (and it's still an *if*) the public don't directly see any benefit. Regardless of what happens, the public *do* see a reduction in services provided by their council.
Option The Second - use meager resources to keep existing services running. Cross fingers that you don't have problems with your one connection to the Internet. If you do, try and deflect blame onto the supplier (Virgin in this case, but it could equally well have been BT - both of which the public are fimiliar with, and don;t hold particularly high regard anyway)
Today Virgin, Tomorrow BT, there's always some fibres just a sneeze away from being cut.
We're getting more and more of these frangile fibre thingies that are worming their way through soil, and today it was some borough councils, not too serious, but London City Airport are remoting their ATC via fibre - okay, they're using three redundant connections and they that say "it will be impossible that all three would fail together"
Triplicated Infrastructure terminated so unexpectedly prematurely.
Hmm.
From memory there use to be a pretty major VM hub on the outskirts of Birmingham that if it failed, took down a significant number of local councils. Don't know if that's still the case.
I have to say I've been pleasantly surprised by the VM contractors laying a new duct along my street. Not only have they not caused an outage, they've done a pretty good job of making good their work.
We used to get all sorts of crap back in my Telewest days (yes I am that old) when big trunk fibres broke - either through classic JCB strikes or something to do with exploding electric grid transformers under roads (whoi new that was a thing?).
Anyway - even when the majority of routing failed onto redundant circuits for most traffic we'd always find the phones continued to ring because of strange undocumented static route configurations. It was usually quicker for the engineers to finish resplicing 1000 fibres than it was to dig through the routing tables.
I wonder if they'll fix it faster than the 12 days it took to resolve a SNR issue on my street. The only thing that happened on schedule in that situation was the little bit on the service status website getting changed at the exact minute it should have been fixed.
"ETA for the fix is the 12th June, 10:00"
(9:59.... clock ticks over.... status site immediately updates to "ETA for the fix is the 12th June, 16:00". Repeat day after day, if it gets to 18:00 then just go to 10:00 the next day)
Hilariously, it seems my pokey little terraced house has better resilience than Bromley Council, with a 4G MiFi hanging out of one of my router USB ports providing a surprisingly good backup (the 4G upload speed on O2 was 12x what the Virgin 50Mb service provides)
Bromley Council apologised for its phones and online systems being out of action...
Doubtless numerous others experienced a similar fate.
Does anyone know the date on which Do not put all your eggs in one basket was the subject of a Cancellation Notice?
that being to save money we used a single fibre cable without any redundant backup routes.
It's easy to criticise, but as mentioned multiple times in this thread, it is quite common to have multiple redundant routes that at some point will share the same duct, whether that be near the premises or near the exchange.
For commercial developments and industrial estates (where most datacentres are situated) it is very rare for more than one duct to be provided - wayleaves cost too much to allow otherwise, and again as cables / fibres enter a telephone exchange there is commonly only one large duct space entry. Both these areas are vulnerable to roving JCBs.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
