UK's Virgin Media celebrates the end of 2019 with a good, old fashioned TITSUP* Some Virgin Media broadband customers remain free from the shackles of connectivity today following some over-enthusiastic digging last Thursday. Problems kicked off on 19 December at 09:00 UTC as cables were accidentally ripped from the ground during construction work in the southwest London suburb of New Malden, leaving many …
To be fair to Virgin, and let me be clear, it's not something I often am:
- Virgin's business customers get more frequent updates and quicker reconnects because they're paying for a 'better' service
- having several thousand customers go through one 'cable' is not unusual.
True, they could have given more updates to their residential customers but they're always going to give better service to business customer (hopefully).The real issue is that contractors are careless (both the 'third party' and then Virgin for causing a gas leak). Just my thoughts, I think Virgin are a dreadful company who care little of their customers (business or residential).
Yeah, something like this Virgin can expensively source third party connectivity to keep their business SLAs intact. If they did that for consumer broadband they'd have to bump their prices to a level people would then complain at.
While accidentally cutting a gas line suggests process opportunities, it does feel entirely reasonable that they stopped work on the cable until that was sorted.
Nobody wants outages but the answer is clear: If you really must have always-on broadband, don't go with just one supplier, and do pay the extra to get the SLAs. Don't want that expense? Accept that sometimes things will go down.
(My second supplier is my mobile service provider; lower bandwidth than Virgin but I've yet to lose both at once, and still have power to the house)
What's the point of having two suppliers if they both use the same fibre ?
Because if you want to have two fibre lines from two different suppliers and you don't want them coming from the same direction, you're asking one supplier to lay a special cable just for you. A billionaire could undoubtedly get that level of service, I'm not sure a mere millionaire could, but I'm sure you can't.
"What's the point of having two suppliers if they both use the same fibre ?"
" you're asking one supplier to lay a special cable just for you."
In cabled areas, that's pretty much what they did - except not just for us. They put new ducting down in the surrounding streets and everything. (I remember it being done). Yes it was co-ax, but it was a vast improvement on the wet string that BT often refuses to replace.
In the UK at least, the cable companies couldn't recoup their investment, got bought out for peanuts by increasingly bigger fish and rebranded to become... Virgin Media.
As opposed to the incumbent supplier BT that used telephone poles and overhead wires for the last few tens of meters. I have no idea if that's still how ADSL happens. I mean, the poles and wires are still there...
Its worth pointing out that it doesn't matter how diverse the lines leave your building are if the entry point to the backhaul (exchange, datacentre, whatever) is otherwise constrained.
If I put a spade through a cable outside my house, I lose my internet.
Put a digger shovel through some ducting outside an exchange...
We have heard the tales on here about "multiple points of entry" that ultimately come from the same duct.
We have heard the tales on here about "multiple points of entry" that ultimately come from the same duct.
Yup. I blame sales. And language. A lot comes down to vague definitions of diversity, resilience and separarion. Openreach is usually good at that as they have defined services, ie route seperation from memory ensures at least 2m between routes. Theory being a backhoe won't take out both routes.
In practice, it's more complex to design strict separation and often requires additional civils.. Sometimes a few km of new construction.. Which isn't cheap. Some services pretty much can't provide it, ie xDSL via 2 ISPs that use the same LLU copper from Openreach.
This outage sounds unfortunate, but reasonable. So locate the cut, send out the field team who discover the gas leak. So their work is stopped until the job site is declared safe. That's something that goes with the territory, and I've had field guys stopped due to discovering suprises like buried fuel tanks, unexploded bombs, gas leaks etc etc.
At a previous company I did that, the buildings (2 of them) were designed for multiple occupancy so each had multiple ducts.
We were on a road which had a single duct running down it - however one exchange was left out the building and the other right, so we got BT to run one cable out of the left duct and turn left, and the other out of the right duct and turn right.
Mostly resilient (I will add we had two 30way fibre cassettes coming into the building) the weak point was the single comms room but if that went down we didn't really care about the external links as we failed over to the backup site, which also had the same design. The backup site was worse due to the location but I managed to get BT to minimise the cross over for the fibres to just one junction on the road which was an acceptable risk.
My office has its very own three telegraph poles - two on prem. and one in the pavement alongside. One was put up solely for fibre - real fibre as in FTTP. The other one carries 150 odd pairs of copper. I have no idea why the new pole was needed. The
The previous owners, NHS, of the building put an exchange line in per desk! They also had four ISDN2 lines. It took me two days to strip out the old phone cabling from in and outside the place. The hospital is only a few 100 metres away, surely an exchange could have been set up for this outpost. To be fair, some of the cable clips I stripped had GPO stamped on them ...
You can try and request for a diverse connection from your ISP if you are a business, BT very happily provided this for us (Although at that point we had only sent a query to our ISP on cost, we are also the same distance from 2 BT exchanges), much to the annoyance of our neighbour who couldn't take deliveries or let customers in whilst they dug up that side of the road.
Oddly still havent had our ISP bill for this yet.
We are inside the M25 in a business park, we pay a lot per month for dedicated links and are similar distance from 2 exchanges. Obviously if more rural its more likely BT/Virgin/Insert other suppliers here may refuse.
"I'm not sure a mere millionaire could"
Large (but not massive) businesses can, and while their turnover might be in the £1-10 million range, I doubt they pay quite that much for the actual service.
IIRC at my last job we were being quoted somewhere around tens of thousands of pounds to have a leased line installed. So yes, you have to be pretty flush, but potentially not a actual millionaire.
As a home based worker most of the time I got an Openreach based ISP connection as a backup to my Virgin media (consumer) connection. Put them through a dual WAN LinkSys router and I have a load balanced and fail resistant network. Some websites get funny as my IP address changes during the session, but surprisingly few.
Connections are 350/35 on Virgin and 78/19 on the FTTC one, overall cost much less than a business line from one - which wouldn’t help with outages anyway, just might mean more info and compensation.
I am mere mortal, but I have two fiber runs to my home, through different entry points and ducts. And 200Mbps service costs 5 euros each. Yeah, I live in Bucharest, Romania.
One of the supplier is the former national telco, and they had their own ducts, which they did not share. The other suppliers use city-owned ducts, which run on different paths by design.
Internet access is the only well designed thing, however. Road congestion is worse than anywhere.
Yup, worked in Bucharest for a number of months.
I was working there before Ikea was built between the airport and the city and the new motorway, during the both being built and when they were both open and especially the chaos when Ikea opened!
Also, the rebuilding of all the tram rails, which originally looked very scary to be in a tram riding over.
A lot of stuff IT related was very good.
The roads, very bad!
Not forgetting "yellow cheese" with everything.
BT have a new offering called 'halo' where they provide 4G connectivity as a contingency for prolonged loss of service (such as this) where they will courier a 4G router over to you until normal service has been resumed. It'd be unlimited datawise as well..
For once BT actually have a decent edge on Virgin... It is however an additional cost.
There are numerous routers out there that can do this automatically. You just supply your own data-enabled SIM. Some ISPs can supply you the complete package as well.
These options aren't cheap - but if you can't run your business without an internet connection, then it may be worth it.
I had both Business and Home connections from Yorkshire Cable which eventually became (part of Virgin). I paid a significant premium for the business connection. The difference when it went down was that when I called Business support I was promptly put through to someone knowledgeable, within minutes the issue was flagged on the support web page and an automated phone recording. The problem was identified and a technician dispatched to fix it.
The Yorkshire Cable domestic service on the other hand... At one time I was experiencing daytime midweek connectivity speed dropping to a crawl. Support tried to fob me off with "someone else on your street must be overloading the connection". After a bit of a battle I got an engineer to visit. He diagnosed the problem as being in the cable to the cabinet (about 600 metres he said). They replaced that at considerable cost to themselves. Fault persisted. Competent engineer turned up, changed a small barrel connector (attenuator?) on the line, problem fixed but start to finish I was on a flaky service for 3 weeks.
That said, recent experience with Virgin's domestic service seems to have good remote diagnostics and people who know how to use them.
My business relied heavily on good connectivity, without internet I'd have staff sat idle so I was happy to pay a premium. No idea if Virgin maintains the price differential and standards but surely the principle of "pay more get better" is a universal expectation. I expect @Slavo12628611 will be really angry that his BMW 1 series car isn't the identical spec as the 8 series at 3 times the price, "it's just not fair that someone paying more gets something better is it?" (Exception: Veblen goods like certain fruity consumer IT devices).
In contrast, dealing with another well known british telecommunications provider for services to the "household name" multinational I used to work for was a total nightmare. The smallest change request would require 4 weeks lead time. When there was a tech problem commonly it wouldn't get logged. It would get fixed and then a call back to say "no fault found", such a common experience that one had to infer that someone needed to massage their service availability stats.
"No idea if Virgin maintains the price differential and standards but surely the principle of "pay more get better" is a universal expectation."
Our experience of Virgin (1GB/s leased line connection) was, shall we say..... less than stellar. Delays of 2 working days from fault report to someone showing up were the norm.
This weighed heavily on our decision to stop using them.
When I was dumping Virgin after much more than few days repeat outages, their commercial 350 down/50up business service was slightly cheaper than the consumer 100mbit service! Also no chance of being constantly nagged to add TV services or phone lines. Just had no faith they could actually fix the myriad of problems significantly faster, same ancient cable, same crappy engineers, the difference was mostly just a quicker way through the frontline support crap Virgin provide.
Many years ago, they got our power line, or most of it, and we felt it for months afterwards. In the following months, we lost over 50% of our switches. After the third one in a month, someone at the vendor asked if we had seen a power cut...
31 hours[1]? Try 31 weeks and you're getting a bit closer to Virgin's ballpark.
I guess being in London gets it noticed and reported.
[1] As per the quote from Mitch Benn. Same argument applies if you treat Dec. 19th-23rd as the duration: never known Virgin to be half that quick.
Without seeing the full scope of the work it's hard to see how much of the complaints can be justified.
I mean, the photo of the auger/piling drill with all of that cabling wrapped around it to me looked quite bad.
Big difference between a "Cut" and a rotating bit of metal pulling thousands of feet of cabling out of the ground. Assuming said photo I have seen was of the actual problem.
No, it's a generic stock image (shutterstock_laying_fibre2.jpg) because El Reg think we're too thick to know what a hole digger looks like.
Apparently, as per other articles, it also helps to see what a person shouting at their phone looks like when we read artciles about customer complaints.
Articles about people frustrated by some piece of software are only clear to us when we see someone sitting at a laptop shaking his fist at the screen.
And who the hell would know what a network outage meant, without photos of spaghetti-riddled switches and routers?
Of course, when an author does use a valid photo, it leaves us thinking "Is that the mentioned grifter just before his court case, or is it file_dodgy_looking_bloke_enters_court_looking_angry.jpg"
https://i.dailymail.co.uk/1s/2019/12/22/01/22534808-7817277-A_worker_on_the_New_Malden_site_The_company_said_During_construc-a-31_1576976438834.jpg was the photo I seen getting passed around.
Not sure if it was legit. But as you can see some of what's sticking out has been cut before then it's possible that was wrapped a lot higher than what you see.
Virgin Media has an awful customer service.
I am without the Internet since last Saturday. I was browsing when it stopped. I looked out of the window and saw a Virgin Media van parked in a street and a guy walking around opening cable covers. So I thought he was doing some service and it won’t last long. Then the van left but the Internet did not appear.
I called the customer services immediately but all they could do was to make an appointment after Christmas.
It is so infuriating - their idiot engineer broke it (probably by mistake), but there is no way to contact him and make him come back and fix it. The call centre is staffed by people following a simple script who have no connection to people who do cables.
VM engineers working on someone else line also cut mine, and they then quickly disappeared before i could collar them .. it took over 3 MONTHS to get back online
fortunately i had a good neighbours BT wi-fi, which i used for that time, but without it i would have been totally cutoff
VM never accepted that it was their employee that caused the problem in the first place
so ever since that incident, i have always been very watchful anytime a VM van turns up in the street and continually check my internet link whilst they are in the area ... just in case
We have had our phone line tagged in the local box because we are registered as a priority line (we have our 8 year old and 3 year old taught in how to talk to 999 for a reason) but data line is less of a success.
I tried to explain that our disabled 8 year old can trigger my Alexa command far more reliably and be able to talk directly to me, than pick up a phone and talk to a 999 operator. Thankfully my wife's disability has not required my intervention in a few years.
We still can't get a priority on the Virgin Media data connection though.
Very wise. When our neighbour joined VM a few years back our service vanished. I called customer services who said “...that’s funny, we have an engineer next door...”! When I found him in the cabinet down the road he thought the phone cables to our house were not in use so had reused them rather using a new pair. Great documentation.
I always make a point of talking to any engineers or digging teams that I see in my street just to get an idea of what they may be about to break or turn off.
Not normally a problem but if you do need to call customer services later it helps if you can let them know that their teams were in the area recently and what they said they were doing.
I had Virgin for a year and while the TV service, when it was working, was actually quite good, I was plagued by problems:
1. The TV service on a certain set of channels was digitally corrupted (pixellated) and two fruitless interventions (many days) later by an engineer who came to the house, I eventually got through to him that he'd temporarily fixed it (he didn't believe me at first). So he then went back to the street junction box and got it working. Apparently he fixed it by swapping my port over (which was defective) with one of my neighbour's. I assume my neighbour wasn't too pleased with that temporary workaround
2. Their DNS service regularly (almost on a daily basis) went down, stopping all connectivity not related to the TV service anywhere from 30 minutes to 4 hours
I've switched to NowTV and don't get any of these issues now and the service is £15 cheaper/month
While I appreciate that it happens very, very rarely to Virgin (and it apparently wasn't their fault anyway), in this day and age I would like to see something much more pro-active, like providing a basic, backup connection service via their mobile network, even to get the e-mail working.
That said, why would they bother, when for most people these days, the core internet service comes through their mobile (and not on wifi either).
p.s. yet, it's staggering how much actual work I was able to do in those 48 offline hours! And I don't even mix up with social network suspects, or stream anything. Kill de internet, I say!
...
BUT NOT BEFORE I POST THIS!
and check this... and that, and...
I am happy that you had mentioned the increase in productivity with the internet down because the other day I had one of those dangerous stray thoughts: How much time is spent in this place reading the comments and having ethereal experiences in lieu of the tasks before me? And what of the rest of the Register's thundering herd?
I can almost forgive striking a basic fibre optic cable, but I fail to understand why you would strike a major trunk line, these things are typically well mapped out.
But following things up by striking a gas pipe?? I hope they can convince the HSE they even switched on their Cable Avoidance Tools.
CAT make fine Cable Avoidance Tools. Available in a range of bucket sizes with teeth to yoink up a cable so you know to avoid it in future.
Well mapped can also be a matter of interpretation, or just availability. So one being how well any survey data gets logged, the other knowing how to find any online portal or contact. Sometimes GIS doesn't give much accuracy beyond a rough idea which side of a road infrastructure might be on. I've also used GIS that have Google maps as a front-end, and although the survey data is accurate, Google's mapping data isn't.
But civils contractors should also use lo-tech approaches, ie walking the planned excavation and looking for suspicious lines of tarmac, or duct covers. But that can be fun, ie I have Torch chambers near me, and can't remember who their fibre ended up with. Then there's the proper methods, like using detectors & spray chalk to indicated buried stuff, and then care when digging. Fibre routes should have metallic locator wires so they can be found, and plastic tape above the route warning of buried cables.
But that doesn't always stop a merry digger. VM (and other new entrants) also have a challenge given utilities were/are assigned layers or burial depths to deconflict utilities. Telco being newest, it's usually shallowest so it takes the hit first.
"CAT make fine Cable Avoidance Tools. Available in a range of bucket sizes with teeth to yoink up a cable so you know to avoid it in future."
What surprised me was that the teams using CAT/genny tools on our site didn't know how to use 45 degree triangulation to work out the _EXACT_ depth of the cable (rather than reading the rather inexact measurements the meter gives), or how to back off the squelch to locate the exact horizontal position of the cable vs "it's somewhere in this 3 foot wide swathe"
I suspect my 5 minute tutorial in doing this saved us a lot of trouble.
Just because someone has the tools, doesn't mean they actually know how to use them.
Yup. I like these-
https://leica-geosystems.com/products/detection-systems/cable-locators/dd-smart-utility-locators
I think training is certainly an issue, but so can be telco civils. So a lot of CATs use a transmitter that needs to be clamped to utility for best accuracy, but for fibre and trunk runs, access to locator wires is harder. But that's the challenge with cheaper EML vs GPR systems.
Or civils gangs just don't bother. One morning I was waiting for the bus and watching a VM crew lifting pavement slabs and starting to dig with a shovel. Cue flashbang and street lights going out. Luckily the digger didn't get electrocuted, just a face full of spatter from the arc. Crew didn't have eyewash in their van, but luckily I had a bottle of water on me to do that while waiting for the ambulance.
They can be expensive mistakes though as people have died hitting buried power cables, or just injured, and then there's the invoice from the utilities for damage & restoration.
This post has been deleted by its author
This post has been deleted by its author
It was a lot more than just VirginMedia... I have a multi-WAN router with 500Mbps Virginmedia, and two backups: BT Openreach and a Vodafone 4G dongle.... all three were down... even the dongle would connec to LTE, but be unable to transfer data... then reconnect with such a low signal, I could ping IP addresses, but nothing more. Both the voice lines linked to the cabled solutions were also down, so the only comms we had was Vodafone (voice and SMS only) - I resorted to putting a SIM card in a tablet and taped it to an upstairs window - I was able to get ~200kbps on a 4G link, presumably from a fairly distant, overloaded mast... #firstworldproblems
Various neighbours on BT fibre and VM were all impacted - the only people unaffected were a retired couple around the corner on Talk-Talk!
It's worth noting that it NOT ENTIRELY FIXED YET - after 8 days, I have a working BT phone line, but still no DSL.... and then working Virginmedia broadband, but no VM phoneline... presumably they're not keen for me to talk to them....
The internet might be able to re-route things when something happens.
Firstly this was not just a "cut".
It is not until you get deep inside a ISP's network that the multi honed benefits can take over.
From you to the Street Cab is one connection. From there it's a single connection back to some form of headend and it's possible that more than one headend is connected in a ring allowing some redundancy. Once you reach the core network from the headend then there should be multiple points of redundancy.
The core network will have multiple peering providers in multiple peering locations. Really small ISP's might only have one provider.
The fact that the post above yours shows that multiple ISP's were down including mobile networks (They probably use Fibre backhaul instead of Microwave) shows just how much was "cut".
A few years ago at this time of year I had a problem with my Virgin Media landline. After they said they couldn't fix it for two weeks I emailed the chief exec of Virgin about the problem explaining that the phone was essential for receiving calls at Christmas. Amazingly I had a reply from his PA saying they were going to get it fixed post haste. An engineer turned up within 48 hours and said they would have to pull through a new cable to my house. This made no difference to the problem so after about two more visits they eventually sent someone more senior who changed an amplifier in the green box in the street. I'm guessing this was more expensive than the cable so they tried a cheaper fix first and only the more senior eng had the authority to change the amplifier.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
