Parts of UK booted offline as Virgin Media suffers massive broadband outage Liberty-owned UK broadband pusher Virgin Media has fallen on its face this morning, with users across the country reporting complete broadband failure for a number of hours, among them some of the reporters on this news desk. Earlier today, even its status page was unavailable, although prior to this readers who alerted The …
One of two possibilities, beyond not paying their internet provider bills....
1. They had decided to try and send all their internet traffic through a satellite launched by Virgin Orbit. Oh, wait. That failed to launch didn't it.
2. Someone put the wrong bit of Virgin into receivership and accidentally tanked Virgin Media instead of Virgin (not in) Orbit.
A notspot networker at wavemobile raged on Twitter that they'd had a "sleepless night thanks to @virginmedia and ZERO information or apology..."
If you're in a life or death occupation, then you should have more than one 'net connection at home so this should not have affected your work. If you're not on life-or-death duties, then get a life and put things into perspective.
Excuse my ignorance on these matters, but as I understand it, your 4g equipment connects to the nearest cellular tower, and the cellular towers are connected to switching centres via dedicated cables (or in some cases microwave links, where cabling would be difficult). Your assertion that your 4g connection goes via a street cabinet outside your house that is shared by your broadband service would imply that you understand somewhat differently. Can you expand on the role of this street cabinet in your 4g service?
Also the virgin media broadband uses a different street box to the BT/EE/plusnet cabinet.
Virgin media stopped using the BT phone system a few years back, they sent out a gizmo that enabled the phone to be plugged into the back of the virgin hub.
The only reason the 4G would be using the BT street cabinet is because the user has a femtocell which piggybacks on the broadband.
O2 only very recently became owned by Virgin Media, and Virgin Media don't have cable service in my area.
The cell tower is on top of a big pole that comes out of the top of the street cabinet. Any environmental factors that affect my BT line are going to affect my O2 line as well.
Many years ago, O2 was spun out as a separate company from BT, previously they were known as BT Cellnet.
Orange, now one half of EE, now owned by BT was at one point owned by NTL who were one half of Virgin Media.
If you trace the corporate musical chairs of all the players in the market, you will find they are all linked to each other at one point or another.
Whomever runs the underlying infrastructure then...you know what I meant.
I've since done some testing with another network engineer mate of mine...we pretty much confirmed that there are points where traffic is concentrated into smaller areas or bottlenecks.
He is in the Midlands, I am down South...we did a bunch of traceroute tests to various IP addresses around the country for various clients of ours from various different ISPs in our respective areas (mobile and fixed line) and there were common hops all the way along...admittedly at different points in the route, but they were there nonetheless. No matter which ISP we tested with, after bouncing around a load of internal routing kit, we would ultimately end up on identical backbones...the exit from the backbones would differ, but that is probably some sort of load managing mechanism...i.e. Virgin goes this way, BT goes that way etc etc.
The most common "hub" point seemed to be somewhere around London Heathrow, but we also noted commons "hubs" in Liverpool, Manchester and Bristol as well. We didn't find a single one in the North East, even when routing to the North East.
"Whomever runs the underlying infrastructure then...you know what I meant." That would be.. Openreach for over Openreach connections, Virgin Media for Virgin connections, KCom for Kcom connections, etc... You know, the individual companies. None of which are the local authorities.
As for the rest, congratulations, you discovered ISP peering. And just because your packets went that route today, doesn't mean it will go that route tomorrow if something happens.
I used to have fun with that, when Demon was around, because packets to the US could go via LINX, or via Demon's own link to NYC and come out from the NYC office.
Sure, the last stretch might be individual conduits, on your street, but they meet somewhere...otherwise roads would look like patchwork quilts. Quite a lot of network infrastructure meets at railways and follows railway lines, or meets at sewers etc.
Also, if you live in an area where the houses were built before the internet, there's a solid chance there are either no conduits or just one that was put down in the 90s. The road surfaces and pavements round here are insanely thick concrete.
My house and those around were all built in the early 40s and my Virgin line comes out of a common conduit, that was once local authority but now falls under the remit of the local water authority.
When I had my dropped kerb done for my driveway, the guys that came and did the work had to get through 2 feet of concrete. It took them two days, the concrete was that hard after decades of sitting there. For ISPs to run their own conduits round here would require enormous expense and huge amounts of time...which is why they use common conduits that have existed forever and are still used for other things, like water or sewage.
It's similar in very built up areas...in Vauxhall, London, if you have a leased line, it all meets in a common conduit regardless of who your ISP is. I don't know specifically where, but there have been cases over the years where some workmen have gone through the conduit in a couple of places and knocked all the internet out for everyone in a small area around there. It's happened to my client there twice in 10 years...also around the Monument area, Central London. Most of the conduits round there are BT...because of this, it's actually really difficult around there to get a provider that isn't BT if you're a business below a certain size, the price sky rockets for connections over a certain speed and of a certain type round there. I know a few businesses round there still stuck on ADSL because they either can't get a leased line or the price is insane.
On top of this, way back I used to work with an MSP that partnered with Spitfire in London to provide internet connections, which meant I got to work quite a lot with a load of comms engineers they would regularly have problems installing lines because of messed up conduit sharing etc...they'd reach a certain point and there would be a stretch of conduit that they couldn't run a cable through because they had to wait on someone from a different organisation to show up...that would halt the installation until that stretch could be traversed to get them back to conduits they could freely use.
Lot of customers would rag on Spitfire for being shit, I'm not going to say whether they are or not, oh alright, yeah they suck, but it isn't always entirely their fault. It certainly isn't the fault of the engineers!
Eh? When did local authorities start running IXPs and backhaul for ISPs and telcos?
IIUC Hull Council used to have its own telco until it got sold off ~20 years ago.
Local authorities may very well scrimp and save. It's unlikely they do that for backbone telecommunications since that's rarely a service offered by a local authority.
They don't offer the service, but the cables have to run through and past their infrastructure at key points to save on constantly digging up and patching roads etc not to mention making the installation a lot easier.
When BT converted the area I lived in yonks ago from telephone poles to underground lines, they didn't dig loads of new trenches to run new conduits, they used existing paths and ran heavy duty, watertight, pipes through it then ran the cables through the pipes.
You can usually tell which existing infrastructure your Internet is running through by looking for an inspection cover near your local distribution points. It'll be gas, water, sewage or electrical. Most common round here is sewers. I haven't seen a DP near a gas conduit myself, but I wouldn't rule it out as a route for fibre.
Some common meeting points for all this infrastructure are canals, railway lines, water pumping stations etc etc.
All on assured services through VIRGIN MEDIA. I think it was a bit harsh to judge the decision to check on our business when it went lights out at 2:30am. It wasn't a rage either, it was a statement of fact. VM are useless at assured customer management - they got rid of their customer facing support team and are still reporting faults to us on circuits that were deactivated three years ago, whilst not reporting faults on the circuits we actually use. This is a wake up call to the fact that we need a totally independent provider however, I'll give you that.
Lesson learnt years back: don’t expect to get any meaningful response from a service provider within the first hour or so of a major outage.
So if you are needing to monitor the office, then ensure you have multiple independent paths for such alert transmissions.
Obviously, at 3am without any form of communication you have to make a decision - do I investigate now or ignore it and simply arrive at the office early…
Whilst living in a village does limit your choices, it can be done, it’s the reason after nearly 15 years there is still a Three high gain 3G antenna on the roof pointing to a specific place on the horizon; yes a 2MB 3G connection isn’t great by todays standards, but it is my third line (line 1: Fibre, line 2: EE 4G, line 4: the in-laws in a village the other side of the valley attached to VM) it got used last year for 5 days when we had a cable theft as both 4G and residential broadband shared the same BT duct out of the village.
I've got 4G via EE. Did manage 70+Mb/s, now lucky to get 8 Mb/s. Cant really understand why as in a scattered rural community. Every fault seems to result in a 3 to 5 hr support call which gets a new SIM and a random large charge.
> I've got 4G via EE. Did manage 70+Mb/s, now lucky to get 8 Mb/s
Worth checking the age of your router and lookup the frequency bands its radio modem supports, then compare with the frequency bands etc. actually available in your area. You may find EE are limiting speeds on “core” frequency bands.
The other change I’ve seen is in upgrading the masts things have moved resulting in signal level changes. So worth doing a little surveying. A friends caravan got really good signal once we mounted the 4G Mifi on a pole six feet above the roof, only possible because of a new mast in a more favourable location.
We've got several work sites like that - and although not massive they still have too many people for 4G to be really viable so I had to get leased lines installed (a fraught process, as anyone who has ever had to discuss ducts and poles with Openreach will know!). In one case it was literally the first building over the boundary of phone exchange territories, meaning it connects to the small exchange in a village 3 miles away rather than the large on in the large town just down the hill! No flexibility with consumer grade connections of course, but they did agree with my suggestion that rather than their initial plan to run the leased line for three miles on poles, it made more sense to run it the other way (a duct then polies, about half a mile) to the nearest cabinet where it could be connected. Ended up taking about six months after they repeatedly claimed to have installed the one new pole needed and I kept telling them they hadn't. Then it mysteriously appeared one wekend about two feet away from the stake with the large laminated notice saying 'place pole here' with a big downward-pointing arrow. I subsequently won the argument with the cable installers about who was going to extend the duct the two feet from where the pole was supposed to go to where it actually was!
Oh - and for the benefit of the other El Reg readers, this is what I posted during my rage: "A sleepless night thanks to @virginmedia and ZERO information or apology. Things go wrong, the least you can do is tell us via a reliable service page. Would have saved me a trip to the office at 3am."
You can't have full IP/VRRP failover with another provider - The internet doesn't work that way.
They had a leased line from the sounds of it, which has actual SLA's attached to it. Even with VRRP failover via a completely different circuit with the same carrier (Copper-based SoGEA or L2TP/IPSEC tunnel over LTE), if the entire AS number of the goddamn ISP disappears of the internet isn't going to help. Sure you can hook up some random 'backup' circuit for basic emergency internet access, but it's not the same as your actual IP range failing over. (Which again, if the entire AS went dark, still couldn't have failed over to another provider).
Redundancy does help.
A contractor managed to sever the landline connection to our house with an excavator earlier today. Not really his fault - he'd asked me where the service cables were and I forgot to tell him about that one. First I noticed was the router under my desk started blinking like crazy. My work PC carried on ticking, as it was on the 4G network (better speed than the 2Mbps copper I get).
Or you could just buy an LTE-failover router.
Then it works with any networks, any number of networks, any SIM, on any package, with any broadband, at any time.
I have a Draytek model with two 4G USB sticks, and Smarty SIM (that I can change to unlimited data from the cheapest package any time I like).
The Draytek's followed me through four houses, three ISPs, and means I never have to renumber or reconfigure my devices on my network.
Currently plugged into VDSL2 with Vodafone, but equally happy to make it failover to Ethernet / USB any time I like. Was seriously looking at Starlink for this (probably as the main connection with DSL or USB failover) as I'm rural, but to be honest, VDSL2 and 4G are fine where I am, which I wasn't expecting.
Article and DrayTek’s track record doesn’t exactly warrant the “pretty brave” accolade.
Yes it is worrying the attack vector isn’t known and that DrayTek has EOL lifed these specific (SME) “enterprise” models (I have clients with these models). However, from the information Lee gives, they are unlikely to be the model they are using.
However, since Lee gives the impression this is an “old” DrayTek router, they may see both a performance and functionality improvement with a current model; but then I’m using a couple of out of support DrayTek routers….
Show me a product that doesn't have flaws.
And at least the older Drayteks are still getting updates - thanks to this article I was alerted to look, and downloaded the latest firmware for the three Drayteks I run. Given that they were replaced with newer models nearly six years ago, which have themselves been replaced with even newer models (I reckon the oldest one I run is 11 or 12 years old) the fact that they are still getting firmware updates is pretty good, though every time I update them I have to deal with the self-signed certificates.
I inherited some Cisco switches when I started a job in 2008, they were about four years old at the time. There was one IOS update for them and nothing after that. Performing that update was pretty scary, given that I'd never used IOS before; finding out that there was only enough flash for one copy of the OS such that updating meant overwriting the "working" copy was one of those deep-breath moments because if something had gone wrong I would have been left with a pretty blue brick bolted in my rack. Ugh. The Draytek routers (and indeed other switches I eventually bought to replace the Ciscos) are a doddle in comparison.
M.
I'd definitely trust a Draktek more than the crap routers which most ISPs supply. Especially Virgin - we have a number of staff who work parly from home and have Virgin Media connections and their routers are the worst of the lot. The latest are completely unconfigurable in any useful way - you can't even change from the 192.168.0.x range for the local network - literally the only bit of the LAN setup which can be altered is the starting address (within the 192.168.0.x range) of the DHCP scope. Putting them into modem-only mode and attaching a separate router is beyond what most users could do.
Anyone who did get through to an agent this morning was told "no, no problems, everything is fine here, it must be a problem with your house or localised sunspots. have you tried changing those hideous curtains?"
Spoke to 3 users this morning and PRTG shows them all losing connections during the early hours, getting it back briefly and dropping off around 06:45.
Still, if they don't admit it, nothing happened, right?
This.... and I generally am happy with my VM fibre and mobile... is their big drawback.It seems to be a company policy to keep their own frontline staff, and the customers, in the dark when things go wrong. You can have a dead connection, phone them up and they'll make a big play about testing the line..must be your equipment, we'll send an engineer ( to be fair free of charge). And then if you're lucky it will suddenly get better. And if you aren't you'll eventually get told, possibly by said technician, that there's a major outage in your area. If you are very lucky you'll get a support agent answering your call who'll say "Funny, I've had a lot of calls like yours in the last hour or so, I'll ask a manager". Goes away, comes back and says "Apparently the whole area is out. They just hadn't told us".
And yes I've been that lucky, once.
Given how usually rock-solid our VM connection is, outages of any sort, let alone a one like this lasting several hours, genuinely are noticeable to us, and the comments that were being made on the local neighbourhood groups suggests others in our area were similarly taken by surprise to find their connection dropping like this.
There may well be some parts of the country where VM genuinely is bad often enough for this to feel like business as usual (I've heard it suggested a few times in the past that it might come down to whether your area used to be served by Telewest or NTL prior to the merge/rename into VM), but please don't just fall into the tired old trap of assuming this is the case for all VM customers.
Same here, though I now have a City Fibre connection. Had the VM* cable connection for 20 years with excellent reliability. Others in same city on the ADSL option were crap.
[*] originally Blueyonder, then Telewest, before VM (and then without name change Liberty)
I loved Blueyonder, was quite happy with Telewest, and tolerated VM until we moved house to an area not served by them. That broke the chain, and now we're back in an area "served" by VM, there's no chance I'm going back based on my mum's experiences and what I read here.
In general if you are in an ex-Blueyonder area then its pretty reliable as they built things properly.
If you're in an ex-NTL area then its a total bag of shite - cables tacked to wooden fences and left loose on public pavements outside terraced houses is the norm. There's a cabinet where we used to live which hasn't had a door on it for 9 YEARS - and yes its in use.
Virgin Media "customer service" is non-existant unless you want to upgrade, took me 5 hours and 13 phone calls to cancel last year. I finally worked out that if you selected cancellation on the phone menu it never got answered so the best option was to let the phone menu time out and then someone eventually answers. Admittedly its some twat in an Indian call centre who deals with multiple companies customers & couldn't give a shit what you want but that's the way of things with Virgin Media.
Was Cable London, who got taken over, by either Cable and Wireless or Ntl then got taken over again by either NTL or Cable and Wireless (IOW I lost track of which way round it went). Then again by VM. Who are actually Global ( maybe always were or maybe they got taken over too, who knows with this stuff?)
"Strange. Making that selection is supposed to immediately take you to the customer retention department."
I suspect the reason for that may have been I was on a 30 day rolling contract (and always had been), therefore no possibility of retention so they simply didn't bother answering the call.
Doesn't change the fact it took 5 hours on the phone to cancel.....
I use Virgin Media. I also use Virgin Mobile. Previously I've simply switched to tethering from my mobile when an outage has happened. Today I could not. I had no mobile data coverage and very "selective" internet access through broadband. It was difficult to find out what had happened because status.virginmedia.com was inaccessible, it hadn't hit the BBC website and twitter would not load due to the broadband issue. I have no idea if the two services were impacted by the same cause...
Fortunately both perked up by 9am, but the wide-spread loss of access was concerning. I was thinking I'd have to get virgin to fund a day off work...
It was more than just Virgin Media, Downdetector displayed numerous others including BT, TalkTalk, Vodafone and other non ISP platforms all showing the same trend on the graphs. I have a VM FTTP, a Voicehost FTTP (BT Openreach) and a Voicehost FTTC - all these went down at the same time, given they are different technologies, routes and platforms (VM/OR) I'd say there was something bigger happening. Service came back on, then off a few times, all providers seeming to recover simultaneously & then drop simultaneously. My gut feeling is DNS might have been at work or perhaps not working as it should! Thank goodness for 4G!
Yup, noticed those spikes too, though I wonder if that was as much down to those networks responding to increased loading as anyone with the ability to switch to a different provider did so to try and maintain connectivity having lost their primary link via VM, as to it being something higher up the food chain that affected them all to a greater or lesser extent.
From the Cloudflare link..
https://radar.cloudflare.com/as5089?range=1d
And a quick pre-caffeinated eyeball of the routing advertisments.. disruptions to other UK ISPs makes sense. So it went from announcing a normal 625 routes to 616k, and ultimately 802k at around 0800. So looks like another AS7007 event, and.. oops. Curious why Cloudflare says this-
No route leaks originated by or affecting this AS detected
At the last CIDR report, a full routing table was 938852 routes, so it looks like Virgin leaked pretty much everything. Unless they managed to do some pretty spectaculary inverse route aggregation. So not suprising there were wider issues with peers and transit customers. Sometimes adding redist all-the-things isn't a good idea. Once properly caffeinated, will have a look at the links in the artical, because if they caught the actual announcements, the problem should be fairly obvious.
Leaking referrs to routes being advertised when they should not be. This is a bad thing. This causes traffic to take an incorrect route towards destination (shifting congestion points) or a wrong turn and never getting to its destination (blackhole).
In this case that statement is confirming that the cause of the disruption is not bad BGP configuration that has been accepted by BGP neighbours concerning leaked prefixes.
Which leaves the cause squarely with the management team at VM for simply dropping off the Internet via such things as equipment failure or incorrect configuration of BGP with regards to their own AS and network.
This really bothers me. Massive service issues, key systems broken, nay the squeak of a church mouse on the status page acknowledging the issue. Customers left in the pitch black not knowing what's going on.
Meanwhile, a status page has something utterly trivial from 8 days ago that was resolved in about five minutes.
Seen this far too often with too many companies and service provider, also happened with Google's AppEngine domain a week ago - was down for 18 hours, couldn't authenticate with a webapp we run, Google said nothing. Things like this are absolutely maddening.
I spent yesterday sat on a Whatsapp[*] chat with "Maria" [**] at Virgin Media, since Virgin wanted to charge me more at the same time as Hyperoptic is offering to charge me less for a faster service.
Said chat started at 15:40, and didn't finish until 19:00, during which time Maria:
* Tried to offer me a new deal which would cost more than the proposed increase in charges
* Tried to offer me a slightly cheaper new deal which would still cost more than double Hyperoptic's offering
* Threw some FUD into the chat about "welcome deals" from other providers who might then put their prices up[***]
* Ignored the screenshot I sent them of Hyperoptic's significantly cheaper 24-month deal with a "no price increase" guarantee
* Tried to offer yet another deal which was still 50% more expensive than Hyperoptic
* Threw some more FUD into the chat about competitors increasing prices
* Tried yet again to "review your account and see where we can improve your value for money with a package built for you"
And at every step in this process, I simply repeated "No thank you, please cancel my contract as requested", until - after over three hours of this laborious process - I finally told them to stop wasting my time and JFDI.
Then, this morning, I woke up to a major service outage.
And just to add a bit of icing to the cake, I then got a phonecall from a random 0800 number, which is apparently linked to an "aggressive" virgin telemarketer process.
Needless to say, any regrets or concerns that I may have had about leaving Virgin have very much been washed away now...
[*] I had to spend over an hour being bounced between "helpdesks", the last time they tried to triple their charges. Online chat seemed like a better option!
[**] English was clearly not their first language, which leaves me to wonder if that was just their work name...
[***] The irony of Virgin flagging this as a risk was not lost on me, given this was their second attempt to increase my package costs after the welcome deal expired!
"English was clearly not their first language, which leaves me to wonder if that was just their work name..."
The first two nationalities that spring to mind when I hear the name "Maria" are Austrian and Russian, so maybe broaden your horizons a little?
You'll also find that this may not specifically be their name, but just one given to them by the call centre to use, which is exactly what the OP pondered.
Case in point are many of the scammer videos that have people in Bengaluru and elsewhere using westernised names (despite being clearly Indian by their vocal accents) whilst attempting to scam people out of Target and Google vouchers/giftcards and the like.
So, the name per se does not indicate *anything* about their nationality. Oh, and those might be *your* first two nationalities that spring to mind. Mine are Italian and Latin American (any of the Latin American countries). See how that's not an exact science?
"You'll also find that this may not specifically be their name, but just one given to them by the call centre to use, which is exactly what the OP pondered."
Pondered only due to it sounding like English wasn't her native language.
"So, the name per se does not indicate *anything* about their nationality."
Err, yes, that rather was my point, wasn't it...
> The first two nationalities that spring to mind when I hear the name "Maria" are Austrian and Russian, so maybe broaden your horizons a little?
And both of those countries are obviously famed for their English call-centre suppliers!
Beyond that, and from what I can see from a very quick rummage online (and as of 2020, at least), Virgin's call centres are in the UK, India and the Philippines. And given how strongly roman-catholic the latter is, it could well be that Maria harks from that part of the world.
In any case, I don't have any particular feelings towards Maria; they're clearly stuck with a highly aggressive customer-retention script, and had to keep plugging away at it until they'd either exhausted all options or I hit the Big Red Button Of Potential Escalation.
My frustrations mainly stem from the fact that not only did each step in this process involve a 15-20 minute round-trip delay, but everything I said was completely ignored by Maria's script in favour of making an overpriced offer and/or attempting to throw FUD at me.
And I can't help but wonder if an English national would have perhaps been better able to recognise the futility of repeatedly offering me an overpriced deal, by the third time I'd uttered the magic words "No, thank you. Please cancel my contract as requested"...
"Said chat started at 15:40, and didn't finish until 19:00"
There's your problem right there. Don't allow them anything like this amount of time. Either they can beat it or they can't. Once they've failed to beat it, tell them your contract is cancelled and put the phone down. Once you've sent an email informing them of their failure to cancel according to contract you're free to cancel your direct debit. Screw giving them any more than 10 minutes of your time!
> Don't allow them anything like this amount of time
It's worth bearing in mind that this was on Whatsapp rather than a voice call; after the previous set of shenannigans around ringing their call centre, I foolishly hoped that an online chat would reduce the amount of retention-spamming, as well as giving me a permanent record of the conversation.
Then too, Virgin are a bit wierd; the equipment they give you is "rented" to you, and has to be returned when the contract ends[*]. So I wanted to be absolutely certain that the return process had been engaged1
[*] Presumably for refurbishment and reuse. Which may explain why the first Tivo box I received died after less than a week...
Some tears back I had a monster run in with BT. They had increased prices mid contract and I exercised my lawful right to leave. After some weeks of getting letters telling they were going to charge me for the rest of the contract (illegal) and the router (not sure), and then told to ignore them, I escalated to their complaints team who quickly sorted that out. I still have the BT router. Vodafone got my business.
and much more than inflation - 16% profit given directly to the shareholders. FFS our system is broken if a provider that is actually this bad can still make so much money squeezing it's users. Telecomunications is now a cartel, there is no real price war it seems they put the prices up to match what the others do and the customer loses.
sorry about my 'metoo', but every time I see a virgin fuckup, I'm soooooo glad I managed to leave, even though the process of leaving itsself was MUCH. MORE. TRAUMATIC. than what I expected it to be. And this with my VERY low expectations in general, across the board. Virgin beat my low expectations with ease, it was a (...) nightmare.
Funnily, enough, now I have an option to sign up to a direct competitor (who have the cheek to call themselves 'community fibre' or such), for less than half price of what Virgin would want to charge me, but I have no use for this, even with 2 teens in the house a regular adsl is just fine. And the other day I got a call which my phone flagged as 'potential scam', or such, which I did accept, just out of sheer curiosity of what the scam would be about. After some clicks and grunts, somewhere along the 1st or 2nd sentence, a word 'virgin' appeared, which satisfied my curiosity, I told them politely to have a nice da and blocked the number. But it did give me a shot of satisfaction, I see it as poetic justice that they got to such a point with cutting costs left right and centre that they cut the cost of selling their core service. Plus their phone numbers are automatically pre-flagged as 'spam', I don't know at which point this blacklisting happens (I'm with 1p who are piggybacking on ee), but I think it's a brilliant idea :)
...I've got them to dump these use incompetent muppets.
If the TV AND Internet drops EXACTLY Levery 5 minutes, rebooting the TV box will not work.
If an engineer says the equipment is fine , rebooting the router for the 200th time will not fix the router.
If the 2nd engineer says its not the cales, sending a 3rd engineer to replace the cables will not fix the issue.
Sending a 4th engineer to replace the card in the street box "because it's a hot day" doesn't fix the issue, especially when it happens on cooler days.
Oh did I mention they cut you off on your first attempt to call every fucking time?
Oh we can only speak to your deaf, non technical 79 year old account holder.
Every time at LEAST 30 minutes to get through.
From day one I told them it was a clocking / sync issue, what having 22 years in networking and telecoms experience, but hey, gotta stick to the fucking script.
The moved to Sky TV ( they can't get a signal via an Ariel) and had an issue on set up. What a change, got through after 2 minutes, guy was happy to talk to me, not my parents. Skipped several steps as I'd said I'd already tried them. Up and running in 5 minutes. Not only that, set up all the configs they wanted and they allow anyone to call and report a fault.
The sooner VM due, the better.
> Cloudflare noted that VM's authoritative nameservers are also hosted on AS5089, which explains the nameserver/ website outage.
All of them? Idiots. DNS 101: Always have at least one nameserver on a different network, ASN if possible.
No it wouldn't have helped the connectivity issues to whatever DNS was point at, but it would have allowed them to at least drop the TTL and repoint their website to a status page or something.
Ever since Virgin swallowed NTL and Telewest all those years ago growth on the network has all but ground to a halt.
I was a telewest customer, just before the merger I moved house. Telewest didn't serve my new address so I signed up with another ISP. Telewest however promised that they were looking to expand to my new address within six months to a year. In that time Virgin took over. Within a few weeks that changed to a promise that they were planning to cover my area, but they weren't sure when. A few weeks further on they were "investigating the viability" of covering my area. Unsurprisingly within a few weeks I called again and was told somewhat tersely that Virgin didn't cover my address and had no plans to cover that address. That they had no plans to cover my address was a bit perplexing. Here was a company that said they wanted to rival Openreach who were openly stating they had no intention of expanding in order to compete with Openreach.
Since then I've heard all sorts of horror stories about them failing to expand. Just down the road from where I'm sitting now is a large housing development. When it was built the developer spoke to Virgin about the possibility of laying in fibre as high speed broadband is always a major selling point especially if other nearby developments don't have it. Virgin seemed keen so the developer laid in a load of ducting for them along with those silly little triangular access covers they used to fit. Then Virgin decided they couldn't be arsed to connect the estate to their network. Their rather bizarre response to the developer was that there wasn't enough demand. The developer responded that they hadn't actually build most of the houses yet and as such Virgin wouldn't get any queries from potential customers until the houses were actually sold. Then when people started moving in they called Virgin only to be told "we don't serve your postcode".
I really can't see anything changing. Virgin have missed their oportunity. The reach of Openreach fibre is massive now compared to what it was only five years ago and it's expanding at a rapid rate. The time for Virgin to expand their home fibre network to new areas was a few years ago when Openreach were dragging their feet.
Take my home town. Every property that is fed from a pole now has full fibre available. Only properties fed underground are left on FTTC. I'm told by Openreach engineers that this is because the ducting for the properties fed underground is shocking. In some cases the cables are a few inches below the pavement with no actual ducting. As such putting in fibre to these streets would mean major civils. Basically digging up the road and laying new ducts. This simply isn't happening within the timescales imposed by Ofcom. As such these streets are staying on FTTC for the forseeable future. Those few streets will not offer Virgin enough revenue to make expanding here profitable. But why would anybody who already has fibre from Openreach (or rather an Openreach reseller) be at all interested in Virgin? As such there will be close to zero interest from all the streets already served by Openreach.
Move other towns nearby and you'll find that not only are Openreach rolling out full fibre, but there are other vendors in the market too. So that's an even more congested market that will make interest in Virgin services even lower. So then they won't be able to justify the investment.
And all this before you consider that 5G services to homes and businesses are becoming more and more attractive. My provider will do me an extra unlimited SIM for 13 quid a month. If I had a better signal I'd go for a 5G router. At the moment I can get a decent signal upstairs but not reliably. All it will take is a new mast and I'll be all over that and dropping the fixed line like a shot.
As such while Virgin may have an ambition to compete with Openreach as a wholesale fibre provider I don't think they have the momentum to achieve that.
As such while Virgin may have an ambition to compete with Openreach as a wholesale fibre provider I don't think they have the momentum to achieve that.
The problem was more with the management, or lack thereof. Plus internal divisions, ie splits between wholesale/retail and sometimes some regulatory PITA to deal with. But basically boils down to budgets and silos, and then finance deciding speculative investment was a bad thing. Even though as you say, if you don't make sensible investments, you don't grow your customer base and risk eroding it. So the first customer to move into that new estate gets told 'no service' because the installation costs would mostly get loaded onto that first connection.. Which is especially frustrating when the developer's on board and willing to do the civils, and even help with a bit of marketing. Virgin high speed broadband is available!
So all rather short-sighted. Retail gets new prospects, Wholesale gets to keep some revenues even if the customer decides to switch because you can still rent that VLAN to another ISP.
Very happy to have switched from Virgin to back to Zen last year, ditching the coax "fibre" from Virgin for proper optical fibre from Zen/Openreach. 1,000 Mb/s internet connection now, as fast as the internal network!
A faster service is nice, but Support people who know what they're talking about is so much nicer!
...Probably something to do with VM still selling every detail of what you connect to online to "Phorm" (even though they publicly claim not to....).
The fibre arrived in my street last Thursday, and today I started the dance with VM to cut them completely out of my life.... Zen have been very helpful and have suggested that they may be able to help with cessation of VM "service".
Interesting VM factoid: A "Speedtest" on VM shows down speeds of 255Mb/s here, and a couple of the others show similar speeds. A proper test shows real speeds of around 22Mb/s - less than a tenth of the rate I pay for. The sooner this crowd are consigned to history, the better. I'm considering legal action to get 90% of my bills back!
"We have asked VM why there was no resiliency in terms of call centers and its own website, a situation that suggests failures of what should have been built-in network redundancies. "
in this case, the call centres would have hit capacity and the website would go down due to people continuously refreshing it. On their mobiles of course. That aren't on Virgin mobile. Or on a cell being backhauled on Virgin infra.
Yup, gone again.
After over 15yrs of being a happy VM customer, the past 6 months have been awful.
Midweek, working hours so called planned maintenance x2
Terrible customer service when reporting issues.
Constant "higher than normal" IVR message but refusing to hire more staff.
When YouFibre finish installing round here, I'm jumping ship.
Snap.
Virginmedia, despite their sewage level customer service and constant price gouging have, until now, provided a fast, reliable broadband service for the 20+ years I have had them. It's their one redeeming feature. I am now in the position that 2 of the 3 worst outages in 20 years have happened this week.
Communityfibre called to ask if they could cable up our block of flats. They are currently offering 4 times the performance for half the price. Like you, I'm off as soon as the new fibres are installed.
To be fair, that's most likely coincidence. Like the local Tyneside Metro train system with a major borkage in the middle of the network effectively severing the system into two separate networks on the same day the prices went up 14%. 800 metres of overhead power line came down and took a few days to replace. Being fair to them, they did immediately apologise and said it was major and could very likely take up to a week to fix then got the system back up and running by day 4, Sunday. Luckily for the users, it happened during live running times so there were trains running on either side of the break. If that had happened overnight, the trains would have been in the depot and 50% of the network would have no trains at all. Resilience is the dreaded "bus replacement service" bit in this instance just between about a three station length of track. Slightly better redundancy than VMs network :-)
"PHB: Show us what you were doing when the network went down.
PFY: Well I just did this, and this, and ... oops."
Fifty years ago when I worked at Harwell we had a guy who did exactly the same thing.
But the question was, "Show us what you were doing when you got the electric shock..."
I still remember it with a smile after half a century.
"150 phone contact goes to number not available."
Not that this would make any noticeable difference to the service. Getting to speak to an actual person at vermin media is like waking the dead. You're fobbed off to their website, their app, their WhatsApp, then you're on hold, and then you get put through to some Muppet who is no use whatsoever.
I've been with vermin for years, and generally I find the service pretty reliable - certainly far more stable than my neighbours' ADSL. But their support/ customer service is diabolical at best.
I work for A N Other ISP who uses multiple providers, including VMB and have experienced multiple outages of late due to their inability to identify what services are affected by their own maintenances. What made matters worse was they did the same balls up twice in a row on consecutive nights.
Their comms on the matters involving multiple 10gig services has been abysmal.
This latest balls up doesn't surprise me. I moved my home services away from them years ago due to poor service and I feel a certain sense of smugness at the rightness of that decision.
I noticed it was partially working at 9am. It's been on and off all day. Seems OK now (9:30pm).
Luckily, we have an O2 sim in a mobile broadband device - good enough for my wife to run a decent Zoom session.
This is the longest outage I've had up here. It isn't enough to make me leave Virgin.
Alan
Customer service doesn't exist. Bit like 'Net Zero'.
If your internet connection goes down, make yourself a nice cup of tea and read a book, do some exercise, count the kids to see if any have left home, watch a Kdrama or dig the garden.
It will eventually work again. Complaining won't make any difference. Loads of angry El Reg commenters will be using their satellite internet, neighbour's internet, semaphore or land line to yell at cubicle droids, and all to no avail.
You don't get a snow day working at home, but if your net is off, you just can't do the worky stuff until your LEDs turn green again. Devastating, but you cannot fight fate.
Redundancy is important. Make sure you always have enough biscuits in the cupboard to get you through this sort of thing.
Plus side : fast, pretty reliable (but then I'm on old NTL cable here and not much has been changed by vermin)
Meh side : the price they charge
Downside: bollocks 'customer service' I say customer service more like it runs
"lets answer the phone and disconnect it.. then answer it again and read the reboot script for the 100th time to the customer... until the customer demands being put through to level 2"
<on hold for 30 mins while listening to classic music played through 2 cans and some string while being told your call is important to us... then randomly disconnect to make the customer go through the menu /customer service agent again>
"Level 2 answers , read the reboot script, then claim as your connected via your own router or Linux installed PC, they cant help you until the customer starts swearing... then put through to lvl 3"
<on hold for 30 mins while listening to classic music played through 2 cans and some string while being told your call is important to us>
"Level 3 answers (usually a nice scottish lass/lad) connects to your router, goes I see the problem...... and you're online"
Also... just how do I call their call center with the IP phone they provide when the internet is down?
Customer service for residential customers has been shocking for a long time. My particular favourite being a customer experiencing intermittently very high latency. Traceroute showed where in Virgin's network the issue was occuring, but after several days of calling them the best the customer could get from them was the offer of a replacement "hub" (it's not a fucking hub) however the customer was told that as they were "out of contract" they would have to sign up for a new contract before they could get that. Finally the customer got hold of a number for a second line engineer who isolated the fault to a link at a POP and got it fixed. Basically it seems to be that their first line's main job is to fob customers off and if possible get them to sign up to a new contract.
Their business support used to be good, but has been sliding gradually down hill for a few years. Recently however it has fallen off a cliff.
When I worked in local radio my boss was very proud of the fact that he had "obtained" (somehow) the direct-dial number for a telephone in the racks room at the local BT tower. If we ever had a problem with one of our circuits (analogue lines feeding our FM and AM transmitters or running to local sports venues etc.) after checking it wasn't anything at our end, the next thing to do was to ring the phone in the racks room. Usually an on-duty engineer would answer, would exchange a few pleasantries with my boss, would wander off to look at the circuit in question and either get it fixed pronto, or be able to say what was wrong and escalate the issue.
Much, much more efficient than trying to get through BT's service centre*.
NTL cable, of course, was some kind of offspring of NTL Transmission who actually ran our transmitter sites so when they started cabling up the area we had a couple of their ISDN lines installed at half the price (installation, rental and call costs) that BT were charging. Our OB services were moved from analogue lines to ISDN dial-up around the same time.
Virgin has recently been expanding their network in the next town along. There's no way they will come to the hamlet where we live, but I had been wondering about signing mum & dad up, not so much for the internet but for the TV. Maybe now is not the time.
M.
*Of course, the absolutely most efficient fix we ever had was when one of our AM services suddenly went off-air. I'd just come back from lunch and nodded "good afternoon" to two engineers up to their waists in an underground DP jointing cables and listening to our station on a little radio. On the offchance, I popped my head out of the door to find two sheepish engineers looking at their silent radio... the AM programme was in the same bundle of "telephone" cables they had just disconnected**. Needless to say, we were back on air before the automated systems had "failed over" to rebroadcast our FM signal.
**The FM programmes went up special starquad cables, so were much less likely to be confused
One, where a certain Islamic Republic decided to hijack an AS, causing intertubes wide outage of, oh, Googleish.
The other, a fat finger outage, at an odd point, doing exactly the same thing.
I'll say, based upon a certain invading country's MO and sudden emphasis on targeted online efforts, one could wonder, but then, Hanlon pokes his head up.
Suspicious timing, a bit off target, to be generous, not a lot of specific targets on the domain, I'm thinking Hanlon, for a change.
In short, a finger in need of going on a diet. We've all had one of those.
Once, by order, designed by me, of a mandatory reboot of XP machines weekly and always before a new operation, rebooting my installation commander's computer - during his briefing of his General.
There was much mutual laughter over that one, hoist upon one's own petard humor and precisely zero "additional guidance", per both authorities.
Another, due to an ill behaved, passed an abbreviated test group patch from Adobe, forcing a reboot on patch. Adobe didn't like the hate mail from senior government officials.
I loathed the downtime.
We are aware of a Major Service Outage and all of our support teams are working together to have this fault resolve ASAP.
Current Situation
Update 7 (Apr 4, 2023 6:24 PM BST)
FPC-2 at Brentford was successfully reset from 16:54, which restored connectivity for Aorta services via Brentford temporarily. FPC-3 at Brentford then went in to a 'wedged' state, shortly after FPC-2 was reset. FPC-3 at Brentford Backbone-3 was successfully reset from 17:26.
Juniper Engineers retrieved the recent log data from Northampton Backbone-3, before the affected FPC cards were reset.
Work to restore the routes via Manchester Backbone-3 is underway. The FPC card reporting errors is being remotely reset as a first step.
FPC-2 at Birmingham Backbone-3 also began to report issues at approximately 17:25. DINMC 2nd Line also reset this FPC card.
Utilization on FPC-3 at Leeds Backbone-3 was also noted to be rising from 17:57.
Following advice from Juniper, further DDoS Mitigation was applied at Brentford Backbone-3 from approximately 18:00. The remaining Backbone routers at Northampton, Birmingham, Manchester & Leeds have also had the mitigation applied.
Impacted FPC's cards are being reset to stabilise the hardware pending further advice from Juniper.
The technical conference bridge remains ongoing.
Plan
- Support teams to reset impacted FPC's
- Juniper to provide next steps
- MIM conference bridge ongoing
This post has been deleted by its author
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
