back to article Virgin Media struck dumb by NATIONWIDE packet loss balls-up

Virgin Media's broadband network suffered what appeared to be a widespread service disruption on Saturday – the second such weekend, packet loss blunder to have struck the cable company in recent months. Customers complained on VM's forum and Twitter about major performance problems blighting the network. However, it took …

  1. Anonymous Coward
    Anonymous Coward

    Widespread, but not everywhere

    No disruption noticed in this VM-served household, despite spending much of Saturday online.

    1. Anonymous Coward
      Anonymous Coward

      Re: Widespread, but not everywhere

      Us to, North London. No problem with my service.

      1. returnmyjedi

        Re: Widespread, but not everywhere

        No issues in Northampton either. Go Saints!

  2. Glen 1

    ...and the cause was?

    I think while the "server reset" fixed the glitch, it is inportant that VM investigate why it glitched in the first place. Otherwise, we will be reading the same article in a few weeks time...

    1. Anonymous Coward
      Joke

      Re: ...and the cause was?

      Woah Glen - you're so right! I bet a company like Virgin never thought to investigate the cause of such an issue. Now you've pointed it out though I bet they'll get someone right on it.

      Have you thought about applying for a job with them?

      1. Naselus

        Re: ...and the cause was?

        Yes, I for one will certainly be advising my IT staff to start actually looking into problems from now on rather than just resetting stuff randomly.

    2. Anonymous Coward
      Anonymous Coward

      What really happened - A transcript

      This is a verbatim transcript of a call inside Virgin Media management who were discussing this outage.

      VM Management – Yeah, who’s that waking me up at this time?

      VM Technical staff – Sir, we have another outage on the network. This looks like a big one again.

      VM Management – So, whats that got to do with me? You’re the fucking technical staff, you plug this stuff together. You sort this out… [voice in background, do you want another line? Reply – Yeah, line it up on the mirror, let me get rid of this techie git]

      VM Management – Sorry, <sniff>, not talking to you, somebody else was in the room. No you’re doing a great job. <sniff>. So whats the problem?

      VM Technical staff – Sir, we have no idea what went wrong again, we could do a Root Cause Analysis but since we used those 1st year undergraduates to put the system together as their end of year project and we carried on using it and we never employed anybody to document it and we don’t have anybody who knows the passwords now, since they left as they didn’t get paid its difficult to say.

      VM Management – So, throw the power switch and kick the fuckers off. Tell the punters to reboot their routers and their PC’s. Everybody knows its always the punters fault for the first 12 hours. Do I have to do everything ?

      VM Technical Staff – Sir we’ve already done that but its still not working and its been 12 hours.

      VM Management – Well reboot it again, keep doing that until it works. That’s what Comcast do and they seem to do OK.

      [ Thirty mins and two reboots later]

      VM Technical Staff – Sir, it looks like its coming back now. No idea why.

      VM Management – <sniff> That’s good. Any idea what happened?

      VM Technical Staff – Sir, no idea at all. We still don’t have the passwords and if we did we don’t have any log files as the 1GB ATA disks were out of space so logging got turned off by the 1st year students.

      VM Management – Yeah, yeah, yeah don’t get technical with me, so lets just say its been fixed by our expert staff who worked around the clock, yeah I know I know, and that further investigations continue and we’ll update you when we reach a conclusion.

      VM Management – Since we still have no fucking idea what went wrong, and that bearded twat blew all our budget on adverts with that sprinter bloke, we’ll just delay and delay and then hopefully all the stupid punters will forget we had a problem until the next one. We can spin this out for months and because the only professional part of the company is the one that wrote the service contracts the punters agreed to, we can basically do anything we like, when we like, how we like including giving a piss poor service, customer support that Comcast would be ashamed of as well as throttling back anything that gives our sealing wax and string network a hard time.

      VM Management – So we’ll put some tweets out again, yeah use the same ones from last time, saying we’re really sorry, thanks for the patience shown, we’re working on it, blah blah blah, Use any old crap, they’re only the fucking punters for gods sake.

      VM Management – No, don’t give out any real information as a) the punters could use it against us and b) we really don’t have any fucking idea what went wrong.

      VM Technical Staff – Thank you sir, that’s very helpful.

      VM Management – Now fuck off, I’ve better things to do [ Voice in background – Another line – Yeah? Rack em up, that’s bloody good marching powder, let me get rid of this prat of the line]

      VM Management – Nah, not you, I’m <emmm> on a second line sorting another problem out. Yeah, yeah <sniff>, yeah, <sniff> brilliant. Now piss off and leave me alone.

    3. This post has been deleted by its author

  3. CaNsA

    Tldr

    ISP had a problem. People bitched. Isp fixed the problem.

  4. Anonymous Coward
    Anonymous Coward

    One server to rule them all

    "I can confirm that the problem has been resolved thanks to your reports on the forum. The offending server has been reset, which fixed this issue."

    One server?

    1. Anonymous Coward
      Joke

      Re: One server to rule them all

      > One server?

      Yeah - that's my IT dream - one huge server with everything on so you only have to restart one thing and no need to spend time working out which one went wrong. Saves a ton of money on OpenView as well. I think I might give it a snappy name: "main server" or "main something" at least?

    2. MR J

      Re: One server to rule them all

      There are a couple of "Key" servers on their network that when they go down, the whole network goes down. It wouldn't normally relate to generic packet loss, but I have seen it before. The basic stuff is that when the servers that protect us from porn or illegal downloads goes haywire, their whole network goes haywire.

      Years and Years ago there was a lot of theft going on, I asked why simple steps like duplicate mac scanning didn't take place and they said it wasn't really needed as customers didn't complain about stolen service.. It took them about 4 or 5 years after I pushed them for valid reasons for someone to turn around and come up with a great blinding idea... Scan for duplicate MACs on the network to thwart theft.... Wow.. I asked a lot of other questions and even now some of those basics have not been put into place. Most of my questions came from the black hat public cable hacking that was going on at the time..

      They have moved email over to the google apps system and its utterly shocking how many "features" don't work. Other ISPs can make it work, why cant VM.

      Their reason for customers not getting IPv6 is that there are still enough IPv4 numbers to be passed around to customers. Mull over how the two are not related, and how VM staff think that running out of IPv4 is the ONLY reason the move has to happen.

      I must say, I didn't spot the outage, and I have to wonder how difficult it would be for them to write some scripts to test end to end connectivity across the network. I have a feeling its wouldn't be that difficult to do, but eh...

  5. Anonymous Coward
    Anonymous Coward

    Unaffected in Gloucester...

    but we don't use Virgin's DNS servers, don't know if that's relevant or not...

    1. Down not across

      Re: Unaffected in Gloucester...

      but we don't use Virgin's DNS servers, don't know if that's relevant or not...

      Considering cablemodems (standalone and Tivo's built-in) failed to sync I'd say it is not relevant.

  6. SuccessCase

    I had a problem with my Virgin service. Kinda packed up completely for a day. Now at least it's clear why.

    Almost as annoying as the parent next to me in my local Costa Coffee, who thinks it's OK to let his daughter play on her iPad with volume on full so together with the Costa music soundtrack there is a sound cacophony going on.

    I've asked him if he has headphones for his daughter and at least he has grudgingly complied. Now we just need to organise a nationwide campaign. It is NOT ok to play media through your tablet speakers in public places like cafe's, for the whole cafe to hear. I'm a parent, but this creeping "children rule the roost and the delicate little dears mustn't have their play space in any way shape or form inhibited" is just infuriating.

    Sorry, completely off topic, but I just had to vent on this. Arrrgh!!!

    1. FraK
      Mushroom

      Next time that happens . . .

      . . . try firing up your media device of choice at full volume in retaliation.

      I heartily recommend the gentle sound of "Napalm Death - Scum" or perhaps "Discharge - Hear Nothing See Nothing Say Nothing" to get the party started.

    2. John Tserkezis

      "It is NOT ok to play media through your tablet speakers in public places like cafe's,"

      Nor buses, and it applies to adults who haven't grown up yet and insist on playing brain-numbing idiot games on the phones over half a friggin hour trip.

      It's sad when not even a cell jammer will fix the problem.

  7. Zog_but_not_the_first
    IT Angle

    Once again

    Is this a "happened in London, therefore happened everywhere" story?

    1. Captain Scarlet Silver badge

      Re: Once again

      Happened in Harlow so not just in London

    2. Terry 6 Silver badge

      Re: Once again

      Didn't actually happen in my bit of London, see above.

  8. Anonymous Coward
    Anonymous Coward

    Is there ever any compensation from companies like Virgin Media?

    I'm with xilo.net and recently had an outage (first in 4 years) that lasted some hours, they gave 1/3rd of my monthly bill as unasked for compensation.

    Having thought about it some more, these consumer focused companies would end up paying people to use them rather than vice versa.

    1. Lee D Silver badge

      As per your terms and conditions.

      Do you get compensation every time your electricity blacks out for a few minutes?

      Do you get compensation if your phone goes off for a while?

      Do you get compensation if your milkman doesn't turn up one morning?

      Do you even get compensation if your train is fives minutes late EVERY MORNING FOR A YEAR (and some days doesn't come at all)?

      No.

      Because you don't have it in your contract (express or implied) and it's not reasonable to provide. It's not just Virgin, most companies would go bankrupt overnight if you could automaticallly get out of them anything more than what your contract cost. If your Internet connection, or power connection, or transport MATTERS, you'll have a contract with a cast-iron guarantee written into it. And it will cost orders of magnitude more to compensate. And similarly, it will cost orders of magnitude more when you mess up and it's YOUR fault that it's off but they have to send out an engineer to prove it.

      Have you SEEN the prices of leased lines? And invariably they still can't guarantee 100% service. You can still be off for X amount of hours per year and have NO comeback whatsoever. And the law generally holds that, unless specified otherwise, your maximum compensation would amount to the direct cost of the lack of service and not any cost of your reliance on it. So you'd get back, say 1/30th of your monthly cost for a day without it, at most, on a standard home package. What's that? A couple of quid? Everything above and beyond that is goodwill. And I'm betting Virgin have a few more customers than Xilo. So much so that they don't need to provide goodwill payments (but I have, in fact, had one when a pay-per-view movie on their cable service stopped mid-way, and they refunded and gave me a free movie). And I'm on Virgin and it's NEVER down long enough to even suggest faffing about. Last time I rang them it was because the local kids had pulled all the cables out of the their green box at the end of the road and cut off the Internet and TV. It was fixed within the hour.

      If you're really that reliant on your Internet, you need to get a proper contract guaranteeing it. But something tells me that you'll whine that you wouldn't be able to afford that and it's totally impractical unless your very life/business hangs on it working. And there's your answer.

      Nothing in law entitles you to compensation beyond the "direct loss" of the service (i.e. what you paid for but didn't get) unless there's extreme amounts of negligence or they were aware that it would have a much larger impact on you and still agreed to provide that service.

      Compensation is, in fact, extraordinarily rare in any contract unless your dealing with serious amounts of money. And rightly so. Take your money elsewhere, rather than provide the opportunity to use every company and organisation as a honey pot for the slightest mistake one of their employees might make.

      1. Zacherynuk

        Errrrm.

        Do you get compensation every time your electricity blacks out for a few minutes?

        -Usually yes. If you ask. EDF user here.

        Do you get compensation if your phone goes off for a while?

        Everytime. If you ask. BT user here.

        Do you get compensation if your milkman doesn't turn up one morning?

        --Absolutely - Milk and more user here.

        Do you even get compensation if your train is fives minutes late EVERY MORNING FOR A YEAR (and some days doesn't come at all)?

        -Yep. Southern rail season cards are discounted due to poor trains regularly. If you ask you get - though I think cut off is 10 minutes.

        I wonder if what you are doing wrong is not talking to your providers. It's a market out there, don't you know ?

        With regards to SLA lines - I manage several dozen 10mbit upto 1Gb connections and absurdly these are the ones so difficult to get recompense for when they fail. They fail less for sure - but your options for change are mind numbingly difficult. Ever changed fibre provider cheaply have you ? me neither.

        Shut up. You are clueless. People have to complain to get stuff, consumers especially. Let them complain. let the complaints sit around for review down the line and let market forces dictate that we all get better everything.

    2. batfastad

      Have an upvote for name-checking Xilo.

      Moved to them just after Be started pushing people to Sky, and couldn't be happier. Was using O2 Wholesale through Xilo so basically the same connection as I had on Be. Then O2 Wholesale was sold off so migrates to TalkTalk Wholesale and still happy. Central London and no FTTC when all that happened, so sticking with FTTE (fibre to the exchange (ADSL)).

      Xilo are just resellers but their customer service is outstanding... Just like Be.

      Plus all the net nanny stuff like Camron's national censorship filter and blocking of the obvious t*rrent trackers is disabled.

    3. penguin42

      I've had bill reductions for outages on Virgin; they're only at approximately the number of days a problem has carried on (or maybe one extra days worth); but in my experience if you ask after a particularly nastylong screwup they'll give you something.

      1. werdsmith Silver badge

        I had no problems with VM at all on Satur......

        Oh wait.. what did you say there? Compensation?

        Couldn't do a bloody thing on Saturday at all, couldn't even get on youPorn. Where's my money?

    4. Down not across

      Is there ever any compensation from companies like Virgin Media?

      Yes. But you need to complain about not getting service you're paying for and the drones will apply credit to your next billl.

  9. The Onymous Coward

    They're having problems again. High-traffic websites like eBay and Google are not accessible right now.

    I can get to Bing though. First time for everything.

    1. StooMonster

      Yep, many sites like The Register are fine ... but sites like BBC iPlayer have been troublesome today (Sunday).

    2. Sheep!

      "I can get to Bing" - well I'd definitely demand compo for that!

  10. Dave Bell

    There are so many different companies involved in getting data from A to B, and weekends have sucked generally for a long time, with different people blaming somebody else for every problem.

    And then you have to deal with the helplines. Some are much better than others.

    There is probably a college course in Bangalore on not listening to customers.

    I know how to use traceroute, and I know exactly who is carrying my data between A and B. Company B doesn't want to say just where their server is, but they put the info in the server domain name.

    At the end of the day, I am still the paying customer, but that doesn't seem to make a difference. At the end of the day, my name is on a contract, but nobody cares.

    My email probably arrives at GCHQ rather sooner that it arrives at the addressee's account.

  11. The Onymous Coward

    I phoned VM and spoke to someone in tech support who seemed to genuinely want to help, asking me to run pings, trace routes and nslookups (although I use my own DNS). After half an hour of sounding reasonably interested in fixing things, he advised me to restart my router :-(

    As we were finishing the call and I was waiting for a reference number, the helpdesk chappie exclaimed "oh my system's just restarted for no reason!". "Oh, it's just like my TiVo box then," I said. "Different help desk..." came his response.

    I spent some time this evening looking at broadband-only providers and Freeview HD PVRs. If Xilo are widely praised and offer a monthly rolling contract, they may well have a new customer...

  12. werdsmith Silver badge

    I was using VM all Saturday, for both VPN into work and straight to the web for other stuff.

    No problems at all with service. I reckon rival ADSL providers spread this story and what really happened is that somebody in London (a VM customer) had a cheap TPLink router in their house, and had to reboot it because it got its knickers in a twist. Told their neighbour who works for rival and they turned it out as a nationwide outage.

    1. Down not across

      I was using VM all Saturday, for both VPN into work and straight to the web for other stuff.

      No problems at all with service. I reckon rival ADSL providers spread this story and what really happened is that somebody in London (a VM customer) had a cheap TPLink router in their house, and had to reboot it because it got its knickers in a twist. Told their neighbour who works for rival and they turned it out as a nationwide outage.

      Good for you.

      The cable modems dropped out. Would sync again and drop out again. VM said it would be fixed in 4 hours. ~10 hours later they sent SMS that it is fixed.

      Well it wasn't. Eventually it worked for a while before borking out on monday again.

      So you reckon wrong.

      And no, I'm not in London. And I have no TPLink stuff in the house (not that it would've mattered in this case).

  13. Anonymous Coward
    Anonymous Coward

    VM outages in North London

    Virgin Media broadband was down in my part of North London on Saturday morning from 1.30 to 3.30am. Earlier in the year the service had been very reliable. But there have been sporadic outages since September:

    September 23: Out for an hour around 9.30pm

    October 13: Out from 7.30 to 9.30pm

    October 26: Out for an hour at 5.30pm

    November 1: Out for an hour around 9.30pm

    November 6: Out for an hour around 3.30pm

    November 7: Up and down all afternoon, but never for an extended period

    November 11: Out for an hour around 8.30pm

    November 15: Out from 1.30am to 3.30am

    When it's up the performance is pretty good - around 25MB/s down and 2.5MB/s up.

    1. Lallabalalla

      Re: VM outages in North London

      I'm in N11 - we've had a few ups and downs over the last 9 months.. mostly ups, once I had complained about the very slow connection (<2.5mbps) after supposedly being "upgraded" to 50mbps and they poked something with a stick*. Didn't notice this latest one at all though, and everything seems normal at >= 50mbps

      *BT used to actually do this to jolt an ADSL connection back to life!

  14. Narrator Jay

    Virgin in Hampshire three month refund

    In late October our internet service worsened considerably (mid-Hampshire).

    I rang up, the Virgin techie apologised, ran tests, explained that everything was going slow due to lack of capacity. Households were connecting smartphones and other items that they weren't using a few years ago. (True, my house has gone from one device in 2010 to seven this year.)

    He explained that Virgin were working with the council to get planning permission to dig trenches and lay more cables.

    Now I get 50% off my broadband bill until end of December.

  15. Anonymous Coward
    Anonymous Coward

    Um, wrong tweet methinks...

    Er, isn't the first tweet is some punter who hasn't paid his bill>? He's "renewed his service" and they've not yet turned on his connection. Not exactly related to "woeful" performance of VM.

  16. Amorous Cowherder
    Facepalm

    Had a few drop outs. So I went off, cooked dinner, sat and watched some TV for an hour and it was fine afterwards. Jeez, never mind 3 meals from anarchy, it's more like 1 packet loss from anarchy these days!

  17. Mayhem

    Google DNS in London

    We had a weird issue last night - we completely lost access to anything Google related - bing worked fine, yahoo worked fine, but google.co.uk and google dns dropped off the net and we lost all packets routed via their networks.

    Redirecting our router DNS to the automatic Virgin provided ones brought everything back up.

    I plan to revert the change when I get in tonight, but it was probably related to this.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like