back to article Midnight theft left Vodafone users bereft

Vodafone's Monday morning outage was caused by thieves who broke into the operator's Basingstoke exchange and lifted a load of switches. The break-in happened around 00.30 this morning, and the police were quickly notified. Vodafone noticed its own network collapsing and assembled its "War Room" which is supposed to deal with …

COMMENTS

This topic is closed for new posts.
  1. Anonymous Coward
    FAIL

    Fixed? not yet

    I'm in Basingstoke. Still down

  2. Tony S
    FAIL

    What?

    None of the users here could get a signal first thing, and we are about 1 mile from the Atantic ocean - hardly "South East England".

    As for "only 100,000 people affected" - pish, tosh and other words. There were probably that many complaing on the various forums, but that is not the sum total of those affected.

    Mind you, I love the comment on Twitter by someone that tried to cal someone in the Vodafone PR office, only to hear the message "Welcome to the Orange Answer phone service".

  3. SnowCrash
    FAIL

    Fixed for any given definition of not fixed

    Yup - still down in Reading.

    1. Gordon 10

      You sure?

      Working intermittently in Caversham (across the river) since 12 and solidly since 1.

  4. despairing citizen
    FAIL

    BC/DR?

    Ok so how does 1 break in knacker service delivery for such a large area, and is still in trouble 12 hours after the incident.

    Maybe the customers and share holder should be asking to see vodafones BC/DR plans before handing over any more cash.

    1. Anton Ivanov
      Grenade

      It is considerably more difficult than you think

      The architecture of the "intelligent" networks and its IMS based 3GPP successors has a considerable number of elements which are very difficult to have on hot-stand by and/or in failover+load-balancing mode.

      This is something inherent to the arch in question. If you take out a BSC or RNC everything slaved off that BSC is dead and you have to reprovision a significant chunk of your transmission as well as commission a whole new BSC to fix it. If you take out an MSC the story repeats just multiplied by a factor of several thousand times for all of those BSCs or RNCs that have been orphaned by the death of their master. If you take out the policy function, the diameter or some of the key bits of the EPC a big chunk of the network dies in a similar fashion.

      In fact it is inherent to the idea of the intelligent network. The more intelligent the network the more vulnerable it becomes to someone taking a big axe and surgically removing key bits of the "intelligence".

      1. Anonymous Coward
        FAIL

        One step forward two steps back?

        "In fact it is inherent to the idea of the intelligent network. The more intelligent the network the more vulnerable it becomes to someone taking a big axe and surgically removing key bits of the "intelligence".

        Technical issues aside, Vodaphone sell a business service, they should be planning on bits failing, or clearly stating that their service is not suitable for mission critical work.

        If they want the business service profits, then they need to pay up for the backup coverage

  5. Paul Eagles
    FAIL

    Fixed? Nope.

    Definitely not fixed here in Guildford. I can make calls but anyone calling me is helpfully told that 'the number you have dialled is not recognised'

    Data is extremely sporadic too, I had a bunch of emails appear on my Blackberry about an hour ago but the thing has been silent since.

  6. Anonymous Coward
    Anonymous Coward

    The morning was wasted

    by posting complaints to Twitter, Facebook , El Reg, the BBC and anywhere else that would take them, along with constant refreshing.

  7. CarlC
    Unhappy

    Still down

    I can not recieve calls here. We are just south of Reading. Any incoming call rings once then shows as a missed call on the handset. On the calling landline we tested from you get a single ring then a number unobtainable tone. Definately still down.

  8. Chris 100

    For anyone who is still down...

    For anyone who is still down, try rebooting your phone. That should get you back on the network.

  9. Anonymous Coward
    Anonymous Coward

    Swindon, this is Swindon

    Finally back up in Swindon.

    Swindon isn't in the South East. Your article is incorrect in terms of the areas affected, as it covered much more than just the South East, as your earlier article about this issue correctly mentioned.

  10. Anonymous Coward
    Anonymous Coward

    Sorta fixed

    Colleague in the Penines has service back but others in Reading still off air.

  11. Paul 191
    FAIL

    Millions

    Yeah, took out our entire company, and everyone I know with vodafone - how exactly do they work it out at only affecting a few customers?

    Service back 10 minutes ago, no calls/txt before then.

    Vodafone clearly has massive single point of failures in their network, and very poor backup planning!

  12. Stephane Mabille
    FAIL

    Single point of failure?

    Hi,

    14:45 still unstable around Reading.

    This poses serious question about disaster recovery plan at Vodaphone... If the Basingstoke datacenter was destoyed by fire it seems it would bring south (eastish) UK network down.. for how long?

    I suspect other key premises are in the same situation.

    Any SLA / compensation planned?

    1. Mark61
      Thumb Down

      If you pay £30 a month line rental...

      ...that'll be 50p compensation for service lost for half a day.

      Frustrating as outages are, I don't think Vodafone contractually owe retail customers more than that (and even then, it's a stretch, as they cover themselves with protective clauses in their T&Cs).

  13. Anonymous Coward
    Alert

    TomTom Traffic Updates

    Anyone have problems in the affected areas getting Traffic Updates on their TomToms today?

    These use Vodafone for the data service.

  14. dave 46

    Nothing new

    Everything gets cheaper, the only thing we care about is how much we get nobody ever stops to question the quality, what backup is there, what resilience.

    I don't imagine any other carrier is any different.

    They may review their security procedures but that is it - nobody really cares and everybody will have forgotten about this when the contract comes round again, they'll care more about which network has the shiniest phone and the biggest data package / most texts.

  15. BorkedAgain

    Meanwhile, on the Hampshire coast...

    ...no difficulties whatsoever with Vodafone. Full signal, making and receiving calls quite happily. Maybe I'm one of the 100,000 unaffected users... :)

  16. Paul 191
    Thumb Down

    NETWORK BUSY

    What happens when a few affected customers finally get their service back? It jams entire capacity on the network it seems...

  17. Tron Silver badge

    Well...

    Corporate seriousness chart.

    1. Bonuses.

    2. Wage level.

    3. Company car.

    4. Perks.

    5. Executive drinks cabinet security.

    .

    .

    134. Infrastructure security.

    135. Customer satisfaction.

    As a back-up, get a POTS phone that doesn't need mains or a wireless connection. No frills. Under a tenner.

    The simpler stuff is, the more reliable it is.

    Never trust or rely on technology. Never trust or rely on those who provide technology or tech services. Alway back-up your data. Always have a plan B for when the tech fails (which it will).

    1. frank ly

      Plan B

      My pigeon loft is ready and on hot standby for a situation like this.

    2. Verne
      Pint

      Well... → #42

      The use of POTS as a back-up is indeed a good plan, however there seems to be a move away from providing a simple piece of "wet string" in new buildings. As the IT guys seem to be responsible more and more for cable infrastructure they like Cat5, hubs and switchers. At a recent job in Turkey seven new sporting venues had been built without any from of POTS infrastructure.

      Back to Vodafone, I hope they learn the lessons from this incident. They have provided me with a good service for over ten years now and, in my book, still have the best coverage.

  18. Linbox
    WTF?

    SureSignal?

    Not sure how their network joins together and it might be a red herring, but I'm about 12 miles from Basingstoke and have enjoyed 5-bars uninterrupted service all day both home and at work. All of my colleagues are offline. The only difference is - I have a Voda SureSignal box at home & work?

    Coincidence?

    1. Jean-Paul

      Suresignal

      Suresignal comes out of the network 'up north' somewhere. That is why I had great reception this morning at home and got my own network wakeup call at 4:28 when some of our services went down. But unfortunately once in the car and into London it was gone.

      It is slowly getting back now but still very patchy despite my phone showing good reception.

  19. Miek
    FAIL

    Still fixed, um, not quite

    If you are not seeing a signal, turn your phone on then off (or enable then disable airplane mode) and you should see a signal. I have found I can place calls but not receive calls, my other colleagues in the office are experiencing the same symptoms.

  20. Anonymous Coward
    Anonymous Coward

    The title is required, and must contain letters and/or digits.

    http://forum.vodafone.co.uk/t5/Mobile-Broadband/Vodafone-Still-Stalking-Subscribers/td-p/671569

    Somebody complains about illegal interception and then suddenly their exchange gets broken into? Coincidence or somebody trying to get rid of evidence? (where's the black helicopter icon when you need it?)

    1. Anonymous Coward
      Anonymous Coward

      what are the chances

      That anonymous broke in dragging an extension cord, moved the 2nd power supplies to their UPS, and removed the equipment without powering it off.

  21. Miek

    The break in occurred when ?

    I noticed a lack of communications on my handset in Oxford at around 10:20pm last night. I remember this because my Virgin broadband had died again and I was making a note of the time .

  22. My Alter Ego
    FAIL

    "Quite why that morning was wasted we still don't know"

    Not everyone is using their mobile to play FarmVille on their mobile. Everyone in our office actively trades with people on the road who rely on their mobiles to earn their (often self employed) living. While it hasn't affected us directly, it's been bloody annoying not being able to get hold of many of our contacts.

    It's guaranteed to have cost a good few people money. Sure, they're only paying Vodafone £1 per day, but it's more than a little embarrassing for Vodafone to demonstrate how bad their disaster recover is.

    As somebody else has commented, you have to make cutbacks to generate £11.5 billion profits. Obviously security was one of those areas.

  23. Chris Harden
    WTF?

    Everything I have to say on this can be summed up with the below video, and one line

    http://www.dailymotion.com/video/x8m5d0_everything-is-amazing-and-nobody-i_fun

    I hear someone stole their infrastructure - with a bulldozer, and you can't call someone for half a day? Quit your bitching.

  24. Scott Broukell

    how long ....

    ... before said switches turn up on Tat Bay then ? Shirley the miscreants won't be that stoopid will they ?

  25. Magnus_Pym

    What I want to know is

    Are infrastructure quality network switches readily tradable on the black-market and if who wants 'em?

    1. PeterGriffin

      Ditto

      Precisely the question I would like answered. Who would buy this pilfered kit?

    2. This post has been deleted by its author

  26. kabuki.usb

    Scotland or Gobi desert

    I have been using Voda fone for the last two years for my phone and mobile broadband connection , I have always found their service excellent .I think a bit of commercial espionage could be behind this attack and it's not the first time this has happed.When my service went down today I rang 191 and went straight through to their bonnie call center in Scotland and as usual spoke to a polite and competent person who gave me the full details of the situation and a free £5 top-up to cover my inconvenience.When I was with T-Mobile I had to talk to some toothless old git in a tent in the middle of Gobi desert reading from a script in badly broken English.

  27. Jacqui

    VF still down in camberley/bracknell areas.

    We have three VF handsets here. Only my tocco lite would rcv a call for a short while but no handsets would originate a call. Then an hour later ~3:30 incoming died and the balance OSD died as well. test calls at 6pm and the VF handsets are still DIW

    I suspect they are still rebooting and recovering from thier outage.

    My worry was in checking our PAYG balances to ensure that afterthe outage we have not lost existing credit (again!).

    My last fiasco with VF was a android handset eating 1UKP/hr when switched off and battery removed. which is why I moved to gifffaff for the android handset.

    Now I understand why others have more than one handset or a dual sim card handset.

  28. Anonymous Coward
    FAIL

    Swindon - Village Near....

    Still not working correctly, was getting calls flashing up for a moment then disconnecting before phone could even ring, 18:15, can make outgoing calls to non vodafone numbers, however cannot call my colleagues on Voda.

  29. pompurin

    I heard on the grapevine

    that they had over £500,000 worth of HSPDA kit stolen. Probably off to China/Nigeria/Brazil/Russia/[Insert Dodgy country here].

  30. Steve B
    Flame

    GPRS was off as well.

    Our dispatch system uses GPRS to communicate bothways. It disappeared at about 7 am for about 7 hours. unfortunately the only back up is mobile phones! That's about 40 self employed people who can't earn.

    Whoever the Vodafone DR person is, they must be feeling somewhat insecure at this point. and justifiably so.

    however I am available should they decide to revisit the plans..

This topic is closed for new posts.

Other stories you might like