back to article Total Inability To Support User Phones: O2 fries, burning data for 32 million Brits

Customers of O2, GiffGaff and virtual operators who use Telefonica's network in the UK have been hit by a spectacular outage across the country. Transport information services have also been affected. Ericsson, whose central user database caused O2's 2012 25-hour mega-outage, told The Reg: "We are aware of the issue and are …

  1. Alien8n

    Not just O2

    There are reports of it affecting Vodafone and EE (not so much Vodafone from what I can tell).

    The Down Detector page for O2 is full of outraged people having a go at O2, but in reality there's nothing O2 can do except wait for it to be fixed. Lot's of "OMG I'M NEVER USING O2 AGAIN". Imagine their shock and horror when they discover these issues affect all mobile providers at some point. Maybe we'll see them shouting "I'M NEVER USING A MOBILE PHONE EVER AGAIN" in the future.

    If you want a laugh though there's a chap on there claiming to be "Nigel from O2" who is doing some expert trolling of the outraged masses...

    https://downdetector.co.uk/problems/o2/news/232190-problems-at-o2

    1. Anonymous Coward
      Anonymous Coward

      Re: Not just O2

      I didn't realise I was using O2 until now - OMG I'M NEVER USING TESCO MOBILE AGAIN

    2. Anonymous Coward
      Anonymous Coward

      Re: Not just O2

      If you want a laugh though there's a chap on there claiming to be "Nigel from O2" who is doing some expert trolling of the outraged masses...

      https://downdetector.co.uk/problems/o2/news/232190-problems-at-o2

      I like his style - particularly impressed with his insights into Falkirk

      1. Aladdin Sane

        Re: Not just O2

        It's close to the whole "matey" style that social media drones take at the moment anyway.

      2. MrWibble
        Coffee/keyboard

        Re: Not just O2

        Love Nigel's work!

        "Hi Harry, Big fan!

        We are trying our upmost to sort this and understand as a professional footballer you need to keep in contact with your many fans! Ludicrous display last night in the game!"

        Problem with Arsenal is that they always try to walk it in...

        1. Lee D Silver badge

          Re: Not just O2

          Station guard?

          No it's more akin to complaining to the ticket office, and then being told "Not my fault, mate, we hire the ticket machine from Job Bloggs Ltd. I'm working for <insert rail company here>. Not my problem."

          If you bought a product or service from O2, your only legal, financial and customer-service recourse is to them, or an ombusdman of their industry. I don't care WHY they're having problems (whether that's that they haven't paid their bills, that their suppliers are useless, that their contractors didn't turn or, or that heavy snowfall in the Outer Hebrides stopped the consultant coming out today). That's up to their business processes to handle.

          My only interaction with them would be via the service they are contracted to provide, and are failing to do so.

          In the same way that no court would entertain someone saying "Well, my contractors didn't deliver the goods, so I couldn't give them to the customer who sued me" (they'd just tell them that's their issue, and irrelevant to the case, they are still bound by the contract whoever their goods come from), customer service, returns policies, etc. work the same. They would actually get sued by the customer, and then they would have to sue the supplier to get their money back if that was the case.

          You only have to deal with the people you bought the good or service from. You NEVER have to deal with any one of their contractors or manufacturers. Otherwise quite literally you'd be given the run-around between 50 different companies who all do one bit of the work, in order to fob you off as long as possible.

          Bought phone from shop and it doesn't work? Take it back to the shop. It's up to them to argue with Samsung/Apple.

          Bought phone service from O2 and it doesn't work? You shout at O2. They can blame anyone they like, it's their responsibility, choice of contractor and problem to resolve, not mine.

          Otherwise, you'll literally end up being told "Yeah, well, your local mast is run by Bloggs Masts Ltd, who we paid to run it. We don't care. Speak to them."

          1. werdsmith Silver badge

            Re: Not just O2

            Bought phone service from O2 and it doesn't work? You shout at O2.

            No, don't shout, whinge or whine at them. You will only get a service rep anyway And it just makes you look a dick. You can usually get much further by dealing with them in a businesslike way.

            Those people who have been conditioned make a huge fuss at every little thing are just bell ends.

    3. Korev Silver badge
      Pint

      Re: Not just O2

      >If you want a laugh though there's a chap on there claiming to be "Nigel from O2" who is doing some expert trolling of the outraged masses...

      I liked

      "Hi Matt, This is Nigel from O2.

      How does the gift of christmas cheer sound! This is the season for forgiveness and we at O2 would like nothing more! :)"

      One for Nigel -->

      1. tim 13

        Re: Not just O2

        Geraldine Wilson • an hour ago

        This is serious - O2 is out in Cambridge & it's been hours! My mobile is my only point of contact & I'm ill with flu right now. What happens if I need to call the doctor, O2?

        1

        Reply

        Share ›

        Avatar

        RealityCheck Geraldine Wilson • 42 minutes ago

        That's terrible news, I think without medication you will probably die, like they did in the old days. Sorry about that.

        Reply

        Share ›

        Avatar

        N-O2 Geraldine Wilson • 44 minutes ago

        Hi, This is Nigel from O2!

        I can only apologise for this, its a terrible time to be ill around the christmas period. I will issue this as a formal complain that you feel you have caught the Flu due to our network being down!

        Best wishes!

        1. Tom Paine

          Re: Not just O2

          In the US, at least, mobile operators and VOIP services were supposed to tell customers not to rely on the service for emergency services, as the networks don't have sufficient Ma-Bell approved reliability and resilience - as of a few years ago anyway. In other words, have a landline backup if you care about 24/7/365 availability guarantees for 999 service. Trouble is that no-one thinks about that nowadays, and the mobile network's probably almost always there. (Would be interested in the number of emergency calls missed due to network outages, presumably someone somewhere tracks that number.) Bet it's an infinitesimal fraction of the total number of 999 calls.

          1. Dal90

            Re: Not just O2

            >Ma-Bell approved reliability and resilience - as of a few years ago anyway. In other words, have a landline backup if you care about 24/7/365 availability guarantees for 999 service.

            Outside of core downtown areas, you haven't had Ma-Bell approved reliability since the early 1990s.

            Better than the cable company to be sure. But the neighborhood concentrators used to reduce the number of copper lines running back to central offices and expand the availability of DSL rely on batteries that die in 24-48 hours. And I live in an area that hurricanes or ice storms produce widespread one to two week long power outages one to two times a decade.

            My state has been slowly building out its own fiberoptic network connecting public buildings for this reason (loss of reliable telecommunication service). As long as the fiber is not cut (unusual from storms, since the the power lines usually take the brunt of the hit) and you have power (generator) for the optical network terminals, voice and data will continue to work.

          2. Telboy

            Re: Not just O2

            If O2 is down 999 and 112 calls will go to another network, try it, take your sim out and screen lock your phone and the dial the emergency services.

          3. TomG

            Re: Not just O2

            That infinitesimal fraction may have resulted in deaths. The fact that the number is infinitesimal is small comfort to the deceased person loved ones. Yes, a functioning land line has its use.

        2. cosmic caper

          Re: Not just O2

          this is why I still have a land line at home for emergency use

    4. Lee D Silver badge

      Re: Not just O2

      "The Down Detector page for O2 is full of outraged people having a go at O2, but in reality there's nothing O2 can do except wait for it to be fixed."

      They could switch to their backup system.

      Oh... you mean "nothing they WANT to pay to have in place for just such an occurrence that'll drastically affect their ability to operate if it ever happens"?

      To be honest, if I were coding things, I'd make sure that when the accounting etc. database was down, that data was still kept active anyway (it's run from leased lines on the masts, not from some central location) - yeah, you might get people use data unaccounted for while you're having issues but an unannounced "we've not counted some of your data because of a problem on our end" is far better than "Oops... everything's down for everyone and there's 'nothing' we could have done about it".

      This is a company with millions of customers that doesn't want to spend on a separate, isolated, failover database that doesn't get software-updated in tandem with their primary database. I have zero sympathy.

      Gimme my data, or stop running a telecommunications firm. Either way, stop running it like some mom-'n'-pop outfit without any way to fail back when the primary database falls over.

    5. MyffyW Silver badge

      Lack of O2

      I did wonder what Project Phlogiston was... but I was afraid to ask

    6. Anonymous Coward
      Anonymous Coward

      Re: Not just O2

      "The Down Detector page for O2 is full of outraged people having a go at O2, but in reality there's nothing O2 can do except wait for it to be fixed."

      Gobsmacked by the number of upvotes this has - because O2 have outsourced something, it's not their problem and people shouldn't be annoyed with them?

      1. Alien8n

        Re: Not just O2

        Fairly sure the upvotes have nothing to do with that... maybe something to do with "Nigel from O2" perhaps?

        And as an IT Manager part of my job is to manage expectations. Do I want a mobile phone provider that never goes down? Yes. Do I expect my mobile phone provider to never go down? Of course not, I deal with hardware and software issues all day, expecting a mobile provider to maintain 100% uptime is like expecting politicians to never lie.

        1. Anonymous Coward
          Anonymous Coward

          Re: Not just O2

          "Of course not, I deal with hardware and software issues all day, expecting a mobile provider to maintain 100% uptime is like expecting politicians to never lie."

          Yep, get that, your post suggested that because it was a third party behind O2, the O2 customers should be venting their annoyance at the third party.

          Agreed the Nigel posts are amusing though. Like a non-real-life version of the poor guy called John Lewis who receives a lot of queries on Twitter as to where their sofa is...

        2. m0rt

          @alien8n Re: Not just O2

          "expecting a mobile provider to maintain 100% uptime is like expecting politicians to never lie."

          Poiticians never lie. They reframe your question and answer based on an entirely different paradigm.

          1. Anonymous Coward
            Anonymous Coward

            Re: @alien8n Not just O2

            m0rt,

            "...Poiticians never lie. They reframe your question and answer based on an entirely different paradigm. ..."

            100% Incorrect with our current politicians.

            The bunch of nomarks we have currently (all parties) are not even capable of 'reframing the question and answering based on a different paradigm' ..... therefore ..... they lie.!!!

            Proof ..... Brexit (both sides) !!!

          2. therealmav

            Re: @alien8n Not just O2

            Poiticians never lie. They reframe your question and answer based on an entirely different paradigm.

            i call bullshit on that... evidence = boris johnson

            oh and everything the leave campaign said

      2. FuzzyWuzzys
        Facepalm

        Re: Not just O2

        "..because O2 have outsourced something, it's not their problem and people shouldn't be annoyed with them?"

        I bet you're one of those people who berates the platform guard at stations 'cos your train was cancelled? Sure you have a point, up to a point, but this isn't just a server room with 20 PCs running a couple of SQL Server databases. This is a huge, global f**kup by the provider, Ericsson I believe. So O2 are much like you when your heating goes out and you're waiting BG to come round and fix your boiler! Stop stressing, go outside and have a fag, come back when your nerves are calmed down.

        1. Anonymous Coward
          Anonymous Coward

          Re: Not just O2

          "I bet you're one of those people who berates the platform guard at stations 'cos your train was cancelled?"

          That doesn't really work as a comparison does it? If the argument was "stop berating the poor O2 customer services rep, it isn't his fault" then that's fine, but it wasn't.

          Presumably TSB customers shouldn't have been annoyed with their bank over the summer, they should be complaining to Accenture instead?

    7. FuzzyWuzzys
      Happy

      Re: Not just O2

      A good laugh over lunchtime, especially the "window-lickers" who think Nigel from O2 is a real O2 employee, some people just don't get sarcasm.

      1. Anonymous Coward
        Anonymous Coward

        Re: Not just O2

        Poe's law in the flesh

        Satire is lost in text form, it is lost even further when it is idiots who are the subject matter, namely the moron masses

    8. Nick Kew

      Re: Not just O2

      There are reports of it affecting Vodafone and EE (not so much Vodafone from what I can tell).

      Datapoint. I'm using EE 4G, and it's just fine.

      Unlike my phone's O2, which has no data.

      1. silks

        Re: Not just O2

        Same, I'm on EE and have had no problems either. o2 PR bull I suspect.

    9. Anonymous Coward
      Anonymous Coward

      Re: Not just O2

      Ref: Nigel from O2

      This may seem like harmless fun, trolling those who are distressed, even if the thing that is causing the distress is imagined.

      Could Nigel from O2 be a George Duke-Cohan in the making?

      First it starts with animal cruelty

      Then arson

      Then upping their game to something more thrilling

      If you're someone who gets enjoyment from other peoples suffering, even if it is just a trivial thing, then you are basically a sociopath. How far will you take it?

      Oh, and there is no cure for you. You are what you are

      Dead inside

  2. Anonymous Coward
    FAIL

    Pffft

    32M customers in a country with a total population of 66M. I hope the PM is requesting a report on what happened, that thing counts as national infrastructure.

    1. katrinab Silver badge

      Re: Pffft

      32m phones, which is not quite the same thing as 32m people, given that there are more phones than people in this country. I have two phones. TfL has 19,500 bus stops and 8,000 buses, and they have their own phones.

      1. wolfetone Silver badge

        Re: Pffft

        "32m phones, which is not quite the same thing as 32m people,"

        Exactly, drug dealers use 2 or 3 phones.

        1. MyffyW Silver badge

          Re: Pffft

          Vodafone have recently been strapping SIM cards to cows butts with an outfit called MooCall, so you can add the number of pregnant bovines to the overall phone count too.

        2. Anonymous Coward
          Anonymous Coward

          Re: Pffft

          metoo - wife, gf1 and gf2

          hey, the simple life is just too boring :-)

          1. Anonymous Coward
            Anonymous Coward

            Re: Pffft

            Your wife has two phones as well. I know this because I'm on with her right now.

            1. AndyS

              Re: Pffft

              *at least* two phones.

            2. Anonymous Coward
              Anonymous Coward

              Re: Pffft

              > Your wife has two phones as well. I know this because I'm on with her right now.

              Well, hurry up and get off it. You're interupting us, and he'll be home shortly.

    2. J.G.Harston Silver badge
      Joke

      Re: Pffft

      32m out of 66m, that means 34m are ok, so YOU LOST, GET OVER IT!!!

      1. wolfetone Silver badge
        Coat

        Re: Pffft

        "32m out of 66m, that means 34m are ok, so YOU LOST, GET OVER IT!!!"

        TITSUP means TITSUP!

        1. Danny 14

          Re: Pffft

          doesnt the smart electric meter network run on O2 aswell? Good job the things are useless.

    3. Anonymous Coward
      Anonymous Coward

      Re: Pffft

      It is less than half of the electorate so they don't count.

  3. Jay 2

    Work mobile on O2 and my own on GiffGaff. So no data for me. Though I'm stuck in an office all day with web and wifi, so not the end of the world. Though the lack of calls/texts earlier was somewhat annoying.

    I now await the examples of rage from users on Twitter who have based their entire existence on O2 infrastructure... If it's that important, have some sort of backup.

    1. Derezed
      Trollface

      Backup?

      I based my entire existence on O2 infrastructure, but fortunately I have a backup mobile infrastructure in my back pocket...stupid normal people probably don't.

      1. tiggity Silver badge

        Re: Backup?

        I have backup phone on a different network...

        .. backup phone is dismal spec, but at least I can communicate

        Ironically main phone is dual SIM .. but did not have SIM tool with me when I found the problem (had not transferred emergency kit (bag with SIM tool, spare SIMS of 2 networks, external charger & lead) to cold weather coat pockets! & no access to anything similarly small & pointy to get the sim slot open)

        So my being prepared turned out to be a bit of a fiasco. due to key items sitting in pocket of lightweight coat, not the coat I was wearing!

        1. TimMaher Silver badge
          Coat

          Re: Backup?

          A brief word of advice. Hang a paper-clip on your keyring. That solves the socket opener problem. As to the rest, maybe buy a burner SIM at the station or somewhere? Only if it’s really urgent.

          I’ll get my coat. It’s the light weight one for Summer wear.

          1. ShortLegs

            Re: Backup?

            @TimMaher

            [i]"I’ll get my coat. It’s the light weight one for Summer wear."[/i]

            ROFLMAO :) Wish I could give more than the one upvote

        2. Mr Han

          Re: Backup?

          People seem to carry a lot of stuff these days. I rarely carry little more than one visa card, one backup credit card, one phone, plenty of cash and two keys. I use a slim wallet for the cash and my mobile case for the cards, so I have no loyalty cards. If it doesn't comfortably fit in my trouser pocket then I don't carry it.

      2. Ben1892
        Trollface

        Re: Backup?

        I don't use any cloud infrastructure, I run my own mobile network on-prem. That way I know where the problem is and who I can shout at, rather than relying on someone else to fix it.

        1. cantankerous swineherd

          Re: Backup?

          but is it open source?

      3. David Paul Morgan

        Re: Backup?

        well, dual-sim OnePlus6T here...

        my main data sim is O2, my 2nd sim for work is, you guessed it, O2.

        My JT sim for international roaming (and does have a bit of data in it) is actually in my Sony Xperia Z4 Tablet, so cue my 'Dom Jolley I'm on the phone' moment...

    2. Commswonk

      @ Jay 2: I now await the examples of rage from users on Twitter who have based their entire existence on O2 infrastructure... If it's that important, have some sort of backup you need help. Urgently.

    3. Soruk

      Personal line is on giffgaff, and since we have to sort out our own work mobile (we get a mobile allowance for this) I deliberately put it on a completely different network. Still, I'm fscked if both EE and O2 go down at the same time.

  4. Buzzword

    Ericsson software apparently to blame

    According to the FT and the Telegraph.

    1. Anonymous Coward
      Anonymous Coward

      Re: Ericsson software apparently to blame

      Yeah, some idiot installed Win10 Autumn update.

    2. The Nazz

      Re: Ericsson software apparently to blame

      Don't quote me on this, but isn't that what Ulrika Jonsson (sp?) said on the ending of their relationship?

  5. WonkoTheSane
    Holmes

    BBC are naming names

    They believe the "3rd party supplier" are Ericcson.

    1. Anonymous Coward
      Anonymous Coward

      Re: BBC are naming names

      Well if Huawei are ruled out then its pretty much a 50/50 between Ericsson and Nokia.

      1. Jellied Eel Silver badge

        Re: BBC are naming names

        I think they're an Ericsson shop, but curious how Telefonica's managed to engineer a single point of failure. But given the outage seems to have started at 4am, suspect a bit of planned maintenance didn't go quite as planned. And neither did any plan to rollback to undo it.

        1. thondwe

          Single Point of Failure

          Engineering out a single point of failure nearly always leads to a complex system which can fail in catastrophically complex ways

        2. Hans Neeson-Bumpsadese Silver badge

          Re: BBC are naming names

          I think they're an Ericsson shop, but curious how Telefonica's managed to engineer a single point of failure.

          If you go back in history long enough, you can trace O2's heritage to BT. I suspect that the culture of being institutionally s**t is in their DNA.

        3. FuzzyWuzzys
          Happy

          Re: BBC are naming names

          "suspect a bit of planned maintenance didn't go quite as planned. And neither did any plan to rollback to undo it."

          So Ops were given the Change Request at 9pm last night, hurriedly signed off by a manager ( now negotiating his golden handshake to leave I might add! ), they accidentally copied the dev creds to the Jenkins cluster and flooded the network with pre-UAT code pulled from the wrong Git repo branch! Ha ha!

          1. J.G.Harston Silver badge

            Re: BBC are naming names

            Yeah, my first thought was: just roll back to the previous install.

            1. Jellied Eel Silver badge

              Re: BBC are naming names

              Depending on what the update/change did, it's sometimes not that simple..

              So a loooong time ago, I experienced a Cascade(ing) failure. They were a trusty and popular frame-relay & general data switch used by a lot of ISPs and telcos. So there was a sofware update. We tested in the lab, and all was fine. We implemented it.. and it found a back-up control card, switched that to primary, but didn't connect properly to the management database, and corrupted that. Rolling back got the switches back under management, but (I think) left updated microcode so the backup database wasn't compatible.. So FUN!

              Luckily(?) we had a backup-backup from doing an SQL dump of each switch's config to trusty plain text files so we could manually rebuild the database. Which took about 48hrs.. Luckily it didn't interrupt services for that long, just delayed any MACDs until we'd got everything back in sync. Early versions of Cisco's config management created similar FUN! situations.

              I suspect Telefonica's issue is much the same, ie update broke state, and given it's size and number of devices/users, much more painful to get back under control.

        4. silks

          Re: BBC are naming names

          Yep, software upgrade maybe?!

  6. The Real Tony Smith

    Sorry

    My fault, switched from TalkTalk to O2 this week, must have been something I fiddled with...

    1. silks

      Re: Sorry

      Moving away from TalkTalk = smart move. Moving away from TalkTalk to o2 right now - not so much :(

  7. Wolfclaw

    Latest update it's a global software fault that is not just for O2.

  8. ItsMeDammit

    Outages get blamed for everything.

    My favourite Twitter post (courtesy of Google News, funnily enough - I don't use Twitter) blamed the outage for them going to be late getting to a meeting this morning.

    So - did the outage locally stop time as well, or were you just too busy messing about on your phone to get up and go to work ?

    Personally, my issue with the outage this morning was that my phone decided to generate a notification telling me that there was no data at my location at 05:38 which woke me up. So I for one got to my first meeting early...

    1. werdsmith Silver badge

      Re: Outages get blamed for everything.

      "I'm going to have to buy a satnav because I'm already late for my first meeting - because I can't find the place".

      -giffgaff user this morning.

      1. Martin-73 Silver badge

        Re: Outages get blamed for everything.

        Satnav function on my phone worked fine, the only bit that didn't was the traffic status which COULD explain the 'being late for a meeting'.

        As it was, my luck ran the other way. I took the route less travelled, because it looked lighter on traffic from the roundabout... and got to work 15m early

  9. Steve Cooper

    Not often being on Three means I have more data throughput than others around the City of London :smugface:

    1. Soruk

      For those who rely on a data signal and are on O2, can do a lot worse than to get a cheap MiFi and a Three PAYG SIM on their 3-2-1 tariff.

  10. TRT

    People are suffocating...

    O2 deprivation.

    1. Korev Silver badge
      Coat

      Re: People are suffocating...

      People are suffocating...

      O2 deprivation.

      Ahh the Oxygen of publicity

  11. Whitter
    Mushroom

    How to get businesses to care?

    No bonus for any board member on any year with a major outage or security fail.

    Of course, the top floor parasites would just spend most of the year arguing about what 'major' means.

  12. adam payne

    O2 acknowledged the problem on Twitter at 07:00 GMT.

    Sadly all the people on O2 had no data so they couldn't get on twitter anyway.

  13. Mage Silver badge

    Quite expected.

    It will get worse and more critical. What is the worst? Having your business TOTALLY depend on internet connections (fibre, DSL, Mobile) or the so called Cloud.

    The biggest risk isn't a solar flare taking out satellites, or even global Nuclear War. Or cyberwarfare or criminal hackers. It's creeping monoculture, maybe eventually only 3 Eco systems. We have maybe only four major mobile infrastructure companies now, one is Chinese Gov owned (ZTE) and one sort of private Chinese (Huawei). Nokia ate Lucent/Alcatel, Siemens and Motorola Networks and there is Ericsson.

    Cloud providers and the OSes they use? Linux is fine, but maybe a patch pushed out by management late on Friday, maybe before a holiday. Might be for Servers, Edge Routers or both. Even MS uses Linux exclusively on some bits of their cloud.

    I wrote a post apocalyptic story with lots of mayhem and death. I decided it was too dark so wrote one with a fantasy setting set slightly in the future where all retail POS, cash machines, Mobile and fixed line billing and even a lot of SCADA relies on the Internet and a handful of Cloud Service providers (Renting space on someone else's remote server like 1960s). "No Silver Lining" Ray McCarthy. You can download 1st 20% free.

    All mobile, internet, cloud services etc WILL fail at the same time, sooner than later. How much retail, wholesale, SCADA (Traffic lights, Electricity distribution configuration, sewage & water pumps etc) now depend on it?

    1. Solarflare

      Re: Quite expected.

      I quite like satellites, I'll try not to take them out :)

      1. JoshOvki

        Re: Quite expected.

        > I quite like satellites, I'll try not to take them out :)

        Not even for a swift pint which inevitably results in them falling over?

  14. Dabooka

    Do people really need reminding buses are still running?

    Are folk that dense to assume no buses are operating because an app doesn't update? Hang on, I think I've just answered that one myself.

    1. David Nash

      Re: Do people really need reminding buses are still running?

      I don't think it's just the app, but the automatic ETA signs at the bus stops. The theory being, if the sign doesn't show any buses are on the way, perhaps no buses are on the way.

      However on main routes in London you can usually just open your eyes and look up from your phone and notice several buses in any direction you care to look.

      1. BinkyTheMagicPaperclip Silver badge

        Re: Do people really need reminding buses are still running?

        My heart bleeds. Chance would be a fine thing for decent bus prediction times Oop North. Especially late on Saturdays some buses just choose not to turn up and I have to revert to taking the slow way home. I would take a train, but the RMT are on strike every Saturday.

      2. FrogsAndChips Silver badge

        Re: Do people really need reminding buses are still running?

        This morning the ETA signs showed a message along the lines of "traffic information unavailable, please check at tfl.gov.uk", so it was quite obvious that buses were still running.

        1. Anonymous Coward
          Anonymous Coward

          Re: Do people really need reminding buses are still running?

          Just think how much in data charges the likes of TfL are saving...!!!

          (Anon because I work for one of their ETA sign suppliers...)

        2. the hatter

          Re: Do people really need reminding buses are still running?

          Except the tfl website also didn't know where the buses were, because the buses themselves also used o2 to tell control where they are.

    2. SVV

      Re: Do people really need reminding buses are still running?

      On the outskirts of the capital, when the board says 1 min, it means the bus will show up in about 5 minutes time, so you learn to ignore them and wait calmly til it eventually shows up, hoping there's room to get on.

      Upside of outage : bus not full today of people gormlessly flicking through Twitter and Facebook.

      Downside : bus even more full of people yapping into their phones, all complaining that Internet's not working. May the person who thought of unlimited call plans spend eternity on a crowded rush hour bus in my London suburb.

  15. Mike Shepherd
    Meh

    Surprising

    I use an O2 reseller so we can keep a remote eye on a vulnerable elderly relative who doesn't have wired internet. So that's not working today.

    Perhaps it's going too far to install a backup connection there, but it's surprising to read that large organisations depend on a single supplier (and that the supplier neither tests software enough before its publication nor has a mechanism quickly to revert to what worked).

  16. Herring` Silver badge

    Hang on

    Isn't there still some super plan to put the emergency services onto 4G? Has this been thought through?

    1. silks

      Re: Hang on

      Yep, I imagine EE and the Government are holding a committee meeting right now :)

  17. Anonymous Coward
    Anonymous Coward

    On the other hand

    AQL's MVNO services were available since exactly this outage picked up pace, while I've had API problems for the last 2 days (they use Three who also have a spike on down detector but that's anecdotal I imagine)

  18. wolfetone Silver badge

    I'm on Tesco Mobile, my phone has no internet anyway so I'm not too bothered about that. But I haven't been able to send a single text message all morning, including now at 12:35pm.

    I'm outraged etc etc. Secretly I'm glad though as I don't have to talk to anyone and can actually do work.

    1. wolfetone Silver badge

      An update 20 hours on:

      O2 say they've fixed the problem and it's all back to norman, but I'm still sending texts and I'm still getting messages saying the texts weren't delivered or sent. So yeah, clusterfuckarooney.

  19. Alister

    I reckon they were trying to remove all the Huawei kit from their backhaul on the quiet...

    1. Dr Who

      An upvote for that but my theory is it's chinese hackers because everyone is removing their Huawei kit. Hua Wei and the Art of (trade) War.

    2. Anonymous Coward
      Anonymous Coward

      Madness!

      Imagine thinking you can quickly whip up some meatballs when everyone just wants sweet and sour pork like last week.

  20. Gricehead

    Awkward

    https://jobs.ericsson.com/jobs/265037?lang=en-us

    1. Flywheel

      Re: Awkward

      Would that be "didn't have one before but really ought to" or "X has left - replace them ASAP" ?

    2. cantankerous swineherd

      Re: Awkward

      so the finger points at the n00b?

  21. anthonyhegedus Silver badge

    I love the way they immediately set the “blame path” to “third party suppliers” and then try to disperse the pain and further divert the blame by saying that other networks are affected throughout the world.

    Then they advise you to use WiFi.

    Why not show a bit of humility and have the outage posted front and centre on their main web page?

    Anyway I’m surprised they haven’t blamed brexit!

    1. TonyHoyle

      So either:

      The third party suppliers, large enough to supply a company the size of O2 with significant infrastructure, doesn't roll out new updates to a test network first and doesn't have a rollback procedure in the case of emergency, in which case O2 picked an incompetent supplier.

      Or O2 doesn't have the above (and they should, even if the supplier already does it.. you never trust new builds until you've validated them internally), and they're incompetent.

      1. tfewster
        Facepalm

        Don't be nasty, of course they have a Test network.

        Some companies have a separate Production network as well.

        1. Anonymous Coward
          Anonymous Coward

          Not so simply

          It's not a case of not testing, it's a case of you can't test enough. I used to managed core network upgrades for another mobile operator. The vendor would test the software for 100s of hours with 10,000s of test cases, then we would test in our own labs for weeks.

          No matter how much testing was performed there was always a bug/service affecting issue found within hours in the real world. The unique combination of users, features, integrations, traffic load etc make it near impossible to find every issue before piloting.

          It was always a question of when would we find it, and how career limiting it would be

  22. Stevie

    Bah!

    It's a good job you guys have that better and faster interwebs you were talking about last week, when you downvoted me for mentioning the small matter of provider uptime.

  23. a pressbutton

    Nigel from O2 on DownDetector

    Cracking...

    ... and his handle is N-O2

    aka Laughing Gas.

    1. Anonymous Coward
      Anonymous Coward

      Re: Nigel from O2 on DownDetector - aka Laughing Gas.

      Obviously no chemists around.

      Laughing gas is N2O. NO2 is extremely toxic. It's what they add to nitric acid to make RFNA, the other half of some rocket fuels.

    2. Anonymous Coward
      Anonymous Coward

      Re: Nigel from O2 on DownDetector

      Well, it would be funny if NO2 was laughing gas. It isn't. NO2 is nitrogen dioxide an nasty brown gas that dissolves in water forming Nitric Acid, very nasty stuff. Laughing gas is nitrous oxide, N20.

  24. James Anderson

    Whats with all these Spanish owned comapnies in the UK.

    Telefonica operate a very reliable service in Spain (albeit with crap customer service).

    Sabadell in Spain manage to keep their computer systems up and their customers happy.

    AENA run some of the best Airports in the world (Badajas by Madrid is superb example of how to do an airport right) -- but the UK gets Heathrow and Gatwick.

    If you gave them Gibraltar back would they leave you alone?

    1. WolfFan

      Wrong solution.

      Right solution? INVADE. And show the cheeky buggers who's boss!

      "Come cheer up, me lads, 'tis to GLORY we steer, to add something more to this WONDERFUL year".

      1. James Anderson

        Confidentially as a member of an elete military unit cunninly disguised as Drunken Revellers and/or greey haired pensioners I can say the invasion is well under way from the Island Fortress of Ibiza and the mainland stronghold of Benidorm we have established secure beach heads on most of the Mediteranean coast (inland areas being deemed of no strategic importance).

        We are currently instructing the natives on the Joys of British culture -- Pints of Stella, lethally hot curry, shopping at Aldi, etc. -- which they have taken to quite well, although queuing and how to make a left turn at a roundabout needs a lot more work.

    2. Anonymous Coward
      Anonymous Coward

      "Whats with all these Spanish owned comapnies in the UK."

      Revenge for 1588. (Napoleon got the blame for 1805).

      1. Anonymous Coward
        Anonymous Coward

        Re: "Whats with all these Spanish owned comapnies in the UK."

        Royalty

  25. Terje

    Given that quite a few networks around the world use Ericsson equipment and not every network seems to have fallen over quite this bad, I would guess it's something to do with network size / design that aggravate the issue for some, if that's the case it's probably not that easy to catch in a test environment. :(

    As I'm Swedish I'm contractually obliged to try and protect Ericsson...

    1. Ken Hagan Gold badge

      Whereas we Brits (*) are contractually obliged to slag off the home team.

      (Whingeing Poms, I believe is the preferred expression.)

  26. Anonymous Coward
    Anonymous Coward

    My work phone is on O2!!

    OMFG the world will collapse!

    do I connect it to some wifi? or do I just ignore it... well for once I have a totally legitimate reason for not giving a stuff about email! long may the outage continue!!!

    1. JoshOvki

      Re: My work phone is on O2!!

      Depends if your local has WiFi. Seems like a legitimate reason so setup camp.

  27. Anonymous Coward
    Anonymous Coward

    First Youtube was closing down, now the whole Internet has!

    I can't imagine what all those snowflake teenagers threatening suicide because they think youtube is closing down must be thinking right now . . . . . RIP the Internet.

    1. MrMerrymaker

      Re: First Youtube was closing down, now the whole Internet has!

      What a surprise, someone having a go at youth, using the alt right term Snowflake, doesn't have the sheer balls to post non-anon!

  28. Rabbit80

    It strikes me that if all the mobile operators in the country cooperated and allowed customers to roam each others networks this kind of problem - as well as the sh1te signal issues that plague the country could be pretty much non existent.

    1. Terry Barnes

      Cascade failure. What do you think the impact of suddenly having 33M devices roam onto your network might be? How much will it cost to size your network to be ready to take millions of extra users at a moment’s notice? Who would ever build our rural cells when you could just let your users roam onto the network of the mug who has?

      1. Intractable Potsherd

        Rural cells are being built?? Citation needed!

        1. Anonymous Coward
          Anonymous Coward

          Rural cells are being built?? Citation needed!

          There's one on the bridge at Bradford on Avon, and there's one in Trowbridge.

          -oh you meant mobile telephone cells.

  29. Anonymous Coward
    Anonymous Coward

    tales from next month

    Will be interesting if dumb comment and downright lies drops over the period of the outage. Then we know which 32 million to blame.

    Next feeble economic figures will be blamed by the tory twats on this no doubt like when its either too sunny or too rainy.

  30. Anonymous Coward
    Anonymous Coward

    Huawei arrest

    Coincidence or not?

    1. Derezed
      WTF?

      Not the US

      What? A Chinese citizen arrested in Canada on a US warrant? I think Brexit is a more likely culprit.

  31. Joseph Haig

    This is awful

    I have to talk to people face-to-face! The horror!

  32. Wandering Reader

    Not really a crash

    It's not really a crash. With the tumblr porn filters on the way, it turns out that 4G just isn't needed any more.

  33. Jeff 11

    So their Ericsson appliances/applications have somehow failed.

    Perhaps they should have avoided this single point of failure for such a critical piece of comms infrastructure (using redundant bits of kit from the same supplier doesn't work if all instances of it are affected by the same issue).

    If other operators around the world are affected then it sounds like a botched update, or a very well coordinated cyberattack. Or perhaps they're all using some cloudy service and THAT has gone down?

    1. Anonymous Coward
      Anonymous Coward

      I just heard O2 have hired Paul Pester to sort this out

  34. Wingtech
    Facepalm

    Oh, the fallability of the BBC!

    In their report about the failure, https://www.bbc.co.uk/news/business-46464730, they quote a lady in Norwich

    "I'm disabled ... I'm in a wheelchair," she told the BBC. "So having no data but also no calls as well means I can't contact anyone if I have a fall or if I need anything."

    It makes you wonder if they used telepathy to contact her.

    1. Derezed
      WTF?

      Re: Oh, the fallability of the BBC!

      Your point appears to be that if someone comes and takes out your toilet, you'd be happy shitting out of your windows because, well, we all did in the good old days didn't we?

      They might have used this thing called "email" she used while at her home (sat at a device known as a Personal Computer or "PC"...or even a device called a "tablet" connected to a magical device called a "WiFi router").

      You can't just scoff and pretend that people don't rely on technology for certain things or suggest that because you're happy without quick access to porn that everyone else is just dandy too.

      1. werdsmith Silver badge

        Re: Oh, the fallability of the BBC!

        They might have used this thing called "email" she used while at her home (sat at a device known as a Personal Computer or "PC"...or even a device called a "tablet" connected to a magical device called a "WiFi router").

        Do you not see the problem with this reply?

        1. Derezed

          Re: Private Eye always has

          Do you not see the problem with this reply?"...

          ...not unless the form factor of a laptop or desktop got miniaturised recently...

          Internet != Telephony...maybe she doesn't like WhatsApp...

    2. MrMerrymaker

      Re: Oh, the fallability of the BBC!

      Where I work there's no signal at the best of times but data, somehow, usually anyway works. So my relatives are on whatsapp.

      You're doing the annoying old thing of assuming your situation is the only one that matters and all others are either the same or you simply don't care

  35. Doogie Howser MD
    Thumb Up

    Lyca Customers

    Time your networked moved off O2 and merged with Virgin Mobile. Then we can finally have a network called LycaVirgin.

    1. WolfFan

      Re: Lyca Customers

      The Catholic Church will object, they own that trademark.

    2. Richard Gray 1
      Coat

      Re: Lyca Customers

      In touch for the very first time?

  36. Simon Harris
    Pint

    Can't chat on Whatapp and Facebook?

    Try the off-line alternative, and go to the pub.

    1. Aladdin Sane

      Re: Can't chat on Whatapp and Facebook?

      But there are people there!

  37. Captain Hogwash

    In unrelated news....

    https://www.theregister.co.uk/2018/12/06/uk_gchq_bulk_equipment_interference/

  38. Laughing Gravy

    Still out for me at 4pm, Oh well, better get some popcorn popping

  39. Rod 6

    I have no network or data in Nottingham. Emergency calls only... impressive levels of ineptitude...

    1. tfewster
      Facepalm

      Or alternatively...O2 have maintained essential Emergency services even though their nice-to-have services have been down?

  40. Daveycoder

    Other mobile operators around the world are also affected?

    ...according to the O2 status page. Why would that be? Has everyone upgraded/changed/implemented whatever has broken the system at the same time...or is everything being routed through a single point?

    Seems a bit odd to me.

    1. tin 2

      Re: Other mobile operators around the world are also affected?

      also not heard anything about any other mobile operator in the world having similar issues. maybe searching would not reveal, but seems odd.

      1. James R Grinter

        Re: Other mobile operators around the world are also affected?

        SoftBank did. Presumably they were running one of the old software versions too.

    2. tim 13

      Re: Other mobile operators around the world are also affected?

      Would appear to mainly Japanese network Softbank are the 'others'

  41. Anonymous Coward
    Anonymous Coward

    2018/12/06 T 16:20 - "Emergency calls only" here in Cambridge

    So no nothing.

  42. Anonymous Coward
    Anonymous Coward

    Expired certificate!

    Apparently the root cause was an expired certificate....

    Press release: https://mb.cision.com/Main/15448/2694397/959040.pdf

  43. Ed 11

    This is terrible. I had to get cash out at lunch because the Korean fried chicken stall I frequent in Farringdon was unable to take cards owing to this outage. Oh the indignity of it all.

  44. Flywheel
    WTF?

    Ericsson - they may well have found the problem

    Ericsson customers affected were using two specific software versions of the node “Serving GPRS Support Node - Mobility Management Entity” (SGSN-MME).

    “An initial root cause analysis indicates that the main issue was an expired certificate in the software versions installed with these customers,” it said.

    According to Reuters....

  45. anthonyhegedus Silver badge

    OK so now their status reads:

    "The network issue continues to be a top priority for us and we apologise to all our customers impacted by this.

    Earlier today we suspended any planned works or updates across our systems and network. This allowed hundreds of people from both our technical teams and our third-party supplier’s teams to focus on fixing the problem, helping us to restore data services as quickly as we can which will also help reduce the high network demand we’re currently seeing for voice calls."

    Hundreds of technical people? Doesn't smack of a software problem, or at least not a simple one. It sounds like a problem that requires a lot of manpower, like manually adjusting settings at thousands of different sites.

    Today lots of things are going down for me and people I know, and we've actually had customers phone up and ask if it was due to the O2 problem!

    1. koswix

      Sounds more like PR spin to me. Can't do any planned maintenance as the network is down, so suspend those jobs and you indeed now have 'hundreds' of technical staff available to focus on the outage... Even if they're doing nothing, they're *focused* on doing nothing.

  46. The Specialist

    999

    Not sure anyone mentioned before but dialling 999 / 112 from any mob regardless of network / with or without sim card is possible. It will latch on whatever network is available.

    1. ARGO

      Re: 999

      Only part of that works in the UK. Although you can dial 999 without a SIM card, UK networks will not connect the call. This is by order of OFCOM as there were large numbers of untraceable hoax calls when it was allowed. (with a SIM card - even an expired one - it all works as standards intended)

  47. MrFacePalm

    Another result for the beancounters

    If we want to start finger pointing, first place to look is O2 probably deciding to push the envelope by continuing to use end of support SGSN software (=speculation on my part), and based on advice from an undermanned Ericsson local support team based in the UK. The second place to point the finger is to the bean counters in Stockholm who have been offshoring work to China and India for years as they plot their voyage to the bottom. (=hard fact)

  48. Laughing Gravy

    6pm and still out, having some beer with the popcorn now

  49. cantankerous swineherd

    giff gaff a lousy advert for a network run by me. luckily gg tell me some of my colleagues are engineers and they're all over it.

    Ericsson not a good advert for Erlang it seems.

  50. Trev 2

    OMG never using O2...oh hang on!

    Chav version:

    "OMG never using O2 again...oh I just signed a 600 month contract for my fancy overpriced iPhone X so I'll never use O2 again, after the year 2068!" ;)

  51. Jeroen Braamhaar

    Related ?

    In Japan, SoftBank's mobile network also went down for 4 hours .... allegedly due to Ericcson sfotware going titsup as well ...

    (Caught this on NHK World news)

  52. Anonymous Coward
    Anonymous Coward

    Maybe O2 should be using Huawei. But hang on......

  53. tomthemac

    Ericsson blame 'expired certificate' in software version

    wow.

    expired certificate.

    yep.

    https://www.ericsson.com/en/press-releases/2018/12/update-on-software-issue-impacting-certain-customers

    1. steelpillow Silver badge

      Re: Ericsson blame 'expired certificate' in software version

      "the main issue was an expired certificate in the software versions installed"

      Can it be that carrier-grade installs don't get updated by operators as promptly as the run-of-the-mill swearware does?

      Whoever woulda thunk...

  54. Anonymous Coward
    Anonymous Coward

    Pretty embarrassing @MarkEvansO2...

    In your attempt to effectively name & shame by mentioning the third-party software company responsible for your national data outage, you spelt it 'Erricson'... whoops

  55. Anonymous Coward
    Anonymous Coward

    MME issue

    It's an issue with their MME implementation, provided by Ericsson. Source: been on calls about it all day because of the impact it's caused us.

    Happy googling, armchair service continuity managers.

  56. Anonymous Coward
    Anonymous Coward

    It's not just transport systems that it's effected. It has also hit emergency services. The trust I work for has lost connection of all the ipads that are used for patient report forms. This is extremely worrying seeing that every emergency service will be using a 4G network for the entirety of their critical communications. This outage would've put lifes at risk if ESN was live!

    1. Inventor of the Marmite Laser Silver badge

      Your ability to spell

      Seems to be affected

    2. Anonymous Coward
      Anonymous Coward

      "This outage would've put lifes at risk if ESN was live!" - Nope

      ESN is EE not O2.

  57. Anonymous Coward
    Anonymous Coward

    Latest theory

    They were getting ready for Brexit by installed radio equipment that uses imperial wave-lengths instead of the current metric wave-lengths.

    1. Anonymous Coward
      Anonymous Coward

      Re: Latest theory

      Sounds legit.

      Are we going to buy that platinum-iridium kilogramme from the French and rename it the Imperial 2.204 Pound? Then if we can just find twelve yeomen coming out of church on a Sunday we can get our standard foot back.

      Just in time for a number of large corporations to stick it right in their mouths.

  58. michael mackey

    Should have bought Huawei !

  59. Anonymous Coward
    WTF?

    I don't mean to be undiplomatic...

    But why is Britain the country that is always having these outages. You can't use your cellphone. You can't buy train tickets. Your online banking is down. Etc. Ad nauseum....

    1. DoctorMO

      Re: I don't mean to be undiplomatic...

      Is it because GCHQ has hired all the boffins away from industry? Is is because open source code that's made in the USA is first used in France and Germany and visa versa and takes years to land in blighty? Is it because IT managers in the UK are trained by Black Adder impersonators? Is it because we have abut as much respect for competent engineers as we do the guy that empties the bins? Is it perhaps because computers don't run on steam and we've been baffled by electrons ever since the end of the nineteenth century?

      All I know is, I escaped the UK and found more respect, more money and more freedom to be a good engineer in the USA. It doesn't sound like things have gotten any better since I left.

  60. Laughing Gravy

    8:30 pm and still out, on the vodka now

  61. Tim Kemp

    Using a multi-net SIM here. Feeling smug today.

    I wonder how many people finding voice calls affected are trying to use VoLTE?

  62. dank_army

    Sorry chaps I don't buy it - a software update causing global issues? Smells like a hack to me.

    1. Anonymous Coward
      Anonymous Coward

      Your confidence in O2 is entirely misplaced...

  63. MrMerrymaker

    It's back for me but not for friends

    Ah well. Hope I get a free day or something though can't complain for one day's outage when I use giffgaff for £7 every month without issue

  64. Weltschmerz

    This is cyber warfare isn’t it?

    China. Huawei. Etc.

    1. diodesign (Written by Reg staff) Silver badge

      Re: This is cyber warfare isn’t it?

      Expired software certificate, apparently...

      https://www.theregister.co.uk/2018/12/06/ericsson_o2_telefonica_uk_outage/

      C.

  65. SteveTM

    Lots of people outraged about people being outraged about people being outraged.

    Nothing new here. Apparently the last bastion of Britishness, being able to complain, is being removed in exchange for lethargic illogical tolerance.

    Imagine if the company in question has 6.5 BILLION in revenue and the supplier to that company had over 200 Billion in revenue....yet the fact that software in SGSN's worldwide was due to expire and shut everything down wasn't detected or dealt with prior to that expiry date. Stupid idea coding expiry into an SGSN anyway.

    Also, for all those complaining about people complaining, its not just people struggling to get onto Snapchat and Facebook, a lot of key services use the O2 network, because of its claimed resilience. The fact that there wasn't some degree of fault tolerance built into the O2 network is poor show considering their operating budget.

    Complaining may annoy people, but its those people complaining which have pushed providers to improve their resilience, so you should be thanking them everyday for reduced power cuts, improved mobile coverage and people being held accountable.

    The moment we go quiet and stop complaining is when we should start complaining.

  66. Anonymous Coward
    Anonymous Coward

    Well, at least the cert expiry logic worked flawlessly ....

    .... and the individual who kept the reminders 'in a spreadsheet somewhere' left O2 and/or was offloaded to an outsource supplier years ago.

  67. Anonymous Coward
    Anonymous Coward

    Highlights some pretty poor support at GiffGaff though.

    I had raised a support ticket for a completely unrelated problem with GiffGaffs "agents" the night before the system went TITSUP. I was surprised to get an update message from them during the outage and duly went to see what the resolution was. The message was telling me that they had a major problem. It looks like they decided to send the same message to every case raised regardless of whether it was related or not.

    The community support pages seem to be populated entirely by copy and paste bunnies trotting out the part line or commentards who try very hard to look like they have a clue but succeed in proving they don't.

    Sadly the only network signal I can get in the office is O2 or Vodafone and I left Vodafone some time ago due to their costs and crap customer service.

  68. jms222

    giffgaff

    > community support pages seem to be populated entirely by copy and paste bunnies

    I agree entirely.

    By the way have you tried fiddling with your APN like you don't have to do on any other network ?

    Or restarting your phone like you don't have to do on any other network ?

  69. Keith Tayler

    Keith Tayler

    A Swedish company is responsible. Let us hope it makes the Swedish government think again about making a cashless economy within two years.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like