Vodafone keels over, cutting off millions of mobile and broadband customers Vodafone fell over in the UK this afternoon, with Register readers reporting that many services including mobile coverage, internet services, and even the company's own status page went down. The outage began on Monday at 14.25 BST, and 30 minutes later it peaked when monitoring website Downdetector.co.uk reported that almost …
20 rappen? That's Swiss, isn't it?
The Swiss had one of the most clever tricks I have seen with coins ever: their 50 'cents' (aka rappen) coin was very small (much smaller than the rest), so it was perfectly suitable for parking meters and stuff - a meter could hold a load before they were full up. Smart.
One year ago I stayed in a Devon AIrbnb that had a thatched roof, an exactly 6 foot high lounge ceiling, fibre to the premises internet... and a coin slot electricity meter! Actually as it was summer a few £ coins lasted quite well for hot water and a bit of cooking. Perhaps the owner of the place was keeping that antique mechanical slot meter - 40 years old at least - as a character feature.
My coin op bedsit of yore took 5p coins not 50p. And I found the landlady was overcharging, she had to have the meter readjusted.
OK, both their mobile and broadband networks went down at the same time suggesting shared infrastructure, OK their status page went down as it's probably connected into the back end, OK their web site went down later probably due to load, OK their phone lines went down as they're probably using VoIP by now, OK their social media team went down as they're probably on the same network and no one is allowed to use another device or whatever... OK it seems to have affected other countries and networks as they share infrastructure, OK people that pay for a backup line got nothing as it's the same back end, OK their press comms was extremely basic and uninformative as the key people were probably too busy to send out clear messages.. but... why do they seem to have single point of failure affecting the whole country?
I think you'll struggle to find a public or private system that doesn't have *some* level of SPOF within it. It might be known about, it might not, but it'll exist. As you'll see from many comments in here, people may well have had "backups", but if they unwittingly shared Voda infrastructure, that too is a SPOF.
And there's not much to protect yourself around a botched BGP update.
If you work in IT, the redundancy is in your dual SIM phone. Just as you'll have two credit cards with (genuinely) different banks using different payment card service providers and cash in your wallet. And you have mobile internet ready to go when your fibre connection fails. And you have a second favourite pub for when your first choice unexpectedly goes out of business.
My broadband ISP Onestream also fell over as they also use Vodafone for their infrastructure. DHCP was still providing my router with an IP address which would change after a few reboots but traceroute wouldn't get any further that the gateway, and pings to anything out on the internet would result in 100% packet loss.
It back online now, DNS was a bit flakey at first but seems okay now.
Cloudflare announced IP address space
By the look of it, anyway, and that also included their own nameservers.
This post has been deleted by its author
Fiber optic AND cellular went down. I just had time to stroll to the Vodafone store where the staff told everyone it would only take 1 hours and was due to the engineers "merging the 3 and Vodafone infrastructure". 5 minutes later the team turned the lights off, closed the doors and went home.
By the time I set up my backup cellular it was back on.
Still, is a 50% monthly discount for an all Vodafone package still worth it?
The problem with all anyone package is straight away there's potential* for both failing. True, diverse sourcing doesn't automatically deliver resilience, but with some knowledge of the underlying businesses it makes it less likely.
*I know, in an ideal world they would be entirely separate, but I think this proves they often aren't.
We've all had an "oh sh*t" moment, you haven't done IT engineering if you haven't, just not all of us take a major ISP offline.... i did take an entire National Insurance's website offline for a period once though... something related to 1066. It wasn't because of BGP however!
Annon for obvious reasons
They already kinda did.
"This was triggered by a non-malicious software issue with one of our vendor partners which has now been resolved, and the network has fully recovered. We apologise for any inconvenience this caused our customers."
Sounds supciously like a bad patch or firmware update from a provider that did something unscheduled, in a place and time it possibly shouldnt have
"This was triggered by a non-malicious software issue with one of our vendor partners which has now been resolved, and the network has fully recovered. We apologise for any inconvenience this caused our customers."
basically they had one of their outsourced teams do some work on the system responsible for their BGP peering and something went badly wrong.
why it took so long to roll back & why it happened during our day needs answering.
as a VF customer years back we where implementing new WAN circuits and their converged voip solution, submitting requests for the voip stuff their offshore teams would ask me for details to complete their change request forms, instead of getting piecemeal requests I'd have them send me their form & I'd fill it out and send back to them, instead of 2 weeks of back & forth before they'd agree it I could get it done in 1 day. They were happy as they didn't have to do it & our customer was happy as their project got back on track.
I realised then how vulnerable VF was to their offshore teams making mistakes.
next job in a largish retailer, VF where the sole network provider, I did query why we didn't have 2 different providers given the risk of 1 provider having a major issue etc.
they went down hard yesterday because of reliance on a single vendor.
its not difficult to use multiple service providers in 2025.
This post has been deleted by its author
It connected to the net but no data was forthcoming, the system reporting a DNS resolution failure. As it wasn't my end, I did the sensible thing and made a cup of tea.
Tech is less resilient because it is complicated. A single component failure/unwise tweak can take anything down. This stuff will happen.
…A reminder for us domestic users to address our own single points of failure. Keeping our cellular and broadband providers separate being one sensible move. As a Vodafone mobile customer I was getting a bit fed up with them spamming me with their broadband cross-sells. I’ll be even more fed up with these now. Vodafone management will be jumping up and down and looking for heads as the broadband initiative looked to be a key part of their strategy.
SS
Am looking forwards to the Who Me? Posting that explains what actually happened. Probably need to wait a few years for this to emerge to allow time for the heat to die down and people to change their jobs.
Interesting that the day after this outrage the NCSC are speaking about balancing the need to plan for using pen and paper if 'hackers' take systems offline. Perhaps this needs to be extended to cover SNAFU and Management Interference Events.
"This was triggered by a non-malicious software issue with one of our vendor partners which has now been resolved, and the network has fully recovered. We apologise for any inconvenience this caused our customers"
https://news.sky.com/story/vodafone-reveals-cause-of-massive-internet-blackout-amid-calls-for-compensation-13449877
Now that Ofcom has approved the merger of Vodafone with Three we can look forward to these outages affecting over half of us instead of about a thid of us. Gee thanks, Ofcom. Fortunately yesterday it didn't affect Three customers - but I'm sure the engineers are working on a more effective merger.
"Readers also reported that broadband was affected by the outage, which is odd since we would have expected cellular and internet connectivity to be largely separate."
I don't know why you would have expected that - companies are all about "synergies" and "efficiencies", I'd 100% expect them to be sharing as much kit as possible.
VodaFone client for home, retired now, but work was also VodaFone
for me, lost interweb access
pull up CMD <3
ping to my router, all good
ping 8.8.8.8 fell over
tracert to 8.8.8.8 stopped at my router too, guessed there was an issue with VF themselves
did the classic, powered off router / fibre hub - restarted, waited
in their defence, it's the first time I have NOTICED they went off
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2025
