BT at last coughs to 'major outage' after broadband went titsup across UK on Sat BT was hit by a huge DNS outage on Saturday morning but the telecoms giant was very slow to respond to customer complaints, it has been claimed. A DNS flaw downed BT's network across Blighty, according to anecdotal reports on Twitter. But the one-time state monopoly was very sluggish to respond to gripes from subscribers who …
I'm glad there is one person on the internet that knows how it works. Now if only we could get you into BT, Verizon, et al....
That's a bit harsh.
I suspect BT and Verizon (since they were named) have quite a few people who know rather well how internet works. After all for example Verizon acquired WorldCom who in turn gobbled up MCI which had purchased UUNET. Quite a bit of knowledge on how internet works there even if some employees were lost along the way. Whether anyone in management, marketing or beancounting listens to the ones in the know is another matter entirely.
I remember back in the day when pipex was registrar for .co.uk and someone saved the 1ary file off (in vi) and accidentally performed a 'delete to end of file and save' command :D
It only took just over 4 hours for the missing half of .co.uk to come back, but in those days there weren't so many.
or at least wasn't *just* a DNS issue.
I got called into work as our primary BTNet wasn't routing data, as wasn't the backup BT Infinity.
We have our own DNS servers, failover is Google and no pages loading.
Traceroutes to loads of sites were timing out within BT.
There looked to be a major routing issue, and even the tech support line for leased lines was so busy it was "leave a message".
Seven years to the day that they tested Phorm (and lied to their own customer service department about what was going on too if memory serves). Their own internal documents referred to the whole business as a 'stealth' trial.
I wonder what has really been going on here?
Yup, though as lots of people on social media were trying to be clever and tell others it was a DNS issue, I fully expect that if BT do cough to the root cause they'll blame DNS.
Was sat troubleshooting the loss of connectivity, and it looks like a couple of interfaces went down on a box near Telehouse based on comparison of traceroutes before and after. Possibly elsewhere too, but that's what we were seeing.
BT's DNS servers, inevitably, are the other side of that hop, so whilst there was an issue reaching BT's DNS, it wasn't _the_ issue.
Email worked, all the forums I use regularly worked, the load of crap my daughter looks at, didn't work.
So effing what if twatter is dead or something called tumbler was broken - who cares!
The friendface ripoff can't remember if that worked or not as I don't use it.
I could check my emails and read posts on fixing cars, what else do you need? (There must have been some porn available)
"Email worked, all the forums I use regularly worked, the load of crap my daughter looks at, didn't work...." Same her, except the crap wifey wastes bandwidth on (including Twatter and Faecesbook) were working fine, which suggests a localised routing issue and not a BT network-wide routing issue.
>> Email worked, all the forums I use regularly worked, the load of crap my daughter looks at, didn't work
Which is quite interesting in itself. Facebook, Twitter, Apple etc. all failed to work. But BBC News etc. carried on working fine. Very spooky! What are they preparing for?
When the router can't be persuaded to give out Google DNS, I go the extra mile and tell my computer to not use DNS information from DHCP and just use Google. Actually, encountering BT DNS (or any other ISP DNS really) on a new computer promptly makes me take a detour to the settings and switch to Google. BT and other ISPs seem to think they're doing you a service when they break DNS hierarchies and ignore that DNSSEC exists.
Not being a BT customer my first inkling of the problem was a BBC radio news item. It included an inane interview with a (typical?) BT customer who said he and his wife and a laptop AND a tablet and BOTH wouldn't work. "I mean it could have got me bank details 'n all!"
Taps run, toilets flush, and the internet works. There you are, that's your life sorted....
Our Virgin service at home was crap all day Saturday as well. Now it works, now it doesn't, frequent complete losses of connection for anything up to half an hour. Using Google DNS or Virgin.
I foolishly tried to phone them to see if there was a message about outages. Spent two and a half minutes entering account number, listening to options, entering characters from password and listening to that bloody irritating recorded woman's voice. Then got to "I'll pass you to a representative." immediately followed by "Sorry we're experiencing technical difficulties. Please call back later." Line then disconnected.
So at least that part of the Virgin Media 'experience' was as normal!
Maybe the cleaners unplugged the wrong thing at GCHQ.
Tried Google's DNS and openDNS and still had intermittent problems getting pages loaded. Google surprisingly was fine (on their dedicated fast line??) but click through to most other pages and it just dropped out.
Tried ringing BT a few times (since my area wasn't on the outage list when I finally did get it loaded via ssh tunnel to work), after a few minutes in the queue each time call either got disconnected or dropped (but still left connected) from queue. First class service as always. Thanks BT!
Certainly wasn't just specific to BT's DNS server (or however they resolve it via the homehub 5). I was unable to connect to many public DNS servers for the duration, but googles services did work which would suggest routing is at fault.
C:\Windows\system32>nslookup
Default Server: BThomehub.home
Address: 192.168.1.1
> bt.com
Server: BThomehub.home
Address: 192.168.1.1
Non-authoritative answer:
Name: bt.com
Addresses: 2a00:2381:ffff::1
193.113.9.164
> server 208.67.222.222
Default Server: resolver1.opendns.com
Address: 208.67.222.222
> bt.com
Server: resolver1.opendns.com
Address: 208.67.222.222
DNS request timed out.
timeout was 2 seconds.
DNS request timed out.
timeout was 2 seconds.
DNS request timed out.
timeout was 2 seconds.
DNS request timed out.
timeout was 2 seconds.
*** R
Call Central Services.
Most frustrating thing about the outage was if you looked at BT's status page all it said was that some customaers in a few areas of the country were having issues with their connection when it was very obvious from looking at twitter or downdetetctor that it was very widespread and effecting a lot of people!
I have used BT as an FTTC supplier for a few years now and they have been very reliable, couple of outages that have been short and very quickly rectifified so no real complaints about the odd outage but their status page really should be properly updated so people dont waste time cheking/changing their own stuff when its something BT need to resolve.
Dear Sir,
not knowing the full extend of these problems, 2 semi-nationwide areas being affected is the same as few areas being affected, seeing that 2 really is only a few... There were some people in those few areas of which BT knew that they had problems. All the rest (of the people) couldn't shouldn't wouldn't report the issue... So all in all, "some people in a few areas"...
No, I'm a not a spin-doctor, but I've seen so much crap coming out of Gov / Big Orgs that I kinda know what they're saying when they're saying stuff... Also when they're not saying stuff :D
Regards,
Guus
One of my clients rang up to tell me he had sporadic internet: Google worked, but none of the links beyond that would.
I've had this before when there is a billing problem: BT cut access to Port 80 until the bill is paid, so you can't access the web but your email still comes in and goes out. Google worked because it uses https.
Didn't get chance to rule out DNS completely. but google's DNS seemed to work fine for my client.
I have never relied solely on my ISP for DNS. Using your ISP as primary DNS is probably best from a performance POV, but why have the same provider for secondary (or even tertiary) DNS?
What amazed me about so many commentards on saturday morning was how many of them thought they were smart because they knew how to change DNS servers, but were still dumb enough only to point themselves at one DNS server.
What amazed me about so many commentards on saturday morning was how many of them thought they were smart because they knew how to change DNS servers, but were still dumb enough only to point themselves at one DNS server.
And were 'smart' enough to change DNS servers, and tell everyone else it was a DNS issue, whilst completely failing to take note of the fact that changing DNS didn't help with accessing quite a lot of sites. There was a hell of a lot of the blind leading the blind on the net on Saturday
When looking into it on Saturday I saw that there was two major routers with issues. One was a BT router in telehouse, the other was a Demon router (I don't recall the location, probably telehouse too). That might explain the virgin media issues some were having? I'm not sure who owns Demon now, but I thought they had some links to Cable & Wireless...
Anyway after getting a banned from CS:GO due to the network issues making me abandon a game, and figured out who to blame, I decided to go out and mow the lawn.
A couple of years ago when we had a major network fault in this area, BT Internet did not seem to have system to recognise that they were getting lots of complaints and the complaints did not seem to be passed on by the Indian call centre. Whatever you say to the call centre they just put you through the standard series of tests to prove your router is faulty which can take an hour so many people do not bother calling them. In that case the network monitoring did not detect the fault and they seemed to have nothing to detect a drop in activity on the network.
I run a ping test to internal and external servers and site, both from within work and without. Pings work -> internal (incuding vpn to datacentre) and external sites using URL and IP addresses all worked throughout. Pinging from home via Infinity to the externally available equivalents showed a mix of sites being found and not found, using both URLs and IP addresses (which absolves DNS issues?). reference sites like BBC were ok. This kicked off at 9:21, giving me the impression we had a problem in the office as I could not ping our London-based sites, but could ping the datacentre ones (Milton Keynes). I eventually remoted onto a PC in Swansea that uses TalkTalk - that had no issues with the sites I could not reach. My mobile phone connected fine as well with 3G (not so when using my WiFi connection).
I guess I changed my local PC to Google DNS just as it was fixed, so I'm not convinced as to what the real issue was.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2025
