BT broadband is down: Former state monopoly goes TITSUP UK-wide UK telco BT is suffering a major broadband outage: it appears the comms giant is caught up in a near-nationwide blackout, with more than 12,000 reports of service problems on Down Detector. The web connectivity monitor has been flooded with complaints from vast parts of the UK, from London and Birmingham to Manchester and …
This post has been deleted by its author
I have BT Infinity apparently working on the other side of the bridges from Bangor, LL.
I am accessing it remotely via Plusnet (aka BT Sheffield) from Birmingham.
Obviously both will stop working shortly, following this post.
Neither www.bt.com nor www.btplc.com are responding from Birmingham.
Only www.bt.com is reliably working from Bangor, btplc often gets a server timeout.
All data from tests in last 20 minutes. YMMV.
Anybody got a Dummies Guide to Disaster Tolerance going spare.
Probably no point asking them to comment, apparently the phones are dead, no reaching customer service, not a huge leap to think maybe they have no internet or email either. One might wonder if this is a denial of service attack, everything works right up to the CHAP authentication but the server never responds with either a success or failure message so after multiple tries a failure is assumed.
"One might wonder if this is a denial of service attack, everything works right up to the CHAP authentication but the server never responds with either a success or failure message so after multiple tries a failure is assumed."
I wonder if it's possible in these days of always-on connections for a small hiccup to turn into a self-generated DDoS if the whole BT network blips and than EVERY customer modem tries to re-authenticate? Does anyone ever turn off their modem/router these days?
"a self-generated DDoS if the whole BT network blips and than EVERY customer modem tries to re-authenticate?"
In the days before fibre broadband, BT Retail didn't use the DSL username+password stuff that practically every other ISP used, they simply checked that the line ID presented to the kit was a valid line ID for BT Broadband.
At that stage, some thought had apparently been given to this kind of thing. Probably all lost in the mists of time (and spreadsheets) now.
"In the days before fibre broadband, BT Retail didn't use the DSL username+password stuff that practically every other ISP used, they simply checked that the line ID presented to the kit was a valid line ID for BT Broadband."
not quite true - it did need a user name and password. But username could be anything@btbroadband.com
Us contractors for BT usually set them up as user= "btuser@btbroadband.com" with password "btuser"
That worked on all BT domestic connections
"That worked on all BT domestic connections"
It would. 'Cos like I said, while the BT Wholesale CentralPlus service was around (which was designed for and exclusively used by BT Retail), it don't need no username and password for authentication, just the circuit ID, which is provided by the infrastructure rather than the end user. So all that typing may have been even less necessary than you thought.
Feel free to look up BT's SIN for CentralPlus if you like (it's over a decade so I could be misremembering, but I don't think I am).
It would. 'Cos like I said, while the BT Wholesale CentralPlus service was around (which was designed for and exclusively used by BT Retail), it don't need no username and password for authentication, just the circuit ID,
Ever so slightly different now (or was 6-9 months ago, whenever they put my Infinity in).
The username/password still isn't technically used by a HomeHub. The Modem's MAC is whitelisted and the authentication works/fails based on that - they managed to send out a batch of HH's without authorising them to connect them to the network, so I had a very confused engineer here at the time.
As others have said, yesterday, CHAP was failing. Prior to that though packets were making it 1 hop into BT's network and then falling off the network
"Prior to that though packets were making it 1 hop into BT's network and then falling off the network"
Falling off the network that **you and other BT customers** can see?
I have no idea how this 21CN stuff works internally, but its 20CN predecessor was at one point using Cisco Express Forwarding (?) to get encapsulated Ethernet packets from BT's broadband customer-side kit (BRAS??? Not DSLAM) over to BT's core internal network and then out again to kit at the ISP in question.
That journey from BT remote access kit to ISP kit could be multiple hops along BT's internal routed network, but none of those hops were normally visible to end users trying to use traceroute to diagnose what was going on. Think of it like tunneling, if you wish.
Things may well be different by now.
Fault resolution doesn't seem to be any better :(
[I was there when Pipex invited mass market broadband in the UK, and there soon after when BTw were having major issues because of end-user packet fragmentation across their CEF network, and when BTw were denying that they had problems with exchange backhaul congestion, and... I've lost track/interest since then, it's all too depressingly (un)predictable].
"If only BT would choose today to give me their regular monthly cold call to try and entice me into leaving my current ISP and joining BT; I could waste more of their time telling them why I won't and would enjoy it even more than usual."
They have been trying but their IP phones seem to be down at the moment.
> Probably a case of "If it's working, DON'T reboot the router/modem"
There may be an element of that, but we've had customers "just go offline" - router not rebooted, just lost connectivity. I did notice that downstairs where the centre manager has a BT Infinity line that the router was showing a red "b" so it's not just lost packets.
Just checked on another customer router - the router hasn't been rebooted, but they've been offline.
Something else interesting, two customers I monitor have dynamic IPs but they've come back up with the same IPs that they had before the outage. That's unheard of - BT normally force an IP change on every re-connect which is a bit of PITA.
Puts me in mind of the last chapter of A Fire Upon the Deep.
I wonder what (who?) the countermeasure was aimed at this time?
Operator services are down dialing 100 to speak to someone gets you an engaged tone dialing 150 gets you a network busy message as does the fault checker line (0800 169 0199) if you do manage to get through to someone their systems are down also got told to phone back in 4 hours.
"Operator services are down dialing 100 to speak to someone gets you an engaged tone dialing 150 gets you a network busy message as does the fault checker line (0800 169 0199)"
All working from here in suburban south Birmingham on a standard BT line.
0800 1690199 option 1 (Broadband faults) goes to an engaged tone.
I think it needs rebooting.
In other (hopefully unrelated, but...) news: any Gridwatch fans out there?
Have you noticed the huge hiccups in the charts, just before Thursday/Friday midnight last week, and then again today at around half past midday? Odd.
Still, at least we've still got electricity.
I have no doubt it will be blamed on a failed server upgrade and that will be it, they won't bother to explain how a failed server upgrade kills so many different systems or why no backup system spring into life. Something must be seriously wrong if you can't even dial 100. I haven't checked but I hope this hasn't also affected to ability to dial 999 or I can see BT being hit with some big fines.
Yeah, had a few drop-outs on TV and internet yesterday according to my wife. VM internet dropped out for about 10 minutes this afternoon around about 2pm-ish, coincidently the time BT seem to have dropped off the map.
On the other hand, everything seem to be working faster than usual. Maybe all those busy websites are not so busy just now :-)
Openreach don't supply products like MPLS directly to end user customers, they handle network maintenance and local access networks.
BBC will have signed up through BTwholesale, Global Services or a bespoke BTnet arrangement. They're not completely getting rid of Vodafone either.
The irony of most of BT's web sites currently being unreachable is not lost on me. (Why bother hosting it all separately, just fire it everything on the same server cluster...)
---
See also, because I still can't put more than a couple of inline links in comment replies:
http://www.homeandwork.openreach.co.uk/Our-responsibilities/
https://www.btwholesale.com/pages/static/products-services/ip-connect-uk.htm
http://www.globalservices.bt.com/uk/en/products/ip_connect_uk
http://www.bt.net/info/
Looks a lot like there's a 'high order failure' on the BT network in their London PoP, trace below.
If the problem is isolated to London, it might mean that northerners will avoid the problem and be routed through the ManIX PoP. Anything hosted in London though, or anyone in the UK who's connectivity will traverse a London PoP/LINX connectivity is screwed!
2c3-xe-1-0-1-0.uk-lon1.eu.bt.net 166.49.208.14 gb 17.414 ms
t2c3-xe-2-2-1-0.uk-lon1.eu.bt.net 166.49.208.114 gb 18.650 ms
t2c3-xe-2-2-2-0.uk-lon1.eu.bt.net 166.49.208.116 gb 19.462 ms
9 166-49-211-239.eu.bt.net 166.49.211.239 gb 18.202 ms 18.221 ms
166-49-211-241.eu.bt.net 166.49.211.241 gb 18.220 ms
10 core1-te0-3-0-10.ealing.ukcore.bt.net 109.159.254.103 gb 22.484 ms
core2-te0-4-0-7.ealing.ukcore.bt.net 109.159.254.162 gb 22.417 ms
11 * * *
I work in tech support for an EPOS company and we were getting so many calls about "we can't take card payments and access the internet" blah blah blah (because they seem to think we are responsible for more than just the tills). This has been going on since about 2pm, mostly London, although did also get a call from Worthing, near Brighton
I've spent most of the past hour waiting for a fun news story to read. I wonder if BT will drop us a statement as to why they done TITSUP
SKY seems to be working fine and IIRC they use BT's equipment.
Only some of it..and you have to be careful what you mean by 'BT'.
Sky use the local loop which is owned and operated by BT openreach. For ADSL they have their own DSLAM inside each exchange but for FTTC (Fibre) they are reliant on openreach DSLAMs in the cabinet. Sky may also at some (most?) exchanges use openreach cabling to get to the nearest Sky POP.
This fault seems to be a BT Retail issue (I suspect a server room outage of some kind). There is no relationship between Sky and BT Retail so no reason to expect faults on one to impact the other. Think of it like a major fire at a DHL distribution hub. All couriers use the same road network but only parcels passing through that DHL hub will be affected. Other couriers can carry on quite happily.
"This fault seems to be a BT Retail issue (I suspect a server room outage of some kind)."
I'm on Plusnet (not exactly BT Retail). I've been able to get to here and various other places, but I've been struggling to get to bt.com and btplc.com.
Maybe, just maybe, there's more to this than a BT Retail fault.
Here's what the normally well informed ADSLguide/thinkbroadband have as a news item so far:
http://www.thinkbroadband.com/news/7336-bt-websites-down-and-out.html
[...]
The main bt.com, btwholesale.com, openreach.co.uk sites appear to be down, we have fired off a question to try and find out why.
This does not appear to be affecting peoples broadband connections, as based on activity on our speed test we are still seeing the expected numbers and speeds for this time of day.
[...]
Your own post seems to explain that adequately:
"I've been struggling to get to bt.com and btplc.com."
and (from TBB):
"The main bt.com, btwholesale.com, openreach.co.uk sites appear to be down"
So nothing there to suggest that the fault has 'spread' to Plusnet. Just a failure on BT's network that affected some of its customers and also knocked some of their web sites off the wider web.
We have a number of VPN connections with BT and whilst a lot of them are affected by this, the odd one or two are still showing as working... Our service provider have amusingly told us they aren't sure if there is a major BT outage as they are unable to access any of BT's service portals to check!
Ah, BT outages. The first of the year is always so sweet.
I usually now refer to this ever more accurate video from 2007.
Time to evac to the pub (for its free wifi, natch)
Bah! The situation shown is business as normal in our office whenever the "planning" is being rejigged, which happens randomly between 11:00 and 18:00. It is positively correlated with the "most important meeting of this week" being suddenly canceled for unstated reasons a hour or so earlier.
Still not properly up yet, have a BT Business Infinity line working slower than a 56k modem at the moment and bizzarely UK internet addresses seem to be OK (ish), but should I wish to log in to any other service or page from elsewhere in the world its really struggling and most of the time unable to connect.
I'm fine here in my little pueblo pop. 2700 in Spain, but the office in deepest darkest Essex is Kaput.
Senior source at BT said 'we have a serious outage but I still haven't got a f*ckin clue whats up. Just a stream of unhhappy calls to my mobile.....'
At least he bothered to call me back to tell me... bless.
This post has been deleted by its author
[former BT Business Broadband call centre drone, ex-brother in law BT Openreach engineer]
Contacted ex-colleague at BT wholesale (they who own the cables, servers), their reply
"The Hamster that powers the servers has escaped, it's hiding under the rack and not coming out for carrot, so management having meeting to see what vegetable to try next. Decision thought to be made sometime next week, [colleague #2] is looking for the vacuum cleaner and a late opening pet shop."
The CHAP authentication was failing here - there was simply no response. Those who weren't kicked off - were provided with a private 172 address. Still no explanations as to why their sites were down, and more importantly - why pretty much all phone numbers for BT including - the operator were down. Surely there are backup systems in place?
This post has been deleted by its author
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2025
