'Major' outage at Plusnet borks Brits' browsing, irate folk finger DNS Plusnet, the internet service provider, has suffered a "Major Service Incident" which started on Wednesday night. The outage continued until this morning. Users are complaining that they cannot visit a number of websites. Some were able to mitigate the TITSUP by changing their DNS provider. A Plusnet staffer told its …
It was NOT a DNS issue but some routing SNAFU.
However the problem stopped access to the DNS servers people were using - changing their DNS settings to point to some other server allowed thenm to resolve the address of a site that was reachable therefore the belief that the issue was DNS.
I can add that it was definitely a routing issue and not DNS, and it was random. So a connection in Scotland on Plusnet with its own private DNS server worked fine. A similar server in England did not, some but not all DNS lookups where failing. A random Plusnet connection down the road in the next village from the one in England also had partial DNS failures.
Funny that after changing my DNS to use 8.8.8.8 (google) it all worked fine.
So for those of us who can't tell a routing from a DNS failure can someone enlighten us?
Why did just chsnging my DNS provider work when an nslookup on the beeb got
** server can't find www.bbc.co.uk: NXDOMAIN
If it is a routing to the DNS then great but for most of us this equates to a DNS failure.
Such a routing failure amkes large amounts of the Internet inaccessible. traffic simply can't get there. Due to the fact that routers attempt to automatically correct around failures, this can mean that some sites are periodically accessible or accessible by one user and not another.
In this case, the DNS servers you were using were one of the parts of the network you couldn't reach (or the PN DNS servers couldn't reach other upstream DNS servers) so it looks to you like it's a DNS problem when the real problem actually lies elsewhere.
In your case, the sites you wre trying to access were in the subset of the network you could actually reach. Lucky you :) - You would have found other sites that were inaccessible.
All the handwaving claiming that chaing DNS servers fixes the issue just muddies the waters.
Take a spider web. Cut it down the middle.
Some flies (users) can use their parts of the web (e.g. DNS) without problem because they are on the "same side" as the service they want to use. Some people can't, because all the vital stuff is disconnected from them. Changing their DNS to one on the "same side" might well solve one problem and at least let them resolve DNS address, but not necessarily talk to the website itself (or vice versa).
Why did just chsnging my DNS provider work when an nslookup on the beeb got
** server can't find www.bbc.co.uk: NXDOMAIN
If it is a routing to the DNS then great but for most of us this equates to a DNS failure.
Because you probably rebooted the router making the change, either explicitly or automatically. I noticed my Internet was down this morning, rebooted the router and it was back up again. Can't be anything to do with DNS since I run that internally referencing the root servers.
You should have got a timeout error if you couldn't reach Plusnet's DNS cache, or SERVFAIL if Plusnet's DNS cache could not reach any of the authoritative servers for the domain you were querying.
NXDOMAIN means that the given domain name definitely does not exist (i.e. the authoritative server replied saying "no such domain")
dig is more trustworthy than nslookup for this; it shows the raw DNS request and response. And if you are running under Windows, I would not trust Windows' own caching. Nor would I trust the DNS proxies inside many consumer routers.
In such an event it is instructive to compare:
dig m.theregister.co.uk a # using your configured caches
dig @8.8.8.8 m.theregister.co.uk a # using a specific cache
Outages happen -- though quite how Plusnet have managed to end up with broken DNS and no websites (except community forums) is beyond me. Hopefully we'll find out what went wrong.
BUT the biggest concern is that users were reporting problems since the small hours, and responses only started coming when the day staff rolled into work around 08.30am. That's no way to run a service for hundreds of thousands of customers.
Perhaps their customers can sit carving wooden spoons while they wait.
It's interesting that their TV ads say they gat higher customer satisfaction ratings than Sky, Virgin and TalkTalk. There's at least one large ISP (hint two-letter acronym) not in that list...and saying you give better customer satisfaction than TalkTalk really isn't much to write home about. Sweeney Todd was probably more popular with his customers than TalkTalk.
"It's interesting that their TV ads say they gat higher customer satisfaction ratings than Sky, Virgin and TalkTalk. There's at least one large ISP (hint two-letter acronym) not in that list...and saying you give better customer satisfaction than TalkTalk really isn't much to write home about. Sweeney Todd was probably more popular with his customers than TalkTalk."
Well it's probably because they do get higher customer satisfaction ratings. Check out ThinkBroadband's (independent) comparison table (you'll notice BT is there). Select the "Big Five" (BT, Virgin, TalkTalk, Sky and PN) and you'll see that at least in the eyes of TBB users PN are top, with Virgin a close second.
http://www.thinkbroadband.com/isp/compare.html
If you compare PN with AAISP and Zen, they don't do so well. However the devil is very much in the details and if you hover your mouse over each graph bar you will see information about the sample size. Plusnet and the rest of the Big Five have very large (and therefore reliable) sample sizes. Zen and AAISP are very low.
If 1,322 people give PN an average overall satisfaction rating of 77% (or 2,144 give Virgin a rating of 73%) then I'm prepared to believe them.
> If you compare PN with AAISP and Zen, they don't do so well. However the devil is very much in the details ...
Indeed.
And you also need to factor in that AAISP and Zen will tend to have self-selected "more technically knowledgeable" customers - or put another way, "average user" probably won't want to pay their prices, leaving those with more discerning requirements. These users will a) have a better understanding of issues and accept that "sh1t happens" from time to time, and b) be more able to deal with issues at the customer end (so not complaining about the ISP when it's not their fault). These customer probably are more knowledgeable about the difference between the ISP and BT OpenReach - not blaming the ISP for OpenRetch issues.
So on the whole, I'd expect higher ratings (from a much smaller sample) for this type of ISP.
"Because they have tweeted from their mobile phone which is using the mobile signal to send the message and not routing over their Plusnet connection."
I guess TehIntraWebTubes users of this century have problems understanding how the entire contraption works. Sad that. And somewhat scary, given the ubiquitous of the situation.
Whatever the actual cause was the fact is that this BT spin off - PlusNet - have had this issue before when they blamed customers for using non PlusNet routers!
Today customer disservices were unavailable and even now their Service Status page is still down.
Generally PlusNet service is OK but support really needs to improve.
No problem with PlusNet fibre here this morning (Friday) as I'm working from home.
I didn't notice anything last night either, but I wouldn't have given that effectiveness of my wife's bedtime story on both my two year old son and me, I didn't wake up until 5am wondering what happened to the rest of the evening!
I found their support to be best around. Perhaps not quite A&A but for well priced broadband with decent packages and sensible bandwidth policies, they are unbeatable.
I constantly laugh at idiots that complain how slow PSN is for them, and it turns out their crappy ISP is throttling Akamai and they didn't know it....
@Andrew Martin 1
Same here, though I think there's been a couple of worse ones ('worse' defined as being unable to reach a work VPN to route around the problem).
Problems are few and far between - they may (like all other mainstream ISPs) have occasional issues with DNS, mail etc but I don't use them services.
Support has always been pretty good on the occasion I've had to use it and will quickly go off-script and provide actual help when they realise you know the difference between an IP packet and a box of teabags.
The last time I tried to call customer services, I was advised of a wait of around 30 minutes and hung up: took 2 days to get a response to my online enquiry and the response was that I should phone up. Support hours for their consumer products have reduced (used to be 24hr, now 07.30 to 22.00). VoIP phone service that used to be available with consumer products has been discontinued.
They used to provide a good value quality service, but they seem to have been chipping away at that for months, possibly in response to price competition. Without moving to a "business" service, I'm not sure any of their competitors are any better, though.
@ Andrew Martin 1
> I have to say that I've been with them for something like 14 years. Problems are seldom - I've never seriously considered switching providers.
Me too (12 years in my case, since ADSL arrived at my exchange), not only have I stayed with them, but now I've moved out I'm planning on going with them again.
@ Warm Braw
> Without moving to a "business" service, I'm not sure any of their competitors are any better, though.
INdeed, although they have been chipping away at some of the "bundled bits" - mail, web hosting, VoIP phone, etc - I still think they offer a reasonable value. Yes, there are plenty cheaper, but having dealt with various ISPs at work, they are very very much not the worst for customer service.
Example: Went out to customer site - internet not working. They'd already called the ISP and been assured (during a 1 hour call) there's not problem with the service - it must be the customer's equipment and they should get their IT people to look at that. I get there, and there is definitely a problem - but it's not "no service" as I can send/receive personal email (SMTP/IMAP) and use SSH. But port 80 and 443 (ie web browsing) are just completely blocked. I ran tests with various permutations to be sure, and then called the ISP back.
Again, they were completely insistent that there was no problem, and unless the customer used the provided router then they wouldn't do any more. I asked about the filtering - no we don't filter. I was insistent, increasingly so. *eventually* I got through to the droid that there was a problem, and it was on their end, and guess what - there was a problem at their end.
Customer had paid the bill online, in-time, but the ISPs internal systems hadn't registered it properly. So customer was on credit hold - but the systems didn't redirect to the usual "please pay the bill" page. I suspect the customer never recovered what we billed them for dealing with this.
"It's good to talk" (showing my age ?) - like heck it is !
I've been with Plusnet for 9 years and have been pretty pleased with the service I've received throughout. However, I've found that you can wait a long time after opening or updating an online support ticket before anyone does anything about it.
I still send them support tickets with all the relevant details filled in but now I ring them up immediately afterwards and say "Hey I've got a problem! Briefly, it is blah, blah, blah. All the technical details I can provide you with are logged in support ticket 123456" and they always get straight on to it.
Although getting "straight on to it" usually means putting my problem at the back end of a BT Openreach engineer's 72 hour long to-do list. To be fair to Plusnet, there's not much they can personally do when my exchange-only phone line gets shot at, chewed by squirrels, waterlogged or stretched as it catches a falling tree while it snakes its way over miles of fields and woodland to my house.
For this latest problem, I've not noticed any interruption to service, having had OpenDNS and Google set as my primary and secondary DNS servers since way back when I noticed I could no longer access certain torrent sites using the Plusnet DNS servers.
Yes, but all the insistence that it's a DNS issue won't help most of people's issues, muddies the waters for people trying to troubleshoot on PN's side and may encourage people to fiddle with settings* that they really don't understand, make a mistake and then totally screw up their own connection.
Incorrect diagnosis is almost alwyas counterproductive.
* That said, not using an ISP's DNS servers is almost always a good idea.
There is one very good reason for using an ISP's DNS servers and that's to do with Content Delivery Networks.
If your ISP has CDNs deployed in their network it will improve your service.
Google, Netflix, Akamai, and others do deploy their (and their customer) content inside certain ISP networks.
Less so now that the "edns-client-subnet" extension is being rolled out - OpenDNS and Google's servers can now pass on the subnet a client is on, so Akamai and co will return "the server IP nearest to that client's subnet" as opposed to "the server IP nearest to that DNS server".
It's not perfect, but apparently most CDNs now either use anycast (which avoids the issue by other means) or edns-client-subnet, so not much of a problem these days.
Also, OpenDNS routes your queries to London, across LINX - so the worst case should be that you're pulling traffic across LINX rather than internally. Not perfect, but not as if it's pointing you to a server in New Zealand either: you shouldn't see much difference. (My ISP has no Akamai servers on-net anyway, so all the traffic comes across LoNAP peering; even if they did, not all content is cached in the local servers, so you end up doing that too.)
I get the best of both worlds, by using my own resolver: as long as I still have UDP connectivity, I have working DNS service, and CDNs all see exactly the right IP address for me too.
Wait, so you voluntarily share all your DNS data with Google?
Every website you visit
Every mailprovider you POP/IMAP
Every messaging server you use
Every update server your installed software contacts regularly
Essentially anything that involves domain names gets shared with Google
Why not have a look at the OpenNIC Project? http://wiki.opennicproject.org/Tier2 You can choose which country the server resides in (No DNS providers outside the EU for me), whether they log any data etc.
Just select a dozen that you trust, run them through NameBench and pick the fastest ones…
Didn't notice an outage myself, but I'm pretty sure I've configured my router to use a DNS server each from OpenDNS, PlusNet and Google.
PN are generally OK, though every so often they do mess up big time (storage migration, getting hacked) every few years. Also they're one of the more open ISPs when saying what they're up to and admitting to mistakes they've made.
Hopefully it will take them less time to fix it than the eight-months-and-counting that it's taken to investigate the ongoing single/multi-threaded download issue. The only saving grace with that (for me) is that I only suffer it every couple of months and it can be cured quiet reliably and quickly.
Then again, it can be nasty when it strikes.
> Then again, it can be nasty when it strikes.
FWIW, I've seen worse !
We had a few customer FTTC connections with Vodafone, and performance was utterly carp. Variable, but generally carp. Luckily here in the office, we also have our leased line we can switch to if the FTTC line we normally use for the office traffic is poorly - not much help for our customers.
Through use of mtr (effectively traceroute) I identified that there was just one node in Vodamoan's network that was causing all the problems - but "nothing could be done" other than migrating lines and that would need new IPs yada yada yada. This went on for about 8 months and they were still not ready to issue new logins/IPs.
Then last month a tsunami of traffic hit the internet - aka Windows 10 was launched. Our own FTTC line more or less died (it had generally been better than our customers), and several of our customers effectively had no connectivity on the Monday morning. It eased a bit, but was bad again on Wed morning when MS released a big patch. And still it was just one node in Vodamoan's network causing the problem.
But, it got so bad that our leased line was affected - and I diagnosed that this too went via the same node although traffic on it had never been affected. As this line has an SLA, and they were out of it, they were actually talking to us ! But it was still "well we did mention that you'll be having to migrate (with new IPs) at some point"*, and "sorry we can't fix it". We didn't take that lying down. I don't know who our account manager talked to, but next morning both our lines were working just fine, and so were our customers' FTTC lines - traceroute shows that they no longer went through this congested node.
So for 8 months Vodamoan had effectively refused to fix a known fault - I suspect because it's in a bit of "legacy" network they don't want to invest in. It just took a big enough fault - one they'd have to pay compo for - and they "found a way to fix it".
* Actually, they hadn't, it was no more than a passing "BTW you might need to migrate sometime, but we don't know when" from our account manager during a chat about something else.
I've lived at the same address for about four years. The first three I used BT, but I was paying about £35 per month for ADSL and getting 15Mb. At least I never experienced an outage. Around the start of this year, I saw a deal where I could switch to Plusnet and pay £2.50 per month instead, so I decided to give them a go. I now get 12Mb, have to reboot my router about two or three times per week and I've had a total of about six days with no broadband at all.
It's dirt cheap, but I think I'll try switching back to BT at the end of my 12 month contract. :-/
@Tom_
I suspect that when you move back to BT you'll still get about 12Mbps. It's likely that your connection speed has declined over the years due to contention, crosstalk, or simple increase in BB customers.
The PN community forum regularly features people either planning to move from PN due to speed issues, or recently arrived from another ISP for the same reason, and in all cases I can remember the speeds were the same for them.
I signed up for the FTTC service in 2013 and was getting 72Mbps, probably because the Exchange had just been fibre-enabled and I was among the first. Now 2 years later I'm getting between 55Mbps and 60Mbps, almost certainly down to crosstalk etc.
I took plusnet when I moved to my new home in 2013 - 80/20. Predicted line speed; 75. Since day one it was problematic - at first not syncing over 40, but having to leave it for the 'line training period' before they would help me (though I was under the impression the line training only tuned down speed and tuned up interleaving to combat errors discovered on a new line). After a month or so of persuading them this definitely was not normal for being 400m from the cabinet, they sent BTOR who replaced the D side and now the line worked to about 60 meg. Other than the occasional outage it worked perfectly
Kept PN for year and a half and then switched to BT. It was a simple router switch out. I immediately jumped 15mbps in sync speed and the upload went to full 20 sync (well, 19999 but whos counting the one). Its been rock solid ever since
But the thing I do miss the most is plusnet have a very low ping time on FTTC compared to BT - BT seem to be around 18-21ms (depending which part of 21CN you bounce into) whereas PN is always a nice 9.
when are they gonna roll out IPv6?
There was a notice on the PN site talking about BT Maintenance. Some of it lastnight was in my area but changing my DNS provider sorted things pretty quick apart from the PN website which I guess has been inadvertently DDOS'd ofline by all the complaints.
The crappy Tecnicolour Router they supply seems to have its DNS hardwired to PN. The episode today have simply brought forward the day when I'll get a DD-WRT router and replace it. I really want to totally firewall off any W10 snooping data from going out of my network.
I've been with PlusNet since dial-up days with Free-Online and have usually had good service from them. Since moving to ADSL with the old Frog modem / router, I think that I have had three outages that weren't down to a more widespread issue and one of those was caused by some silly sod cutting through my phone line.
I've moved house with them and sorted out various questions / minor problems with their tech support and have found them to be generally pretty helpful.
I can sympathise with those who haven't had this experience and I do sometimes wonder when it will be my turn!
Well ty Plusnet for wasting 45 minutes of my life rebooting routers, fiddling with filters and muttering into my museli, all because about 8 hours after your systems started wobbling your 'service status' page said 'No problems'. A simple 'oops we are having issues' message would presumably have saved you about a million phone calls and allowed us to get on productively spoon whittling.
This post has been deleted by its author
When MS rolled out windows10 BT chose to "upgrade" their network infrastructure so as to give very similar symptoms to this fault.
Assuming BT were not adding additional hardware as has been suggested to spy on it's customers for agencies undisclosed then they could at least have learned form their previous error or both. Either way customer service is clearly not a priority with BT who have pushed most support to India and in my experience recruit a high percentage who cannot communicate in English even when they have a clue about ISP support
On the spying issue this would not be the first time that BT rushed to compromise their customers, anyone around during the Phorm debarkle will remeber that BT rolled out unsanctioned montioring of their customers network traffic and unsuprisingly went unpunished by the government agencies supposedly paid to protect us.
at my local sandwich emporium this morning the lady behind the counter was small talking and asked what I did.
"I'm a telephone engineer"
So do you know about the Internet and that?
"A little bit"
I can't get on this morning, I get errors about DNS servers!
Your ISP DNS server must be down (Thompson Plusnet router sat beside the till with green lights for DSL and INTERNET)
"I can have a look if you want?"
Changed DNS servers to google and everything works fine.
She thinks I'm an IT God and I get a free BLT.
Other lady behind counter asks
"When I press @ I get : what could be the problem?"
I think I'm the tech support for the sandwich shop now :-/
I'm on Plusnet (have been since Tiscali days), and this is the first I've heard of the problem. Mind you, I do recall a couple of occasions when problems occurrd and a friend was able to find out what was amiss and advise me to try something along the lines of changing the DNS malarkey as people have pointed out above. Really must read up on that stuff, as I don't understand how, if my service is borked due to plusnet's address-handling thingummy failing (and my connection from my PC goes through it), asking my setup to look at some other adddress-handling thingummy is going to work, as logically I shouldn't be able to get through to it. Except that in some circumstances it DOES work, indicating a severe lack of understanding on my part. Any tips for a beginners read on the relevant subject?
Imagine you want to phone Directory Enquiries to get some phone numbers. But their own phone number is unreachable, because all the phones in their area are down, including theirs. You can't find out any phone numbers at all.
So you try an alternate DNS- sorry, you try an alternate Directory Enquiries. They are fine and can give you all the phone numbers you ask for. However, although you've now got a working Directory Enquiries, some of the numbers you ring still don't answer. Because they are in the same broken area as your original Directory Enquiries number.
This is why alternate DNS allows you to resolve web pages, but doesn't help if the web page you're trying to get to is itself down.
Hmmn.. thank you! I thought DNS servers were more like the local phone exchange, rather than directory enquiries (so local exchange is down = no interwebs till exchange is fixed). Now I'm wondering what is routing things to the new DNS provider if the old DNS provider isn;t... - (heads off to read up on such things) :-}
I've always used an external DNS provider with my ISPs (and I never use their mail, because that's not portable between ISPs) because it can go wrong occasionally. I used to use OpenDNS years ago, but got fed up them intercepting wrong lookups and putting up a search page, which is simply awful.
When my Linux machine boots, I put something at the end of the boot sequence to insert "nameserver 8.8.8.8" and "nameserver 8.8.4.4" into /etc/resolv.conf (overriding whatever DHCP sets) and I'm good to go with Google's DNS. Never had a problem with it ever snce I switched to them - Google's DNS infrastructure is likely to be much better than any UK ISPs I suspect.
I expect that this incident has been going on longer than they reported at the outset, every time I have tried to speak to Plusnet support I've been in a queue for at least 40 minutes, and to progress an incident always incurs the same wait. I suspect if this problem has any level of complexity greater than 'switch off and reboot' this issue could take several weeks to be resolved.
I'm very glad that I didn't try to report the issue
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
