back to article Web meltdown: BT feels heat from angry punters

A raft of BT customers in the UK were knocked offline this morning due to a power problem at one of its web peering partners’ sites in London. According to outage monitoring website Down Detector, customers reported a spike in problems at around 9am today. One customer got in touch to report that all their BT Infinity …

  1. Anonymous Coward
    Anonymous Coward

    21st Century Networks, resilient networks, self-healing networks, global re-routing, circular networks, mesh networks,...

    That was the dream...

    1. Anonymous Coward
      Anonymous Coward

      The dream became

      Has it changed? Oh, hang on a mo...

      vi networks

    2. hplasm
      Unhappy

      That was the dream...

      Profit!!

      That's the reality...

    3. ritey

      ...but they forgot to tell the cleaner not to use that socket for her hoover.

      1. AMBxx Silver badge
        Mushroom

        Wasn't TCP/IP designed to avoid problems in the event of nuclear war? Routing around problem nodes etc etc.

        1. VWDan

          So, the real problem is that we're not at Nuclear War?

        2. hmv

          Yes (at least the IP part). Of course routing around problem nodes only works if you have an alternate path.

          1. Anonymous Custard Silver badge
            Trollface

            You know things are bad when even Dido Harding is attacking you...

            http://www.bbc.co.uk/news/technology-36844712

        3. Doctor Syntax Silver badge

          "Wasn't TCP/IP designed to avoid problems in the event of nuclear war?"

          That was my reaction as well. I suppose design is one thing, implementation is something altogether different. You need those redundant routes and routers and they all cost money.

        4. Anonymous Coward
          Anonymous Coward

          Yes, that was the original idea. The problem we have now is that the global internet is now many, many orders of magnitude larger than was originally foreseen (don't forget - the original "internet" only had 20 or so nodes on it) which makes operating it a whole lot harder. For example the original routing protocols have long been pretty much abandoned (except in small networks where they work just fine) and more complex protocols have been developed and deployed. However these protocols can take longer to converge to an answer if there is a massive reconfiguration of the internet (such as a major node failure). Added to this is that no protocol can compensate for stupid network design (e.g. running primary & backup cables through the same duct, having both primary and backup systems on the same power supply, etc).

        5. RealBigAl

          Good Omens

          Yes, it was, but Nuclear War is as nothing next to the complexity that is BT's national network infrastructure.

      2. Anonymous Coward
        Anonymous Coward

        Apparently BT are fixing it at the moment with their help desk.

        So Far They've:

        1) Disconnected all equipment from other sockets (this is what took down the BBC website)

        2) Plugged their router into the master socket (this took down their trunks and backbone)

        3) Rebooted their router

        4) Taken delivery of another router and tried that

        So they're now awaiting an engineer visit. Should be fixed by next Friday.

        1. JohnG

          "So they're now awaiting an engineer visit. Should be fixed by next Friday."

          However, the engineer will not show up but they will subsequently be told that they will be charged if the engineer is to return and that there is no availability of an engineering appointment for another couple of weeks.

        2. Triggerfish

          I already emailed the office today and told them to expect a connection as momentarily brief as a Tory candidancy, and it's still the same. It's even making it hard to post on here. Office production levels may go up.

    4. JohnG

      Maybe the Chinese are testing the remote control features of Huawei's equipment or GCHQ has screwed up an upgrade of their interception kit.

  2. Anonymous Coward
    Anonymous Coward

    It's been like this since about 7am this morning at my workplace. Currently at 10:30 it's still intermittent. Some websites will load without problems and others will take forever to load.

    Which is annoying as I want to order some Hammerite paint from Amazon for my car.

    AC because I shouldn't be shopping at work.

    1. Dan 55 Silver badge
      Trollface

      El Reg is okay though.

      1. Anonymous Coward
        Anonymous Coward

        "El Reg is okay though."

        Yes, because amazingly El Reg worked so I could tell my boss why Gmail wasn't working.

        I *may* have taken a while perusing the other articles first though.

    2. Anonymous Custard Silver badge

      Had just the same, except in my case it's a hard drive caddy I'm after.

      Just managed to get to Amazon though (11am) and ordered it, so give it a try..

      Also had the irony of getting an RSS feed message from the BBC News about their article on the subject, but not being able to open it due to the issue...

    3. Crisp

      Re: Hammerite paint

      Have you tried eBay?

    4. Version 1.0 Silver badge

      At the pub

      Me and the PFY have adjourned to a remote site to check system connectivity.

  3. Anonymous Coward
    Anonymous Coward

    BT SIP?

    Does anyone use the BT Business SIP service over BTNet, and if so has that been affected?

    I was thinking of moving away from ISDN30s and onto that, and it is supposed to segregate the SIP traffic out from and Internet links so I would be curious to know if this has stayed up.

    1. AMBxx Silver badge
      Boffin

      Re: BT SIP?

      It's all random. My sipgate and voipfone numbers are down.

      No SMTP traffic coming through, but can access the email server's webmail page.

      I can RDP to some customers, but not others.

      I'd hate to be the one troubleshooting this.

      1. Anonymous Coward
        Anonymous Coward

        Re: BT SIP?

        That's why I was asking specifically about the BT Business SIP service as that should, in theory, not be affected by this issue.

    2. PhilipCo

      Re: BT SIP?

      We use SIP with BTNet. As far as I can tell, it hasn't been affected, whereas we are having problems accessing web sites and with remote users trying to access servers on-site.

      That said, it hasn't helped me at all to try and get through to a live person at BTNet. Being told to call back later is not at all impressive.

      1. Anonymous Coward
        Anonymous Coward

        Re: BT SIP?

        Thanks, Quick OT for those using it - do you find the BT SIP service reliable, any issues with it or are you totally happy with it, any unexpected costs?

        Would be great to get some feedback from genuine users.

    3. ursus42

      Re: BT SIP?

      Yeah, we use BT SIP and our internet is affected but phone calls are still working.

    4. Stoolbend
      Meh

      Re: BT SIP?

      We use BT SIP with BTnet and our SIP trunks have stayed up since 8:30. Problems started just before 8 for us.

      Not very pleasant being hung up on by the automated lady on the help number though, especially with the monthly cost.

  4. EnviableOne
    FAIL

    Peering Problems

    Something tells me OFCOM should force BT to Beef the pipes and failover to Manchester and Edinburgh. If one DC going out causes this much issues.

    Hmm if Openreach were independent, would they not have sorted this type of thing?

    1. AMBxx Silver badge

      Re: Peering Problems

      Openreach won't be split out until Brexit. At the moment, BT would just appeal to the EU and it would be overthrown as there's a similar problem in Germany with Deutsche Telecom.

      1. Doctor Syntax Silver badge

        Re: Peering Problems

        "At the moment, BT would just appeal to the EU and it would be overthrown as there's a similar problem in Germany with Deutsche Telecom."

        And if OR were to be split off how long do you think it would be before it was bought by Deutsche Telecom, or Telefonica - or maybe SoftBank?

  5. briesmith

    What's the fucking point?

    Millions, if not indeed billions, are spent on (advertising) network resilience yet still server centres and other installations fall over, go "off grid", suffer "outages" or "unplanned downtime".

    Is it simply impossible to prevent these occurrences? Is all the advertising about resilience etc complete dishonest bollocks?

    Or are the PoP operators just lying to us on the grounds that it is so much cheaper to be a crook than try and actually build in genuine resilience?

    And what about all these certificates they display so proudly on their websites? Are these all lies as well? Are the awarding bodies just in on the scam and taking the dosh while they can? Shouldn't an operator suffering one of these unexpected "inconveniences" lose their accreditation? And what about some com-pen-pay-shun?

    1. jtaylor

      Re: What's the fucking point?

      I don't know particulars of BT, but you hit some good points there.

      "Millions, if not indeed billions, are spent on (advertising) network resilience yet still server centres and other installations fall over, go "off grid", suffer "outages" or "unplanned downtime"." Indeed. Advertising brings in revenue. Infrastructure is just an expense. It's not uncommon to increase spending on the services (like advertising) while cutting expenses on the infrastructure that supports those revenue services. Years ago at a small chain retailer, the manager explained to me that because we were all paid on commission, "we polish the displays but nobody fixes the roof."

      "Is it simply impossible to prevent these occurrences?" Not impossible, but it requires awareness and also decision-makers must be rewarded for solid planning over short-term results. "Is all the advertising about resilience etc complete dishonest bollocks?" Not exactly. I've seen very resilient designs get crippled by small decisions like using the redundant link to handle load spikes instead of renting a metered link. As so often in this world, people prefer data that supports their message and may not even be aware of how the facts have changed.

      "And what about all these certificates they display so proudly on their websites? Are these all lies as well?" Yeah, sometimes. :) The certificates have very specific definitions. "Certified Malware Free" is much easier than "Scanned Every Hour According To OpSec 15(a) Which Is Has Been Due For Review For Two Years And Meanwhile We Changed Vendors And Our Tech Lead Left To Join A Startup So Nobody Really Understands It Any More But It Seems To Work Fine And We Are In Compliance With Our Accreditation." Again, not unique to IT. We probably all know someone who bought a very expensive car and then "saved money" by deferring maintenance. Or bought insurance but neglected to raise the limit after some major purchase.

      Okay, you nailed the big ones. I just spent too much time in Operations!

  6. Snapper
    WTF?

    Power Outage

    Power Outage you say?

    So, let me get this straight, Harbour Exchange in London's Docklands, one of THE most important links in the whole UK network, does not have adequate provision for when there is a power outage, either internally or externally.

    Effectively no resilience then?

    GOD'S TEETH!!!!!!

    1. Kaltern

      Re: Power Outage

      Putin will be taking notes...

      1. Justicesays
        Devil

        Re: Power Outage

        They aren't allowed to have a second site as GCHQ can't afford a second set of intercept kit....

  7. Anonymous Coward
    Anonymous Coward

    all a bit of a mish-mash

    Can't reach: BBC, some business sites, Big Commerce, bits of Microsoft, son's current school website, the Mirror website

    Can reach: Twitter, NHS sites, The Register (who?), local newspaper site, Fasthosts, Amazon, son's next school website, DailyMail (cough)

  8. tentimes

    A small number of our customers MAY be experiencing...

    Small my arse!

    Why do companies use this phrase when they know it is almost everyone who is affected? Get's right up my chuff it does.

    1. Anonymous Coward
      Anonymous Coward

      Re: A small number of our customers MAY be experiencing...

      Glad to see it's not just me who sees the red mist with this sort of wording. It doesn't matter if it is a small number of customers, for those customers it is a complete loss of service.

      PlusNet has been having occasional lie-downs in a dark room with a damp cloth over its forehead for a week or so now. Again, the mysterious power issue might be causing some problems for a small number of customers (which always seems to include me). The falling over seems to have started round about the time they told me my bill would be increasing to pay 22 tattooed millionaires to kick a ball around on a TV channel I don't watch.

      1. Psycho Flump

        Re: A small number of our customers MAY be experiencing...

        If you're seeing an unstable connection via Plusnet you may be suffering the same thing I've had. It seems IP addresses starting with 51. are seeing packet loss. Go to their addons in your account and add a static IP. They'll add it to your bill but I'm planning on arguing for a refund as they've set a precident in their forums by giving it to someone else for free for the same issue. My IP switched to an 81 and my problems (seem) to have gone away.

    2. keithpeter Silver badge
      Windows

      Re: A small number of our customers MAY be experiencing...

      Compared to, say, the number of particles in the known Universe, it was small number of customers affected.

      Perhaps the regulators should force companies to release an estimate of the percentage of customers affected together with any qualifying factors such as geographical limitations on the disruption.

  9. Crisp

    Keep Calm

    And

    Play Solitaire

  10. This post has been deleted by its author

  11. Banksy
    Flame

    Heatwave

    They must have forgotten to apply sunscreen to the relevant components or certain parts of the Internet decided to go to the pub because it was too hot.

  12. Alister

    The end of the world is nigh!

    Or so my users would have me believe. This is because, apparently, Wikipedia is unreachable at the moment, (oh, and our cloud based support ticket system, too).

    Hmm, another cup of tea, I think.

  13. MJI Silver badge

    Down at home

    Only some sites active.

  14. MJI Silver badge

    Just had an email

    Still down at home

  15. Anonymous Coward
    Anonymous Coward

    Something's back working... as I was reading this my VPN suddenly decided to connect and skype connected... probably only temporary... now I can work

  16. Gergmchairy

    Its coming back up

    I've got my VPN to the office back :-(

    Thats from PlusNet to Daisy Telecom via Openretch...

  17. Anonymous Coward
    Anonymous Coward

    Use BT; deserve what you get

    If you choose BT you don't deserve good service. And I think you know that.

    The simple test is: Does your ISP adverse on TV? If they're shit and need to advertise on TV. (They also spent all the money for expanding capacity on advertising).

    All the best things in this world don't need advertising because people just know they're awesome.

    1. Anonymous Coward
      Anonymous Coward

      Re: Use BT; deserve what you get

      > If you choose BT you don't deserve good service.

      While I understand (and mostly agree with) your sentiment, this is nothing to do with the (home) ISP side of things. At work we are not with BT, we are with another carrier, on their own infrastructure even (for a few more months) their own fibre into our office. No BT at all until some point waaaay back into the internet - and we're affected.

      Spent the first bit of the morning trying to puzzle why we were getting so many complaints of intermittent access to our servers. Then got a report via one of our other providers about the BT outage. Their joke of a status page still says "a few customers" may be experiencing problems as though it's a localised thing to a few FTTC cabinets.

    2. RedCardinal

      Re: Use BT; deserve what you get

      >>If you choose BT you don't deserve good service. And I think you know that

      Yeah yeah, whatever.

      Meanwhile, the rest of us will use a large ISP (like BT) and which we generally have no problems with. I use BT at home and they're fine . I see no need to change to a small ISP no one's ever heard of and which only has 3 customers.

      But feel free to feel superior to the rest of us...

  18. magickmark

    DNS Problems?

    I'm not at home to try it out but I've heard reports that if you change your DNS server to the Google public servers (8.8.8.8 and 8.8.4.4) this may fix the problem.

    Numbers taken from here:

    https://developers.google.com/speed/public-dns/

    1. jabuzz

      Re: DNS Problems?

      Indeed I personally think the problem was all down to DNS issues. At least I could ping sites directly with the IP that would just give DNS failures using their FQDN. Changing my forwarders from PlusNet's to Google's miraculously fixed everything.

  19. Ol'Peculier

    Knocked out the Royal Mail's Despatch Manager as well. Very annoying. Seems to be up now though, although we'll see if it stays up...

  20. Anonymous Coward
    Anonymous Coward

    Zen OK

    See subject. I'm currently happily using the Internet via a VPN I'm not supposed to have to my home broadband connection. AC for obvious reasons!

  21. Banksy

    Quite interruptible

    http://www.ispreview...d-services.html

    "UPDATE 11:07am

    A private report from TeleCity, which was sent to certain providers earlier today and seen by ISPreview.co.uk, suggests that there was a power failure with one of their Uninterruptible Power Supply (UPS) systems at 8/9 Harbour Exchange (LD8) in London.

    The fault hit a key router (edge4-tch), although this was technically resolved at 9:15am. However the knock-on impact for ISPs like BT and their clients (consumers, businesses and other ISPs etc.) can often take a little bit longer to resolve (computer networks are complicated animals).

    Connections should be slowly getting back to normal."

    1. Anonymous Coward
      Anonymous Coward

      Re: Quite interruptible

      The report from bt I saw at 11:30 said power issues were still on going and they were unable to stabilise the power at that time

      1. Anonymous Coward
        Anonymous Coward

        Re: Quite interruptible

        > The report from bt I saw at 11:30 said power issues were still on going and they were unable to stabilise the power at that time

        And BT would never try to blame someone else's power issues rather than admit that its own network fails to return to normal quickly enough.

  22. JohnG

    Whilst I can resolve names to IP addresses and I can connect to some websites (gmail, the Register, Guardian, Telegraph), the routing for others seems to get lost in Telehouse (including my VoIP phone service with Sipgate). When connected via a VPN out of Italy, I can access most but not all of the stuff not accessible via BT.

  23. Commswonk

    And after that...

    This is due to power issues at one of our internet peering partners’ sites in London. Engineers are working to fix things as fast as possible.”

    Once the technical problem has been resolved perhaps someone could find the time to find a better term than "internet peering partners", which has an unpleasant ring of New Age Management Speak about it.

    1. Anonymous Custard Silver badge
      Trollface

      Re: And after that...

      "internet peering partners"

      Sounds more like something that you could get arrested for, or at least could make you go blind...

  24. zombieroadrunner

    Openreach possibly offline too

    According to one of my partner ISPs, there are major issues with Openreach's tool stack today as well - they are unable to run line tests and are having issues raising jobs.

  25. Anonymous Coward
    Anonymous Coward

    Plusnet too?

    I wonder if Plusnet's current problems reaching some regions of the internet are related.

    1. Anonymous Coward
      Anonymous Coward

      Re: Plusnet too?

      Yep Plusnet's network is basically "Your Router > BT OpenReach > Plusnet > BT IPP > The internet"

  26. OohAahVicar

    There goes two hours of my life that I'll never get back...

  27. OohAahVicar

    Two hours of my life I'll never get back

    Latency was soooooooo bad getting to AWS that I had to resort to 4G on my mobile to manage my instances :-(

  28. chrisjlocke
    Facepalm

    "Fill out this simple form"

    So when we lost our Internet and it came back on, sites were still broken, but I managed to get to the BT web chat. No go, as 297 people iin the queue. "Submit your query by email". OK.

    I kid you not, this was the form I had to fill in.

    http://i.imgur.com/u5n9eNA.jpg

  29. Scourge

    Did someone mention Telehouse? Home of AC units dangling out of windows? Where a certain BT-occupied suite was a phone-charger away from tripping the room if not the floor? Inconceivable.

  30. Anonymous Coward
    Anonymous Coward

    Good luck getting hold of Telecity (but it's Telehouse anyway)

    Since they don't exist anymore - bought by Equinix.

    But actually I think you're focus is better directed speaking to the nice folk at Telehouse North.

    Hell - if you drive past you'll observe opened windows - never a good sign in a DC- I assume the heat is getting to them ;-)

    https://www.linx.net/tech-info-help/traffic-stats - LINX show a big drop in traffic earlier this morning - presumably that's what it looks like when BT stops peering in THN - but it's back to usual now.

    1. Anonymous Coward
      Anonymous Coward

      Re: Good luck getting hold of Telecity (but it's Telehouse anyway)

      Telecity - error introduced at subbing stage - fixed.

  31. Anonymous Coward
    Anonymous Coward

    Yeah, I'm currently sitting in a pool of sweat doing a pc upgrade in a windowless back office of a retail chain cafe and stuck at 24M of a 60M download. Not helped by the manager turning the power off half an hour ago to fix something that tripped. Probably not helped by the freestanding air conditioner with its exhaust pipe propped up against the wall pointing directly at said switchboard.

  32. Anonymous Coward
    Anonymous Coward

    AD Blocker

    Phew!, installed the ext Ad Blocker last night to my browser config. Got up this morning and thinks hmm, can't get on , this ad blocker lark is really good stuff.!! LOL's obviously....sigh.

  33. DuncanL

    VPN is a workaround till they fix it....

    Plusnet is knackered - which is a pain since I work from home. However, the work VPN will connect and I can get to everything via that.

  34. FlamingDeath Silver badge

    For the internet to have resilience, it needs to be de-centralised.

    It its current form, it is not de-centralised, it is very much controlled and owned by profit making shitbags

  35. mrmond
    Pint

    Rolling my eyes

    Zomg! I couldn't get on Twitter, or Amazon, or Flickr or my bank and some other pages for an hour or two from 8am. Slowly they came back online. Netflix was still up and running but I decided to read a book. For a while I wasn't sure if life would be worth living (insert sarcasm)

    Get a grip. All ISP's go down occasionaly. I've been with Virgin, Sky, Orange.

    It's all running ok now and life didn't end as far as I can tell :)

  36. This post has been deleted by its author

  37. Anonymous Coward
    Anonymous Coward

    Same today?

    I'm seeing similar issues this morning and downdetector is peaking again.

    I guess the London crew have installed some extra slurp servers but not figured the routing properly.

    1. Anonymous Coward
      Anonymous Coward

      Re: Same today?

      Ditto here; trying to connect a VPN to my company's network and am seeing ping times of 70-80ms (normally about 10ms) and packet loss of between 40-50%. Have run traceroute several times and found that the transmission times jump in BT's core network; not only that but there is obviously huge routing instabilities since the traceroute paths are changing on an almost second-by-second basis.

      This is not good!

  38. tomuk5

    same again today here too

    yup, same here, cant get access to certain websites at all, something is not right within BT Land

  39. tomuk5

    same again today here too

    yup, its buggered here too, something isnt right in BT land this morning.

    another power outage? some BGP/peering F**kup? who knows but my bosses are not happy!

  40. Keith Langmead

    Yep, have had many support calls this morning from people unable to connect to our systems (we're not with BT), all of which were from BT customers. The joy of trying to explain the difference between internet connection and internet routing to customers.

    Was amused last night to get home to some junk mail from BT trying to sell me "Super reliable BT broadband".

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like