back to article Smoke on the Tyne: Blaze at BT exchange causes major outages across North East England

Thousands of customers of BT's fixed-line broadband and telephone services are experiencing major disruption in North East England after an exchange in Newcastle caught fire. PlusNet, TalkTalk and EE customers are also affected, as well as smaller broadband slingers using BT's wholesale network. Specialist ISP Andrew & Arnold …

  1. Alister Silver badge

    Fire suppression?

    The last time I was in a BT exchange it was still running Strowger, so I don't know: do modern exchanges have inert gas fire suppression like any normal datacentre, or will the nice little fire bodies drown it in water?

    1. DeanT
      Flame

      Re: Fire suppression?

      Modern exchange technology in buildings built more than 30 years ago having anything like a modern fire suppressant system? The fact that they had local hose fondlers on site says probably not...

      The calls that I had to field today because of this though....lord almighty...

      1. vogon00

        Re: Fire suppression?

        Only 30 Years ago?

        Just for the hell of it, I cycled down to the local exchange building I had always intended to visit*, to find a very ancient sign proclaiming 'Telephone Exchange' above the door, and a date stone in the wall proclaiming it was built in 1966.

        Seems about right, as the condition, materials and (weird!) architecture of the building are consistent with the mid-60s. It's so unusual that I may be tempted to make an offer if it ever comes on the market..

        Having somewhat nosily viewed the premises, I effed off sharpishly just as the 'No Trespassing' signs suggested I should.

        Exchange code obfuscated to protect the guilty, but it starts with 'EA' which should give cognoscenti a clue....

        (*Actually, pay homage to as, back on the day, I was heavily into what is now the the 20CN and it's TDM beauty.)

  2. Cynic_999 Silver badge

    Huh?

    WTF is a "Transformer supply unit"? Something that supplies transformers?

    1. Anonymous Coward
      Anonymous Coward

      Re: Huh?

      The mortal enemies of the decepticon supply units.

    2. Roger Greenwood

      Re: Huh?

      Mains transformers often have an associated RMU (Ring main unit) so maybe the reporting got a bit mixed up? If the transformer failed and was oil filled it would be quite spectacular.

  3. DrBobK

    I'm in Durham City on BT and have had no issues. Durham exchange is 21CN so I suspect we are not dependent on Newcastle (Hadrian House?).

    1. Warm Braw Silver badge

      I understand that Internet disruption spread at least as far as central Dumfries (though not outlying areas), so it sounds like the network topology is not exactly straightforward.

    2. Anonymous Coward
      Anonymous Coward

      I'm also in the lovely city of Durham, a Plusnet FttC customer, and I lost my internet connection just before noon and was restored around 2pm; so at least some part of the fair city is being routed via Newcastle.

    3. ZippedyDooDah

      Tow Law in Weardale. We were out for approximately two hours. The Post Office told me that areas further south, in Darlington, were also out.

  4. anothercynic Silver badge

    Oooooeeerrr...

    Is Garmin1 possibly connected to BT in this exchange? *cough* *wink*

    1 See link

  5. Zog_but_not_the_first
    Trollface

    Smoke on the Tyne?

    Fog, Shirley?

  6. Eclectic Man

    Fire suppression systems

    The older ones used halon gases to push all the oxygenated atmosphere out of the building / room and thereby suffocate the fire, along with anyone who remained inside. They were very expensive (halon gases are not cheap, and you need a lot), and bad for the environment (as they damage the ozone layer, I think, techies, please advise). More modern ones reduce the atmosphere in the room to a low level of oxygen, suffocating the fire, but allowing people to survive. They are also very expensive.

    I don't know what they have in BT Newcastle, but there are lots of rooms, and any one of them spontaneously combusting could well cause an outage, and shutdown, if only to reduce the possibility of further damage due to electrical fires, metal fires etc.

    Reminds me of line from a Victoria Wood sketch:

    Posh, Southern Newsreader: "We'd like to apologise to viewers in the North, it must be awful for you."

    1. Anonymous Coward
      Anonymous Coward

      Re: Fire suppression systems

      Halon 1211 (bromochlorodifluoromethane) is/was super damaging to the ozone layer. It was banned in the UK (mainly) in 2003.

      Replacement systems exist, there are a few and they are less dangerous to humans in the area where the gas is released too.

      I'm sure most of us older techies have some sort of war story about "someone we worked with" that had some sort of run in with a fire extinguishing system in a machine room... or maybe that's just me ;-)

    2. The commentard formerly known as Mister_C

      Re Victoria Wood sketch

      Your comment was fairly accurate last night. BBC news app on my fondleslab had an article about a smoke cloud seen at a shopping centre that got updated to being a fire in a bakery. All dahn sarf. Nothing about this fire though.

      Feels like "the heathens can still send packets using their pigeons; we must alert people to a shortage of sourdough".

      <troll> I'm surprised they managed to spot an extra cloud in all t'smoke. </troll>

      1. John Brown (no body) Silver badge

        Re: Re Victoria Wood sketch

        "Your comment was fairly accurate last night. BBC news app on my fondleslab had an article about a smoke cloud seen at a shopping centre that got updated to being a fire in a bakery. All dahn sarf. Nothing about this fire though."

        It was on the main Beeb news site when I got home at tea time yesterday. According to my wife, nothing we weren't affected at all. But then we use VM for 'net and landline, both working without hiccup.

    3. Anonymous Coward
      Anonymous Coward

      Re: Fire suppression systems

      i've been involved with a DC build in the last 5 years. Modern DC's normally have a VESDA and use either FM200 (less common these days due to regulations from the EU) or IG55 as the fire suppression. I suppose it depends in what part of the exchange the fire was, even if it had fire suppression it the fire might have been outside of the area covered

    4. Scene it all

      Re: Fire suppression systems

      I was talking to the machine room supervisor at the place I worked back in the late 70's, and they had one of those Halon systems. When it was first installed, they tested it, presumably with something other than the real Halon gas. The supervisor told me that it is so vigorous that the incoming jets of gas would knock tape reels off of desks. The entire building shook.

      We had to be careful when snaking cables under the elevated floor to avoid stirring up the dust which might set off the smoke detectors down there. If two detectors smelled "smoke" at the same time, the whole fire supression system would activate. They had placed colored sticky dots on the ceiling just above where each detector was located so that people armed with floor-suckers would know where to stay away from.

  7. The First Dave Silver badge

    Just a question - how is an on-premise fire "Beyond our (BT) reasonable control" ??

    1. EnviableOne Silver badge

      not what MBORC means, its BTs five letter excuse not to pay compensation

    2. John Brown (no body) Silver badge

      Maybe it was 3rd party equipment releasing the magic smoke? That's what LLU etc is all about.

  8. Flak

    Openreach network architecture

    Exchanges host two fundamentally different types of infrastructure:

    Transit infrastructure:

    where the exchange is a node for transit - this is regional/national infrastructure with mostly diverse and resilient fibre and active equipment. Any one node disappearing off the network for whatever reason will have little impact on the transit infrastructure, as resilient routes will pick up the automatically re-routed traffic.

    Local access infrastructure:

    where all local access connectivity (analogue exchange lines, ISDN lines, FTTC/FTTH, fibre infrastructure) from the area that it serves converges on the serving exchange (there are some rare exceptions where resilient infrastructure is taken to another exchange - or an alternative infrastructure provider has independent local access capability).

    If the exchange is also a parent exchange for some neighbouring FTTC 'child' exchanges, then they would have been affected, too.

    This local access infrastructure is what will have been affected in terms of service by the fire.

    1. Anne Hunny Mouse

      Re: Openreach network architecture

      From experience BT's idea of resilience and diverse routing is not necessarily what you would think.

      Had 2 circuits for a network with diverse routing. Initially they routed via different exchanges but both went through the same exchange elsewhere in the area.

      Hence when said exchange had a fire (Denton Burn, Newcastle) lost both circuit and access to that network.

      Other circuits for other networks were properly diverse and only lost resilience.

      BT's response was we only provide resilience to the point of delivery, the circuits may share a common exchange on the route..

      Carliol Square exchange originates from the 1930s so may not but up to date for latest fire suppression

      1. JohnGrantNineTiles

        Re: Openreach network architecture

        So much for the Internet surviving a nuclear war.

        1. Why Not?

          Re: Openreach network architecture

          It would until the accountants got involved.

          1. Eclectic Man

            Re: Openreach network architecture

            When I worked as a security consultant at a well-known company, we did have several different grades of 'diverse routing' available. The top level was one where access to the general comms network was checked to make sure there was no single point of failure at all (very expensive). You gets what you pay for (as long as you read the small print).

  9. Anonymous Coward
    Anonymous Coward

    "restaurants prevented from accepting Deliveroo orders"

    How will the Newcastle and NE survive... no Netflix and no Deliveroo!!

    And they can't even take to Twatter to complain

    1. EnviableOne Silver badge

      Even McDonalds had to revert to taking cash....

  10. CJatCTi

    Numbers Ported to VoIP still not safe

    You would think if you had ported your numbers to VoIP they would have been safe. But as the BT exchange still takes the first part of the incomming call our customers lost their numbers they had ported away from BT.

    Is there way to escape BT?

    1. Alister Silver badge

      Re: Numbers Ported to VoIP still not safe

      Is there way to escape BT?

      Go and live in Hull?

      1. EnviableOne Silver badge

        Re: Numbers Ported to VoIP still not safe

        Our line that Died is Through KCOM,

        Kudos to OpenBreach, this time, they did a pretty good job of re-routing to get the whole lot done in the space of 3 hrs

        Still claiming MBORC for a fire in their own building, especially if it turns out not to be Arson, is a stretch

  11. Anonymous Coward
    Anonymous Coward

    A&A

    Another great example of A&A customer service. Kept up to date by their support page + email/sms when it was known to be a fire. Knew what was going on well over an hour before our IT guys at work issued a statement saying there was a problem and they were unable to say what it was.

    1. SWCD

      Re: A&A

      Spot on, same here! Can't use them at home any more (they don't supply FTTPoD), but still have some customers on there so get their updates via them. Twas indeed A&A who'd updated me before anyone else had a clue what was going on.

  12. Aussie Doc Bronze badge
    Flame

    Well.

    "We are currently experiencing a fire in one of our exchanges in Newcastle Upon Tyne."

    Did anybody else read that in Moss's voice as they sat at a computer and complain "No, that's too formal"?

    No?

    Just me then.

    https://www.youtube.com/watch?v=1EBfxjSFAxQ

    1. xpz393

      Re: Well.

      0118 999 881 999 119 725 3

      (Apparently, the post must also contain letters)

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2020