back to article Multiple customers knocked offline as firefighters tackle flames at Telstra's London Hosting Centre bit barn

A fire that broke out this morning at Telstra's London Hosting Centre (LHC) bit barn in the English capital has disrupted customers' services, with a fire crew called to tackle the flames. An email to clients, seen by The Register, was sent by SIP and hosted telephony provider Voiceflex that confirmed its hosted servers were …

  1. Timto

    Re: Meh

    Who knew that clouds were so flammable?

  2. anthonyhegedus Silver badge

    I’m glad my WiFi isn’t stored in that cloud. I hope the other clouds have room to tweet properly

  3. carl0s

    not-so-cloud

    To be fair, it's not just cloudy stuff. I manage a few on-prem Asterisk based PBXs, and the SIP trunks for connecting to the PSTN world have to go via a provider, who might have their kit in that data centre. As far as VoIP systems go, it's as un-cloudy as you can get really, I think.

    1. Jellied Eel Silver badge

      Re: not-so-cloud

      From memory, there were some.. issues around the building ownership (financial engineering <cough>) and the division of spoils following the collapse of PSINet. So Telstra got some assets, Cogent got the US, and GTT/Interoute the European network, Then some floors leased out to other carriers, including Reliance as a result of FLAG also going titsup and leaving it's transatlantic cable dangling there. But several large/strategic carriers having it as a core PoP & collo site for their wholesale customers. I've done business there in the past for capacity as there's been reasonable physical seperation & fibre availability. When a big PoP goes <pop> though, it's always a good test of capacity planning & management.

    2. FatalR

      Re: not-so-cloud

      Yes this is unfortunate.

      Also annoying that they only go through one data centre as well, apparently.

      Would be nice to see calls routing direct between PBXs but no one seems to care about this, and email is being killed off where possible.

  4. iron Silver badge

    > a supply room in a data storage centre

    All those shelves full of bits, up in flames! Lol

    1. David Lewis 2
      Joke

      It's alright, only the "1" bits will be affected, the "0" bits aren't flammable. So only ~50% data loss! Easily recovered.

    2. Flywheel
      Joke

      It depends if they stored the data in 7 bit or 8 bit format. One will obviously take up more space...

  5. KittenHuffer Silver badge
    Mushroom

    Waiting ....

    This story will probably be back in about six months in the 'Who, me?' section!

    1. Throatwarbler Mangrove Silver badge
      Flame

      Re: Waiting ....

      "I was having a discreet smoke when my pager went off. I hurriedly disposed of the butt in a trashcan filled with cardboard. Fortunately, I had thought to bypass the CCTV feed with a continuous loop a la Speed ..."

    2. Xalran
      Mushroom

      Re: Waiting ....

      And it will be featured in the 'On Call' section in 10 years or so

  6. msknight
    Coat

    " repeated calls by El Reg to its London office failed." - Anything to do with the reported outage, perchance :-)

    1. Anonymous Coward
      Anonymous Coward

      repeated calls by El Reg to ... failed

      Better than being met by a clatter indicating a dropped handset, a lot of muffled coughing, and some screaming along the lines of "aaaa! aaaah! I'm on fire! aaaa! Where? oh, god..."

  7. grumpymike80

    Well, that explains it!

    Hmm, that's probably why we can't build anything then - The customer's SCCM MP lives in there somewhere. Lets hope the water fairies didn't make things too damp!

  8. An0n C0w4rd
    FAIL

    DNS

    Are all telstra.co.uk's DNS servers dead? I can't see any of the pages that the article linked to

    So much for following best practise and having diverse servers

  9. Chuunen Baka
    Alert

    suspected UPS fire

    I love it when a UPS causes an outage. I remember one company's on premises machine room kept shutting down because the UPS battery was low. "I'm sorry I can't *guarantee* uninterrupted service so I'm going to interrupt it now".

    1. TeeCee Gold badge

      Re: suspected UPS fire

      The all time best I ever heard was recounted in the Netwire conference on Cix back in the day.

      This was shortly after "big bang" in the City and took place at one of the dealing houses now part of ${bank}. Their IT director put together a presentation showing that, while the mainframes were protected by UPS, the loss of the phones, screens, lighting (etc ad nauseum) meant that, in the event of a power outage, they'd be effectively unable to trade. He included figures showing that this would cost them £1m a day. Back in the eighties...

      The board approved the stripping out of a basement level and the installation of an online UPS to run the entire building.

      When the UPS went titsup.com, everything their IT director predicted happened. The only thing he got wrong was that his figures were conservative. It cost them over £2.5m a day. They were down for a week. He was so fired it was unbelievable.

    2. Anonymous Coward
      Anonymous Coward

      Re: suspected UPS fire

      Happens often enough that I like to have dual power supply servers. One supply plus into the UPS, one to utility power.

      Sure, I get "dirty" power from the utility instead of "clean" UPS power, but power supplies are pretty tolerant things.

      Maybe I'm just bitter after APC and their special cables (the ones that look exactly like 9pin serial cables, but aren't quite compatible).

    3. Eclectic Man Silver badge

      Re: suspected UPS fire

      When I worked for a well-known telecommunications organisation, in an old building, we once had a genuine fire, a UPS deciding it was time to join the choir invisbule so to speak. We had three fire tenders turn up and most of the day off. Oh and a police officer toured the building telling everyone to leave (it was not time for the official annual test evacuation of the site, you see, that only happened in the week of the August public holiday to save on disruption and to ensure that all 15 people sad enough to be at work then can get out in the required 3 minutes).

  10. grumpyoldeyore
    Flame

    Another Ups Fire?

    Two in 9 days

    https://www.theregister.com/2020/08/18/outage_london_internet_exchange/

    1. John Brown (no body) Silver badge
      Joke

      Re: Another Ups Fire?

      Are you suggesting this might be "cyber war"? Hacker attacking infrastructure and causing damage? Are they Huawei UPS? Or just run-of-the-mill explody UPSs?

  11. carl0s

    My customers are going to ditch me.

    I'm going to ditch voiceflex.

    Maybe voiceflex will ditch Telstra?

    If voiceflex would answer the phone, answer tickets, have some kind of failover, or a working call-divert from their portal, this wouldn't have been so bad.

  12. TeeCee Gold badge
    Coat

    Telstra has not itself acknowledged the fire on Twitter and repeated calls by El Reg to its London office failed.

    So that's both their internet connection and their phones down then?

  13. Snar
    Joke

    LFB @ PI LHC PDQ?

  14. Eclectic Man Silver badge

    Independent back online

    The Independent's web site, which has been unobtainable for most of the day is now back, ut strangely without advertisements, at least for me.

    Was this one of the sites affected, or do they use a different supplier?

  15. Anonymous Coward
    Anonymous Coward

    DEW u know halon IT was burning?

    These UPS-es sound rather hazardous to put in the same room / building as a farm of servers. Current travels easily, so why not make it go further?

  16. Anonymous Coward
    IT Angle

    Did they guy who put in the Telestra UPS also handle Equinix?

    These days, data centers are dropping like flies in London

    1. Jess--

      Re: Did they guy who put in the Telestra UPS also handle Equinix?

      You would almost think that these companies have furloughed most of their staff. Of course less people around means less chance of someone walking past a failing UPS and thinking that seems hotter than usual, we should check that it's ok.

  17. Anonymous Coward
    Anonymous Coward

    Usual story, datacentre installs generators/UPS's/static switches, all redundant and failover capable so the power will never fail, and what happens?

    The mains, with no additional help, is more reliable, and doesn't keep causing power outages.

    1. Anonymous Coward
      Anonymous Coward

      Depends where you are. In parts of London that isn't always the case. We were in a central London DC campus for 7 years and they used to email us every time the building had to switch to generator power due to a mains failure. We used to get 6 or 7 of these emails a year.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like