back to article Protection at last: Operation Emergency UPS succeeds for Telecity

UK internet hub Telecity has successfully completed emergency maintenance on power equipment at its Sovereign House facility. It was Telecity’s third shot at repair work in a week. Telecity has now, The Reg understands from sources, installed systems allowing for the safe switch-over of customers’ servers and storage to …

  1. Your alien overlord - fear me

    How about wiring them up to that new Russian data centre - the one built ontop of a nuclear reactor? That should keep the juice flowing.

  2. M7S
    Coat

    For all those Amazon services

    I guess Blackout Friday ended early

  3. kmac499

    UPS War Story

    Dunno if it's true but should be...

    In a Data Centre far far away in space and time (Watford many years ago). The data centre was having some work done in the car park. A JCB driver was digging a trench and hooked up the mains incoming supply. The Data Centre seamlessly switched to the UPS batteries and the techies sighed a huge sigh of relief as the well tested Diesel Gennies would soon kick in. Silence; No clouds of black smoke from the block house, and the batteries getting flatter by the minute.

    The reason for the silence??

    The trigger circuit for the generators and their supply to the building were in the same trench\duct as the mains power feed...

    There but for the grace go we all..

    1. Commswonk

      Re: UPS War Story

      The trigger circuit for the generators and their supply to the building were in the same trench\duct as the mains power feed...

      There but for the grace go we all..

      Not sure that I would have used the term "trigger circuit", but that's a small detail. You have nicely illustrated the concept of a Single Point of Failure and it is arguable that someone should have spotted what could happen. Did anyone advise the contractors that there were supply cables somewhere in the vicinity of where they were going to dig, and if not why not? And did the contractors have a formal Permit to Work? With the U/G cables in the car park being external to the premises then the cable may have been the property and responsibility of the DNO and they tend not to look favourably on people digging up their cables without checking with them beforehand to find out where the cable runs were. At the same time I'm not sure that they would have allowed the output from the generators to use the same trench at the time of installation.

      Always the remember the Rule of Ps: PPPPPP. There simply must be other El Reg subscribers who know what this means.

      (For the avoidance of doubt I am not having a go at kmac499 as there is no reason to suppose that he was in any way responsible; the points are rhetorical.)

  4. Mark C 2

    Engineer..

    "...accidentally introduced by an engineer"

    Very much doubt it was an Engineer. Perhaps it was an Electrician with the 'cian' bit on the end, just like Technician, for example. I bet if they had a real Engineer this would not have happened.

    An Engineer.

    1. Doctor Syntax Silver badge

      Re: Engineer..

      I take your point but I'm still wondering how an engineer or an electrician could accidentally introduce a transformer.

      1. Sgt_Oddball Silver badge

        Re: Engineer..

        I believe you can be have both in your job title (my father claims to be an electrical engineer, it even says it on his works office workshop door.)

        Though I know his place has 3 hv lines but had the disaster recovery tested that by having the basement of the factory 2 under water including its substation.

      2. nixel

        Re: Engineer..

        Rather than accidentally introduce a transformer, one could deliberately introduce a transformer in error.

  5. Anonymous Coward
    Anonymous Coward

    re: ups war story

    A similar thing happened at least once when the Shard was in mid construction. A digger going through the telephone/internet cables, only it wasn't a clean cut but a snag and of course you can imagine what happened to the kit the cables were connected to.

  6. Vince

    "throughout telecity have continued as usual by not communicating to people, including the press"

    There, fixed that for you.

  7. David Roberts
    Mushroom

    How many people have seen a backup generator kick in?

    Many (MANY) years ago I was involved in transferring mainframe workloads between data centres.

    This was before any major network bandwidth was available so we carried a van full of magnetic tapes. [Yes, in those days sneakernet was the only option!]

    Oh, and one of the TLAs [IRA] was active.

    Just as we drove into the data centre car park all the lights went out and there was an enormous explosion and a cloud of black smoke from behing the data centre. We thought the place had been bombed and were about to leg it (well, van it) when the lights came back on and we could hear loud diesel noises.

    The brief moment of sheer terror sticks in my memory, as does the cloud of black smoke.

  8. PNGuinn
    Mushroom

    Can we test it?

    Oooh goody - they've fixed it at last.

    Can we test it please, pretty please?

  9. Anonymous Coward
    Anonymous Coward

    Not sure if its just all fallen over again, but some link or other fell over at about 9.50pm friday night.

    our aws instances are suddenly struggling to peer connect again but aws itself seems unaffected.

  10. Sgt_Oddball Silver badge

    third times the...

    Charm as they say.

  11. Dan Paul

    Diesel Generators are NOT....

    Uninterruptable Power Supplies. BY Definition Uninterruptable means no disruption (in electrical power), so there must be a battery system and inverter system somewhere. Both of those items should have taken over long before the generators kicked in. Since they did not, then there was NO UPS. Alarm bells should have been ringing at that discovery. Gensets kick in when the main power cuts out or when the UPS is faulty. Both happen often enough that there is no excuse for being so lazy.

    There seem to have been multiple points of failure here, not the least of which was selecting Telecity as a provider. The next failure was believing Telecities hype.

    1. DigitalDisaster

      Re: Diesel Generators are NOT....

      Sovereign House uses DRUPS (Diesel Rotary Uninterruptible Power Supply), which is a combined diesel generator and flywheel UPS with a clutch arrangement.

      There are two independent DRUPS systems in Sovereign House (System A and System B), but I believe you have to pay extra to have feeds from both. System B is the one that has been affected by the recent power issues.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2021