Telecity fix nixed: Borked UK internet hub 'had no UPS protection' Telecity cancelled its plans for a second shot at fixing the stricken power supply in its Docklands internet hub at Sovereign House. Engineers for the data centre operator were due to try fixing the broken system again on Wednesday night/Thursday morning, sources told The Register. However, The Reg has learned, these plans …
Anybody remember them? The pioneers of the $99 dedicated server with no contract? Bringing DataCentres to a whole new value/budget market.
They had a power grid outage. The substation serving their datacentre actually blew up leaving them off the grid for about a week. And none of the operators of the 17,000 servers ever noticed. But they did get the full de-brief a little later. How the UPS silently took the load, how the two diesel generators powered up to provide the power. How one overheated (because they had only tested one at a time and the ventilation of one affected the other) - but they got carpenters to build a new whole ventilation system before it was lost. How they were having to get two tankers a day into refuel.
All for $99 about ten years ago. Somehow I had expected at least something as professional today in London as opposed to the Texan outback. Oh happy days!
* There was one other problem - their telephone system wasn't on the UPS so headsurfer and crew had to drive into the hills to a place their mobiles would work to call in help. But then few disaster recoveries are truly perfect ;-)
The headline says the borked uk hub had no UPS protection, the article appears to say they are awaiting part for the UPS so which is it? Looks more like they had some sort of serious power issue which fried the UPS and it is now running on mains bypass until they can get some parts.
And (I would have thought) an extremely expensive one at that.
Am I the only old f*rt around who can remember that the (then) Post Office used to have Continuity Sets to maintain service at critical network points in the event of a supply failure?
It may or may not be my age but I go all misty - eyed at the thought of a nice big diesel generator running somewhere.
<sigh>
"(then) Post Office used to have Continuity Sets to maintain service at critical network points in the event of a supply failure"
That was on top of everything running off float-charged 50V batteries (where battery can bean anything up to "multiple arrays of cells with total volume more than matching most small houses")
In general you only needed to start the genny if the power was off for more than 15 mins.
My personal pets were ancient dual 300kW mirlees diesel sets in a TX station (bottom of http://maritimeradio.org/himatangi-makara/himatangi-radio/ - in their better days)
Just thought, that would mean the staff there would also not be able to go home :(
Originally I thought it's about the only place in London where storing a couple of inflatable boats could be a viable part of emergency plans, but then I realised I don't know the flow rate of the Thames. Would be rather annoying if staff had to be rescued from the Thames Barrier :)
(I realise it may show that I know zip about boats, ships and other floaty things on account of getting seasick if I just *look* at one of them - curiously, not a problem I have with flying things).
Well most tech companies would expect their staff to connect remotely if they couldn't physically get to the office. Looks like this might not be a viable option for Telecity though as a prerequisite would be a reliable internet connection.
"Its vulnerable to potential Thames flooding"
No it isn't.
"obvious terrorist target so if something did happen there and staff couldn't get to work"
As somebody who used to live right next to it there's *serious* security around it. The only way you could protect it any more is by parking tanks outside and burying it 40 meters underground which isn't exactly viable anyways.
I hope you're not thinking you're the only one who's ever considered terrorism on the peninsula before - hell it's like 100 meters from the site of one of the largest bombs ever set off on the mainland by the IRA - the place is a fortress; and not for nothing all the stuff that's there is precisely where it needs to be which is precisely why it's there in the first place.
"No it isn't."
Yes it is. The Thames Barrier has already come close to be overtopped a few times and unless they
hurry up and build a new one the risk just keeps getting greater.
"As somebody who used to live right next to it there's *serious* security around it."
Well I work near it and I can't see this "serious" security you're talking about. Unless they're snipers
hidden in the trees. Anyway, I'm not saying the terrorists would try and take the place over, all it
needs is staff not to be able to get in for a few days (any staff there would be evacuated) and an issues
start to go unresolved and then it all unravels.
"s precisely where it needs to be "
Really? That would be why most of the big banks have emergency back-up sites outside london
I suppose.
Yes it is. The Thames Barrier has already come close to be overtopped a few times and unless they
I lived right next to it when it was at the highest level ever recorded and I can assure you it was never close to overtopping - and even if that happened it would be limited volume. You're confusing total breach with some water goes over it and you're just wrong anyway; it did not happen there was over a meter to spare. It probably needs replacing in the future but that date isn't now and there's no immediate threat. It's being closed more frequently but that doesn't equate to risk of actual flood merely risk that London would flood if it wasn't closed - and even then it's precautionary and have you seen how raised up above the water line the peninsula is at high tide? There's no risk here and the flood risk maps confirm this.
The barrier is closed at low tide so there's enormous capacity for extra water even if it did overtop - but again that isn't going to happen within a significant period. You know the barrier isn't watertight right?
Well I work near it and I can't see this "serious" security you're talking about.
Well then you're not paying attention - I'm not going to sit here and explain them for fairly obvious reasons. Telecity could probably do more to their own premises in the area and the government should probably directly help them with that but the peninsula itself has various passive and active protections that you'll see if you're paying attention. Plus frankly who only uses a single site anyways.
which looks in better shape than that latest Bond movie suggests :)
Spoilers. :p
"Telecity could probably do more to their own premises in the area and the government should probably directly help them with that but the peninsula itself has various passive and active protections that you'll see if you're paying attention."
That somehow has the ring of 'private army' - possibly with enough armed personnel to occupy Paris? Okay, bad example...
which looks in better shape than that latest Bond movie suggests :)
Spoilers. :p
Nah - the authors took a dislike to it during Skyfall (I guess they were not allowed to film there :) ) so it's simple continuity. I'm in total awe of the realism, though - I had to go and check afterwards just in case something happened while I was abroad :)
"Well then you're not paying attention - I'm not going to sit here and explain them for fairly obvious reasons"
A raised ground floor to prevent vehicle incursion, CCTV, internal barriers and heavy doors, right?
A group of terrorists with guns & high explosives could be inside in 10 mins.
there's *serious* security around it.
It's not that serious.
Years ago, while "in-between" proper jobs, I had a spell driving vans full of rack kit to Canary Wharf. It was a big van - a Merc Sprinter with a Luton body - and was generally pretty much full. I would drive in at least once per day
There's a big police line I had to cross. Occasionally, they would inspect my paperwork. Not once did they actually look in the van to see if I was carrying what I claimed I was...
Vic.
"A source told The Reg customers had paid Telecity for the use of dual, fully independent power suppliers to avoid outages but the fact their service was down indicated they hadn't actually got what they'd paid for."
Who'd have thunk that from an internet company?
Well I'm shocked I tell you, totally shocked!
All those sysadmins who wholeheartedly support colo hosting and decry the cloud have gone a bit quiet.
AWS had intermittent packet loss (enough to disrupt interactive sessions but not fault-tolerant protocols) but their scale lets them work around within a few hours and their customers could even have relocated to Frankfurt, it's the SMEs with their own kit that are stuck in Sovereign House.
As long as that data remains in EU hands I'd be inclined to agree with you.
If, however, you are obligated to keep data close to your chest you'd be up the creek without the proverbial - I think it's not unreasonable for people to expect that a company has actually done its job, so unless their contract specified that outages are just tough luck I suspect this could get costly in compensation claims.
what is this bullshit you are spewing again? amazon suffers from power outages too
http://www.theregister.co.uk/2012/07/03/amazon_outage_post_mortem/
"But despite all the precautions – and there were many – the US East-1 region of Amazon's cloud was brought down after an electrical surge.
And not because of the power outage, but because of faulty generators and some bugs in some of the control software that Amazon has created to deal with failover and recovery of various cloudy services."
---
I've had equipment hosted at one data center that suffered a power outage(I mean equipment failure which resulted in an outage of both power feeds at the same time), one data center in the past roughly 15 years. Moved out of that data center shortly after(I was a new employee at the time), 2-3 years later that facility suffered a large 30-40 hour outage due to a fire ( http://projects.registerguard.com/csp/cms/sites/web/news/cityregion/16561418-57/story.csp )
"The moral of this tale though... ...you need both redundant UPSs and redundant backup generators."
I'd say not really. If you can't afford downtime the moral is that you need site resilience.
Buildings can be affected by more than power outages or other infrastructure failure - they can flood, catch fire, there can be rioting, disease, hurricanes - if you want to stay up your n+1 needs to be in a different building, a different city is better, or if you can afford it a different country is excellent.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2025
