One of our VOIP services was killed by this yesterday afternoon but it was working again this morning and it is still working now (18 November 13:33)
Telecity's engineers to spend SECOND night fixing web hub power outage
Telecity engineers will tonight make a second attempt to fix broken power systems at its critical Sovereign House internet facility. They will be working 24 hours after their first efforts failed and resulted in a second power outage. The internet service giant had planned to repair and fully fix the power supply in the early …
COMMENTS
-
-
-
Thursday 19th November 2015 11:25 GMT Anonymous Coward
Good nice to see they supply 5 9's for telephony, the carrier industry standard...Oh wait.
Due to a transposition error Telecity designed Sovereign House for nine fives, and as a result the UPS and backup power were an empty cardboard box with wax crayon writing that says "N+1 power backup". Luckily the finance department didn't make the same error, so customers were billed for a full fat resilient service.
Makes you wonder why the clowns of DCMS are committing £1.2bn for nationwide broadband rollout so that farmers can watch ovine grumble in HD, when a single data centre snarl up can mung 10% of UK internet traffic, for want of a sub £5m investment in a power system that simply works.
-
-
-
Wednesday 18th November 2015 15:30 GMT Alister
Amazon AWS EU-West1
We run some monitoring software on a VM instance in Amazon's EU-West cloud, which we use to keep a check on our servers which are in various datacentres around the UK. Yesterday at about 14:00 GMT we started getting buried in alert emails, as the AWS instance couldn't see any of our stuff in Union Solutions, 6Degrees, Node4 and various other providers. We ended up turning the monitoring off, and weren't able to start it again until past 19:00 GMT last night, and it's still a bit flaky today.
-
-
Wednesday 18th November 2015 20:51 GMT Lee D
Re: Cheap datacenter providers ......
Honestly, how long do you think a UPS will last, of any size?
And I've seen enterprise-grade UPS just switch off with full battery. They have to do so if they detect cross-phases, bad earths and other faults on the line or equipment or they will, quite literally, kill people and start fires. They are by no means a solve-all device.
There is no amount of money you can pump into something to build a resilient service. It's just not possible. Or we quite literally would have no need for backups, failovers, hot-spares, etc.
The people who are running services on JUST ONE DATACENTER are the idiots here, lying to their customers that they will stay up. There's a reason that Amazon datacenter has such a name. It's just one of very many, and you can migrate instances away to unaffected datacenters, or spin up your backup/failover instances.
But a UPS? Pfft. Even in a datacenter they are there only to level out spikes and hold things up until the generators kick in, and the generators can only work for as long as everything else is safe and there's diesel enough. One engineer on the floor who could get zapped, and you are REQUIRED to switch everything off. Proven by the guy who DIED in a datacenter only a few weeks ago because he tried to work live on the system because the datacenter managers didn't want to turn off the power.
Guess what the downtime is for a death once the emergency services arrive and tell you in no uncertain terms to make sure its safe for them for the entire duration of their investigation?
-
Wednesday 18th November 2015 20:57 GMT John Stoffel
Re: Cheap datacenter providers ......
All decent data centers have a UPS, it only needs to be online for 10 minutes at most before the generator(s) spin up to take the load. This sounds more like a transformer or two got fried somehow.
As for your claim about someone dying, care to post some particulars or point to a source for this claim? I don't doubt that it's hjappened at some point in time... but showing your sources before making claims like this is always better. Otherwise you get flamed.
Me? I'm just showing you the coals and saying "be a shame if they got dumped on you... accidentally like...". LOL
-
-
-
Wednesday 18th November 2015 19:21 GMT Anonymous Coward
Power problems
Shirley renting a BFG (as in Generator, not Gun) to park in the street, plus assorted power distribution equipment and a ton of extension cords would be a short-term crutch?
Granted, there are logistical problems like maintaining security and safety while running on temp power. Hiring security for access control and firewatch would get around much of that.
-
Thursday 19th November 2015 08:45 GMT Anonymous Coward
Re: Power problems
I worked for a company that tried to run a datacentre on a Generator from the street using lots of power extension leads. Didn't work out so well.
Fixed wall mount air conditioning is a requirement to keep it cool otherwise the air temp quickly rises over 50 deg C.
Portable Air con units also running on the Generator are not a solution to a problem caused by being a cheapskate in the first place......
Annon because who would actually admit to doing something so daft even if it was a LONG time ago....
-
Thursday 19th November 2015 10:09 GMT Terry Barnes
Re: Power problems
That entirely depends what the problem is. Sometimes the issue is getting the current from the generator to the racks. An outage of this length almost guarantees that it's not just a case of a blocked fuel filter or stuck changeover relays.
Most buildings of this type have existing external connections for a mobile generator and a reserved parking space all ready to go.
It's not something you can bodge because once the power's back on no-one will let you turn it off again for a long time so a long outage indicates to me that something significant has broken and is taking time to repair properly. That could be a distribution board or a busbar or something of that order.
If you bodged temporary power with extension leads it would be a) quite dangerous but also b) a permanent solution until you can agree a changeover back to normal power with every single customer affected. I'd trust a generator for a week or so, I wouldn't like to be relying on it six months later.
-
-
-
Thursday 19th November 2015 10:13 GMT Terry Barnes
"customers had paid Telecity for the use of dual, fully independent power suppliers to avoid outages "
If you don't have site resilience you don't have resilience - you have protection.
It's the right call for a lot of business needs though - it's a case of balancing the cost of achieving very high uptime versus the cost of being down. Most importantly, don't write contracts for your customers that promise availability that isn't at least equalled by the contracts you have with your suppliers.
-
-
-
-
Thursday 19th November 2015 18:21 GMT Anonymous Coward
Re: That accompanying photo...
The 1970's called and would like their ancient sexist crap back.
Out of curiosity, does this comment mean that IT articles must only be accompanied by pictures not featuring people? Or must only feature geeky/old men? What if the geeky/old men were black or look a bit weird or a bit camp? Any of that allowed? And what about unattractive women? Would Ann Widdecombe be non-sexist on account of her, well, appeal? And what if there's people who actually fancy her? They might be a tiny minority, but we wouldn't want anyone harbouring the wrong sort of thoughts?
-
Thursday 26th November 2015 21:12 GMT Anonymous Coward
Re: That accompanying photo...
"Out of curiosity, does this comment mean that IT articles must only be accompanied by pictures not featuring people?"
No, it means that an industry welcoming to skilled people of all sexes wouldn't have websites where men comment about women thus;
"She's pretty sexy. I'd be very happy to try and fix her critical power hub."
-
-