Virgin Media downed by Manchester arsonists A suspected arson at a Manchester electricity substation left thousands of Virgin Media customers in the North West without TV and broadband for several hours overnight. The Halloween attack in the Miles Platting area left 60,000 homes in the centre of the city without power at about 7.30pm. They were all back on the grid by …
There was a power cut due to the arson attack in the Platting in the morning but that was pretty small in comparison to the breakdown in the evening which hit a huge part of North Manchester, Salford, Bury & Bolton. Affected the kick-off FC United's game against Bamber Bridge and the Bitters against the Trotters as well, I believe.
There were also problems with the interweb: the Manchester Evening News site wasn't available from everywhere: not just from here (Germany) but also for Netcraft but this might have been unrelated and down to a more general problem with DNS over the last 48 hours which seems to have been resolved now.
Details
http://news.bbc.co.uk/2/hi/uk_news/england/manchester/7072236.stm
From what I hear the Miles platting incident was earlier in the day and may not be connected The incident that turned the lights off in the city was caused by an unspecified failure the other side of town a couple of hours later.
One area that did feel the full effects of the outage was Salford Quays which is home to a number of data centres. It is power failures at these data centres which will have caused the internet to go off line.
When the power went back on most equipment came back on however it all turned on at the same time and so came online out of sequence and in some cases some things didnt turn on at all. domain servers would have come on line before DCs and things came on in safe mode etc.
I personally spent all of this morning resolving issues for customers relating to this power failure ranging from firewalls with corrupted configurations, failed VPNs, missing DNS servers, absent databases. In one data centre we had to power cycle the whole of our hosting infrastructure, routers, switches, firewalls, servers, etc to bring it back online in the right order. On another site we had to turn most things off because the aircon had not come back up and the room was like a furnace.
The power outage appears to have been total in some areas with even street lights and traffic signals going off. In one building we look after 30 staff working late were stuck inside the building because the electronic door entry and locks systems were off line and cars where stuck in the carpark because the barriers couldnt be raised.
I have had better Thursdays
But then again , by blaming arsonists rather then properly monitoring maintaining or upgrading obsolete ancient overloaded equipment means they can claim on the corporate insurance rather then take it out of the annual dividend paid to the company chairman !
Very neat indeed , it is a miracle they have not put the finger on the helpless repressed religious minority said to be involved in the endless Tony Blair War on freedom and democracy ?
But I suppose the local plod armed with a few truncheon induced confessions may yet ride to their rescue for the consideration of a small bribe or two in the right place !!
As a minimum personally I would not consider it a proper data centre without full UPS/generator back up.
As I am sure most of you are already aware. The generator feeds the UPS on the essential circuit for the equipment and the air con units come straight off the generator via a ‘non-essential ‘circuit with air con starting in sequence to avoid overload as they fire up. So aircon ‘should’ only be down for a few minutes whilst the generator fires up and stabilises.
The temperature will increase quite quickly during this time but still within acceptable margins and only for a few minutes. The whole system ‘should’ also be tested with a full shutdown (not simulated) at least every six months.
“Admitted the horse has now already bolted but perhaps its time to look at installing a door and perhaps checking that it closes once in a while.....”
> "As a minimum personally I would not consider it a proper data centre without full UPS/generator back up."
You would think so wouldn't you? I was thinking along the same lines.
>"The whole system ‘should’ also be tested with a full shutdown (not simulated) at least every six months."
Not always possible when hosting 24-7-365 apps and services.
The other site with the air con problems was a normal office building and the ac was normal office ac controlled from the wall mounted unit. These have a habit of not coming back on after a powercut. The first sign that they havent come on is usually when servers start turning them selves off and someone goes to investigate.
No problem.
"Not always possible when hosting 24-7-365 apps and services"
Sorry above should be amended to:
"The whole system ‘should’ also be tested with a full <POWER> shutdown (not simulated) at least every six months."
We do this for both of the (24/7) hosting data centres we own (as well as a genset only test under load every three months) and all equipment keeps working (as it is designed to) but as this is a controlled shutdown so if anything didn’t work as it should you can restore power and sort it so that when/if there is a real power cut everything 'should' work.
The downside of the extra testing is that without the proper equipment there is a higher risk that when the power is restored equipment that has been taken off load and back on again will fail so it can be a difficult balance to reach.
Comfort cooling air con units need a lot of power to initially fire up so if they all fired up at once it may trip the main fuse (depending on the type as some are designed to ignore very short spikes) so it is common to either set them to start up a few at a time or set a trip to leave some off when the power returns. The problem with this is that it relies on someone being there and remembering to reset them.
If the first time they know that the air con is not working is when the servers start turning themselves off then much of the damage is already done. IP based single temperature monitors that send reports at high and critical levels are less than £300 so can be a good investment.
Your last word sums up the problem, investment. The company I work for (A very, very large American Telco) has many data centres, pop sites, switch sites etc around the country which have great monitoring facilities but are sadly lacking any investment in infrastructure so unless something goes bang in a big way we are purely reactive. I have 1 site in Manchester where the aircon cant cope on a mild day, the solution? get some fans in, not even portable aircons, just fans.
What we really need to do is replace the 2 ailing, aging aircon units which, when installed were just about adequate for the equipment that was installed in there, several years of shoehorning more and more kit into any available space has left it just about ready to burst into flames (Perhaps when it does we can have a new ACU?)
I am guessing that the company I work for is not unique about its investment policy, if it (just) works then there is no need to replace it.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
