Data centre outfit Interxion hit with outage at central London facility European data centre operator Interxion suffered an outage at its central London campus last night, with no service from its LON1 data centre for several hours and users complaining the company was silent about what was happening. In pre-COVID-times: Ely's Yard, behind the Old Truman Brewery building on Brick Lane that houses …
My hopes for an early night was dashed by this outage. The DC and networks teams figured out all was not well sometime after 1800. As the article states not much info was forthcoming from Interxion, I think we found out via a tweet somewhere what was going on and so a lucky volunteer went off to site. And took them quite a while to be allowed in.
I got dragged into all this about 2100 and spent ~3 hours trying to get our servers back up and running. Some of our servers really didn't take the outage very well and required some manual intervention of various forms.
There was no staged power on, so everything came on at once. Including the chillers/aircon which didn't have much time to get themselves up to speed before everything else. The result being things got quite hot before levelling out. But for a while we considered switching off non-prod and things we think we can get away without. Fortunately it didn't come to that.
I think we're still waiting for the official blurb/reasons.
UPS isn't responsible for changing over to generator, usually an automatic transfer switch is. UPS may of failed by running out of batteries before someone could manually change to generator. Maybe they aren't staffed 24x7 by people that are qualified to do that kind of operation(UPS should provide at least 5-10mins of runtime). Though the nature of the facility makes me think they would/should be. Certainly many smaller data centers don't always have on site 24x7 instead rely on on-call people.
Or maybe they use flywheels and didn't have the time to manually intervene.
DNS is so rarely the issue, but generally the less experienced like to blame it because they don't know any better (DNS admin since 1996). I recall a recent amazon outage some people blamed on DNS when it was the network that was saturated preventing queries from going to DNS(for their internal monitoring etc). DNS wasn't the issue it was a causality of the larger issue.
"usually an automatic transfer switch is."
A lot of this stuff is on cranky "Industrial ethernet" controllers which are fragile as all hell and prone to playing up
I have a half megawatt UPS on my watch which I haven't been able to directly talk to for over 5 years because the ethernet has locked up - I KNOW what's wrong and fixed the crash triggers a long time ago but getting manglement to agree to reboot it is impossible
Its compantion (800kW) had a similar problem but spontaneously rebooted 6 months ago after a similar period of downtime and "miraculously came back to life"
I'd been telling people what was needed the entire time and been shouted down - the "experts" from the vendor companies may know their UPS equipment and the Caterpilar engines but they are _NOT_ IT people(*) and never read the manuals or specifications for the interface equipment to know what takes these ($2000+) ethernet bridges out of service (Hint, never put your industrial equipment on the same VLAN as any other kit, they frequently can't cope with broadcast traffic and as we know painfully well. SCADA kit is frequently crappy as all hell)
(*) Seeing a field tech spit in a serial port plug to clean the pins is the kind of thing that really inspires confidence. I mean, really?? And you wonder why you have a corrosion problem?
I remember it as the Old Street Roundabout.
It wasn't built on sand in those days. It's doesn't feel like a roundabout now with the road layout changing.
There are still large open plots, bombed during WW2 and now surface car parks.
I'll report back in another 35 years time.
TTFN
'cos their power has gone off?
Wasn't one of the Docklands sites affected a few years ago because all their redundant power feeds entered the building at the same point?... one errant JCB wiped out the lot!
try having "geographically redundant data feeds" which run in the same duct for 3 miles, split out at the nearest town but then rejoin in another duct for the final 10 miles
Quality British Workmanship
Have a look ... bet you'll see a Cityfibre team laying cables in the ground* outside the site.
*I won't use the term "install" as that infers some sort of skill, care and being concerned enough to at least attempt to not disrupt other services.
I wouldn't even use the term "laying cables"
"Back hoe fading" is a constant issue. I wonder how bright the sparks were?
The problem with trenching teams is that you can have all the agreements you want about where things are laid and how careful they will be, but if a subcontrator passes it oiff to a labourer without that stuff (and they do) then this kind of thing happens.
There's no comeback on the subbie, so "I'm alright Jack"
Always ensure your contracts include personal liabilities for the person who's supposed to be supervising the job. Nothing focusses the mind so much as the possiblity of losing the roof over one's head
No.
That is the same thing as Cuil supposedly pronounced as "Cool". It is written Cuil, which is pronounced Ku-il, NOT Kool. And that is why they failed.
Writing is supposed to be clear. If they wanted to be pronounced as Interaction, they should have called themselves that. Failing that, they could have called themselves Inter@ction, or some other brain-dead version.
Interxion is pronounced In-Ter-Ksion. Period.
And they are not wrong. Data centre we have some stuff in runs 32A feeds with no additional protection straight into 10A IEC C13 connectors, so in theory you can have a server going into meltdown and pull 30+ amps without anything tripping in the rack. So what then happens is the connector plastic and insulation melts and bare cables start falling into the casing and if you are really unlucky, the main earth bond burns away and a live touches the casing so the whole rack goes live and the first thing you know about it is when the next techie touches the rack, and of course none of this stuff has RCD life protection so one day we will have someone die.
But hey, when I pointed this out they said “but it’s all tested”. Yes, all the way to the 32A connector, but no further. Did I miss the memo that said that we shouldn’t discriminate against faults any more?
Well, that doesn't comply with electrical safety standards in several jurisdictions.
Depending on exactly where this is, it may not be actually illegal but it is rather likely to invalidate insurance.
The wire is more likely to melt first, if that's any consolation.
Legally it is in that grey area as it isn’t part of the building installation and therefore isn’t covered by the Wiring Regs, but when I said it should therefore be covered by the Low Voltage Reg and CE marked (or whatever the Post Brexit Valhalla version is called) they just bounced it up to someone who had no idea what I was talking about who just said “if you have a problem put your own in”.
So yes, it is illegal but not enforceable by the authorities until there is an life threatening accident.
Funny story: Many years ago an event crew person got blown off a balcony of a very well known London hotel venue. Turned out the box that exploded was about the only thing still legal in the whole venue and they ended up shutting for months as they were served a prohibition notice.
I seem to recall that they removed redundant 1.5MVa power feeds from LON1 whilst we were there in favour of a single 10MVa feed because "grid power is so reliable now and our generators have always taken over when a feed has gone down previously" (used to happen a lot back when we used them). Of course something could have changed in that time as it was 10 years ago.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
