Stricken 2e2 threatens data centres: Your money or your lights The collapse of UK IT contractor 2e2 descended into farce tonight as its largest data centre customers were told to each pay £40,000 just to keep the lights on. Clients of debt-crippled 2e2 were told to cough up the cash to keep systems running until they transition to another provider, sources close to the situation have told …
Yeah well, that STRONGLY depends on the SLA.
Does it stipulate that in case of wind-down operations, customers may have to pay to defray any running costs plus VAT, may have to wait weeks to exfil their data from servers that belong to HP and to which they have no physical access and generally be up shit creek without a paddle unless they have a second cloud on retainer? (An even then they will probably want to nuke their first cloud by wiping disks etc. but that is another matter still)
Can anyone verify?
This is a 'poster child' for one of the many reasons why you do not outsource mission critical functions. It is also another horror story in my book of horror stories that gets used anytime a wet-behind-the-ears MBA type starts spouting "cloud" at WROK PALCE.
I keep telling those 20-something who head into the CEO's office trying top sell her on the "cloud" to do their homework. Most of the time, they haven't.
Exactly right. I was telling a customer of 2e2 they were at considerable risk some months ago. Not because of 2e2 specifically, or anything I knew about 2e2's situation, but because they had no, zero, zilch continuity/contigency plans that took account of a 2e2 technical, financial or corporate failure. They must be in complete panic-mode right now, this could conceivably bring their business down as well.
Try telling that to the wet-behind-the-ears MBA types that can not grok what could possibly go wrong when your cloud connection dies?
They like to think in big-vision terms, and often they are not able to see the underlying problems because they lack the ability to appreciate nuts-and-bolts issues.
From an insolvency perspective, this will have to go into a file of its own; not only in terms of how does one handle an MSP/hosting or services business Administration but also ineptitude in the lead up to Administration and beyond.
Rule is, if you outsource, you'd better be comfortable with your chosen partner and have solid terms and conditions backed by back-up alternative actions to by-pass disruption and avoid being blackmailed.
Don't ever recall this outfit handling any other IT insolvency.
Err... isnt it getting to the point where its time someone gives the public a list of who these affected customers are? considering it appears they're either about to have their budget blown to bits to the amounts of whatever the 2e2 administrators demand, or lose their data centre.
Seems the fallout from this is about to get a whole lot bigger.
What happens if half the big customers refuse to pay, I am guessing the £900k+ gets paid by the rest of the customers?
Funny that Channel 3 consulting who stitched up several NHS trusts with 2e2's services is still bragging about the fact on the front page of their website.
Perhaps Sussex Partnership are wishing they had spent more than three weeks on the due diligence of a £37m contract.
Every time something like this happens, as with the Megaupload fiasco last year, it demonstrates in huge 500 point Arial Black exactly why relying on The Cloud is a really, really bad idea. And the sooner everyone realises this and decides to keep their own files on their own servers the better it will be for the whole industry and the public as well.
This business of giving control of my data to others has irked me from day one, and anything that hammers this fact home to businesses and consumers everywhere is always welcome in my book. One day, people will get the message: you give control of your data to other companies only at your extreme peril.
Once again I have to say that this really just isn't the case. Why should a data centre be any less trust worthy than a company employee. There are countless examples of pissed off sysadmins destroying data and backups held on internal company machines.
If you want stuff in house recruit your sysadmins very very carefully. If you're going to outsource, pick you outsourcing suppliers very very carefully. It's as simple as that. There is nothing inherently more risk about outsourcing.
That is why you have a reliable DR set of procedures that take periodic copies of the data out-of-easy reach, and (importantly) that you control the process and the media.
E2E may have the best DR procedures around, but if they 'own' the media that the backups sit on, they are just as unavailable to the end user as the servers that they back up. This is why it is more dangerous.
I wonder how many outsourcing contracts contain clauses that immediatly revert the ownership of the backup media and documentation for the backup process to the end customer in case of insolvency of the outsourcer. Sounds like a good clause to me.
In the case of trusting your administrators, you should spread the responsibility to more that one administrator to make sure that it is performing properly, but this will allow for the situation that if someone trashes all the data on your live systems, you just invoke DR.
Of course, if someone is really malicious, and is allowed to screw with the systems for more than your longest off-site backup is viable, then it is still possible to destroy all the data, but you hope that there is no conspiricy and that such behaviour would be spotted. But you could say the same about the financial director, the pruchasing officer, or even the caretaker. It's not just IT people who could be holding a grudge.
I think some are missing the point, if you have outsourced your entire IT support operation then you don't have either the hardware or staff to keep backups.
The NHS and other public sector organisations using 2E2 need to wake up and ensure in future they carry out full due diligence, don't assume because they are on a government framework that they are sound financially because the GPS did a credit check on 2E2 in December and there were no issues reported.
They also ensure suppliers have insurance in place to mitigate against this.
I quote from my original comment.
"and (importantly) that you control the process and the media."
I think you just made my point over again.
My response to "Dr Who" was actually about rogue administrators in an un-outsourced IT department being as big a risk as an outsourced operation going bad. But I tried to make it a little relevent to the original story as well.
"If you're going to outsource, pick you outsourcing suppliers very very carefully. It's as simple as that. There is nothing inherently more risk about outsourcing."
Well, there is. SImply put, putting your data in the hands of a company that you have no control over is a lot more risky than if it was in your own hands.
I upvoted you, because you made sense.
How in the world could someone who is supposed to be educated (possessing MBA degree) could not possibly foresee the havoc the crash of an outsourcing provider, especially one providing mission critical functions, can be anything but a disaster for a company.
I do now understand what motivates such deranged individuals - executive bonuses for incompetent management manglement.
I disagree, and see this as possibly an over-simplified reaction (for the record, no I'm not with a hosting provider). Businesses are progressively realising that what they are strong at is running their business, whether it's providing products or services, not necessarily running IT organisations.
That's not to say that this use case is suitable for everyone, but the ability to scale elastically on-demand, pay as you use, and leverage broad infrastructure together with a number of diverse and specific skillsets without the millstone of regular and heavy capital investment/maintenence, can provide businesses with improved agility, or simply allow them to focus on what they do best.
Businesses are progressively realising that what they are strong at is running their business,
"Running a business" includes everything from paying the electricity bill to keeping the works canteen going. IT comes in there somewhere along the way. What managers focus on is management and if they are not broad-minded and intelligent enough to properly delegate things they might not understand themselves to people who can do the job, they should not be managers, let alone directors and CEOs.
So 2e2 dont even own the assets they are hosting their clients services on!
IT Directors of the world - you have been warned. Mind you thats assuming that you genuinely have the strategic interests of your business at heart...and not just the desire to make your bonus at any cost and sod off elsewhere while watching the flames grow.
Outsourcing isn't necessarily a bad thing for all businesses. Cloud infrastructure (hate that phrase, personally - should be much more akin to utility based computing or some such phrase in my personal opinion) is also not necessarily a bad thing.
But as others have pointed out, failing to perform due diligence (surely the amount of turnover, profit posted and debt were readily available to these customers? And surely it would be clear that it was a house of cards built on a bed of shifting sand?) AND failing to put contingency in place is business-suicidal.
I remember doing some troubleshooting in one of the datacentres of one of the companies that 2e2 bought some years ago. The staff at the datacentre were very pleased with all the alternatives they had provided - alternative electricity supplies from multiple energy suppliers, a standby generator and UPS's, alternative points of access to the internet via multiple exit points within the datacentre but still only a single data centre for an awful lot of the customers.
I've lost count of how many times I've had the conversation about simple things such as cloud backup. Great. But, for example, only if getting that data back down a fairly narrow bandwidth internet pipe isn't a concern. Oh and what happens if that backup company goes under - how do you / can you get your data back?
Like many projects, this comes down to customers offloading their responsibilities for what they perceive as benefits - usually in cash terms - without sitting down and talking about it, and more importantly, thinking about it - 'it' primarily being worst-case scenarios.
I'm curious about one thing - it's been a while since I had any kind of involvement in offsiting to datacentres but close to a million pounds to run for one week? Seriously?
What a farce and hopefully a wakeup call to those people that think outsourcing everything to a single supplier without any contingency is an awesome idea because it saves cash. At least our hosting provider has insurance in place to protect us against this kind of failure happening to their business, its something we require from all our major suppliers and I am amazed its not for others!
As an employee of a large company who has a significant amount of kit in their DC, I was privy to a conversation yesterday where our head of IT finance came in and said in no uncertain terms that part of the business was ****ed.
I am absolutely certain the legal wheels have already started turning - although I'm not sure I see the point.
Nevertheless it's not as bad as was made out - thanks to the fact that we've just employed our DR strategy for that site.
Would have been nice to have a little more notice though!
Keep beating the drum - Business Continuity Planning
It's more likely a data centre owner will go bankrupt than have a plane fall on the building, therefore the plans should already be in place (if not in motion) to get out of 2e2 (or any other data centre, building, POD, tent).
So you're sitting back smug because you don't use 2e2? Suggest you get out your BCP and your list of suppliers and you see what your contingency is if you lose any one of them.
As well as the £960k demanded to keep the data centre running until the 15th the admionistrators' letter states: ( http://www.2e2.com/docs/jacqui/letter-for-non-data-centre-customer.pdf?sfvrsn=2 )
'As you will understand, we have received a number of requests from customers seeking to gain access to their data
immediately. Unfortunately, the levels of data held in the Companies’ Data Centres are such that this process could
take up to 16 weeks and we will need to ensure that the integrity of third party data and security is maintained.'
That is a lot more money yet to be demanded. I assume that as the numbers of customers with data declines the demands on the remainder will go up.
This is a haemorrhoidal clusterf**k
2e2 have treated their staff like an expensive overhead since they tried outsourcing our jobs to India 3 years ago.
2e2 have not understood that their biggest asset is their staff instead of their wallets.
Thats why you are all drowning at sea and 2e2 are rowing away, laughing, in the last of the lifeboats weeks ago whilst deciding it would be best to tell you two days before payday.
They didn't even have the bottle to tell you face to face, like last time
2e2 don't give a toss about the staff. They are asking you to work for nothing so they can wring out a bit more profit.
Having worked there and left, along with many other good technical guys I know how cold and un-caring 2e2 can be.
Thats because the original concept, thought up by engineers was taken over by accountants who do not understand anything technical except how to put everything on the slate and run away from their responsibilites when it goes tits up.
Good luck to you all and I hope you all find something quickly. O2 seems to have their head screwed on the right way at least.
Good luck - and *rseoles to 2e2
..Look on the bright side, whatever you get it can't be any worse :O)
Total Staff Related Costs 253,336
Operational Costs Cedar licencing costs 26,087
Microsoft SPLA licencing 22,382
Other Software Support 28,011
Connectivity 41,258
Newcastle 3rd pty data centre 21,028 <--- Why not just find out who this is, and pay them to keep your data?
Data Centre R&M 11,714
Customer cheque printing 2,759 <--- So I'll carry on paying you to bill me :)
Data storage (tape) 1,488
Offshore apps support 1,484
Other Costs 5,904
Service Credits 1,526
Total Operational Costs 163,643
Sub Total Costs 416,979
Datacentre Costs Gateshead (Rent, rates, utilities) 5,471 <---keep this one
Reading (Rent, rates, utilities) 27,784 <--- shut this one down
33,255
Central expenses (including a provision for the Administrators' costs and legal expenses) 150,000 < SHIT THE BED!
Grand Total Costs 600,234
Central expenses (including a provision for the Administrators' costs and legal expenses) 150,000 <!!!!!!!
So, adminstrators can actually charge for extorting money from the end customers? Parasites like that dont deserve to be paid. Silly customers who outsource all their data and don't have contingency plans deserve a fair few things, but not to fall victim to extortion from people who make their living out of the misery of others.
Administrators are bound be certain legal rules. They get paid for following those rules. Without administrators there would be no access to this data. The company would have went insolvent rather than into administration and no one would have a chance to get access to anything bar the tin (which would almost certainly contain comprimised data) which would no doubt end up in court with law firms getting paid even more to sort it out. I does seem like extortion but the alternative would be worse. If I'm reading correctly the alternative may still happen. That is switch off, walk away and have a firesale of the equipment at an auction.
Clouds are a bit like banks. When they burst, everyone wants to withdraw at the same time. And then you realise it isn't actually possible to withdraw all your money/data at the same time as everyone else. In the case of clouds it actually costs them more than normal for their links to be hammered 24/7, so even if you're prepared to wait, getting your data out is more costly than just keeping going. Maybe there should be a clause in the contracts that triggers extra keepalive payments when insolvency is declared.
On second thoughts, there's always the bulldozer method to retrieving your data...
I've had to migrate one of 2e2's customers today who received one of those pre - legitation invoicing (akin to ACS:Law) demands for £4k ... there were quite a few 'choice' phone calls ....
Plenty of IT Managers fretting about their own jobs because of silly decisions....
So a quick thought if you may ... for the developers who had a lot of stress over the past 24--48 hours, and a big thought out for the 2e2 employees.....
Ironic though that 2e2 service desk were hiring new staff some 4 weeks ago!!!
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
