EDS mainframe goes titsup, crashes RBS cheque system HP managers are reaping the harvest of their deep cost-cutting at EDS, in the form of a massive mainframe failure that crippled some very large clients, including the taxpayer-owned bank RBS. An IBM Z10 at EDS's Stockley Park site, west of London, fell over this week after vital microcode fixes* had not been applied, because …
Similar thing happened - all "consultants" were fired at once. Then someone read the contracts realised that all the major contracts have a fixed amount of consultancy included and they had to *beg* the now embittered consultants to please come back and work for us - people had been frogmarched(some beaten!) out of thier offices by hired thugs.
Failing to appreciate that experienced, skilled workers who know the business are an information processing company's only asset of any real value.
Ladies & Gentlemen a round of applause to our cost cutting management consultant overlords -
http://www.zeroplayer.com/images/matrixManagement.jpg
...to catch up on the back-log. If the box has been stuffed and has little spare capacity, there may not be enough left to catch-up and carry on - either with RBS or the other users of the tin.
I guess there's more to this than meets the eye? Or is it literally "nobody in EDS knows how to apply patches to the machines?"
More likely, nobody who knew that there were patches that needed to be applied, or perhaps nobody who appreciated that the consequences of cutting this particular corner would be certain catastropic failure. If someone had appreciated the need, I'm sure IBM could have supplied a man to do the work (for a price).
They'll be able to catch up on 12 hours downtime over Xmas. Worst case they pay people to work on Xmas day, though I'd guess that one of the other bank holidays will suffice. That bit of the story makes me think that they let go the man who did the capacity monitoring and planning as well, or deliberately cut everything too close to the bone to postpone the inevitable during the run up to the complete failure of RBS as a bank. Fred "the shred", remember?
Idiots hate people who are more intelligent than themselves. Most bosses are idiots. This goes a long way to explain not just this story, but the entire banking crisis of 2008-9.
you can guarantee that the process is well documented. More likely the person receiving the bulletins about the critical firmware updates was amongst the people who was let go. Also, HP were probably doing the hardware maintenance, and you can guess how good they would be at that!
Does anything about HP/EDS surprise us any more. Bunch of muppets
Support staff are always the first people to go - the morons up at the top don't understand that things don't break because their support staff do their jobs. They see that their systems don't fall over and so obviously all those support staff aren't needed.
I remember once having to justify my team of 4 DBAs to my manager. He said that we'd only handled one call during the week so how could he justify paying for 4 DBAs. I pointed out that it was because he had 4 hard working DBAs that we'd only had one call all week.
I don't think he really understood.
I see idiot managers make choices like this all the time.
Another example is this true story that I witnessed.
At this bank where I was just hired at I got the run down and saw a server guy let go for being an idiot. As the story goes he really did mess up things and things were well always having problems. The next day they hired a new network server guy for the bank and things slowly got better and so good that soon he just sat and read tech manuals all day and studying for more certifications to add to his expertise. Well no sooner than 3 weeks went by that a manager who was not his manager stopped and commented on that he was sick and tired of seeing him reading books all the time and not working and that he is going to be fired.
Well that manager did get the new awsome server guy fired and well I brought this up with my manager and I got that manager fired for letting go a perfectly good employee who did what he was supposed to do.
Lesson #1 If nothing is wrong with your network then nobody needs to be fired.
Lesson #2 if something is always wrong with your network and the server admins are always running around with their heads cut off then they need to be fired!
Although Lesson#1 is desired let it be known that even Scotty from the Star Trek enterprise always knows when to sass up his work duties to make it appear that he is busier than it appears to be and that is what Lesson #1 type people need to learn.
No, stop, please, my sides are aching too much. What a fantastic advert for the competency of HP and (ex) EDS management that they managed to get themselves into this position.
How can such ineptitude inspire any confidence from existing unaffected customers? How can it do anything than completely destroy their ongoing sales pitches?
As somebody who used to work for EDS I realised quite quickly that the company didn't give a jot for preserving and building on the skills of their engineering staff. All that mattered to senior (I stress senior) was meeting the next deadline and if it worked, so much the better. Middle management, good and bad, get trapped between impossible targets from on high and teams of people who know enough to realise that they're onto a loser, or know so little that thay didn't realise the job couldn't be done.
Remember, the lowest bid isn't always the best, contracts should specify minimum head count and require specific skills/qualifications/certifications to be held. No, you really shouldn't have to have this kind of thing in writing but in this day and age the bog corporates would offshore everything to a $1 a day workforce halfway around the world if they could get away with it. No ethics, no consideration for the workforce, just the end of quarter targets and absolutely nothing else matters. Your shareholders won't thank you for taking the cheapest bid when it all goes tits up like this...
Kipling's poem "Tommy Atkins" applies, although IT staff may not be quite as mission-critical as soldiers. The operative lines are these:
'...it's Tommy this, an' Tommy that, an' "Chuck him out, the brute!"
But it's "Saviour of 'is country" when the guns begin to shoot...'
http://www.faxmentis.org/html/kipling.html
I really, really hope RBS sues the arse off HP - and doesn't agree to settle out of court. The useless bastards deserve to be taken to the cleaners, hung out to dry and never see another penny of public money for as long as they continue to exist.
I'm going to add, "MBA and nothing else" to my personal skills blacklist for next time I'm recruiting.
Pic of someone rifling through someone else's pockets, as we've no lawyer icon.
is just a file transfer system, somewhat like FTP, but with a list of trusted nodes that it can connect to. Any cheque clearing software wouldn't be based on it, at best it would be a simple source and destination for data files.
Why not having updated microcode would suddenly stop file transfer software from working is the more relevant question. More likely the version of Connect:Direct was updated and it wouldn't work with the old microcode. This kind of problem should have been spotted in testing, and at very least they should have been able to roll back quite quickly to the older version.
It is insane for a large IT intensive business, like a bank, to outsource its IT, it loses control over its business, as we see here! Management heads should roll, and the IT should be in-sourced again!
Banks may not like cheques, but they are the only option in many situations, particularly for large sums of money, and where people refuse to get stung by card transaction charges!
http://www.theregister.co.uk/Design/graphics/icons/comment/fail_32.png
EDS & RBS really haven't a clue.
It would be best if over time both companies could be run down and the business transferred to companies which understand their businesses, care for their customers, and are not driven wholly by short term gain.
P.S. Off-topic but to suggest that banking is a profession is laughable.
"Previously the updates would have been applied by the Stockley park hardware team, who have all been made redundant"
I suspect this isnt actually what happens as applying MCLs is actually done by IBM (unless of couse they don't have a support contract ) Of course the hardware teram would have been involved, in arranging the work and coordinating with IBM etc..so if they werent there then yes there would be a big hole in ownership of the process..
From everything I've seen of the way EDS have treated some loyal hardworking and essential staff at my employer who were unceremoniously sold off to EDS (now HP), the only surprise is that stuff like this doesn't happen more often. Morale must be so low that anyone with any clue and hireability (in the form of good solid experience on their CV) must be jamming the exits in their rush to abandon ship.
Pint because remember: alcohol makes the managers go away.
I would love to have been a fly on the wall for this one!
I can here the conference calls now, the flapping of the incident management chimps, client management on warp factor BS and all for the want of a techie who knew what they were doing!
As I have been told before, I am a COMMODITY, my skills can be sourced form anywhere! Ha ha!
Did you try switching it off and on again?
Hate to be pedantic, but:
1. IBM applies microcode updates - not the customer
2. IBM promotes the z10 as supporting concurrent microcode updates whilst continuing normal operation
If z10 mainframes are to run multiple workloads across multiple clients concurrently then they have to support concurrent microcode upgrades to avoid mainteance outages across multiple customers.
It may be that this was one of the non-concurrent microcode upgrades, but if it was that vital then IBM should have forced HP to apply it.
It may be that HP had not scheduled the microcode upgrades because of their customer requirements - especially at this time of the year with the Christmas/year end change freezes.
If seeking RBS and HP to comment, perhaps IBM should also be asked to comment on the non-stop capabilities of their mainframes?
The zSeries is 98% concurrent upgrades. IBM can NEVER dictate when I client can do their upgrades. In the several large environments I have run, I would never do an upgrade (concurrent or not) during normal hours. In addition, most such upgrades need to be checked for compatability with other upgrades AND, more importantly software levels (very time-consuming in most mainframes environments with hundreds of thirdparty software products.
There MAY be extenuating circumstances here, but it definitely sounds just like the article says: HP canned the EDS staff that did the planning and change control for such changes, so it didn't get done, and KBOOM. There is one question I haven't seen asked: if the upgrade had been done, would the outage have been stopped?
its about time one of these deals actually went bad enough to get some press. At the last place I worked when outsourcing was done the mainframe old timers had to "knowledge transfer" to the fresh faced graduates from India who had never worked on mainframes before. Cynically the company concerned kept on a small core of techies who knew how things worked. How I used to smile as they were kept busy fixing bugs created by the cheap labour.
For some reason people who know their stuff always appear to appear in the wrong way to those who do not know their stuff but have the power to decide who stays on the payroll.
People who know their stuff seem to be slow, happy, content and still worried in a funny way. They have this dangerous habit of not agreeing with the power and worse still they are too honest (or too stupid) when avoiding a straight answer to a complicated question.
They lack the ability to laugh or laugh at the right time in the political sence.
People who know nothing will mostly employ people who know less.
This is one of the problems with big companies. Small "men" are attracted bye big companies.
(the extension, you now).
There are exceptions I hope.
One, or actually the best boss I ever had, had this habit of asking us how things where running.
My habit was to say, well quite fine BUT we have this problem with that and that customer and this and that software and this and that time schedule.
Then, one day, one of my programmer colleges came to me and said.
Look, Lars, the guy is looking for good news, and I always tell him about happy customers, fantastic progress and great wins.
And well, I think he was right, while I was honest.
And I also remember my boss in one of the biggest Scandinavian IT companies, in charge of web programming, who came to ask me.
What is PHP and what is Apache. Warmed my hart that he would ask me, but then again!.
He was also the guy who laid me off about six months later. Did my face revel too much, I do not know.
Myself and some other guys were tasked with modernizing and upgrading a point-of-sales system for a company. No sooner than we had it all worked out than they let the lot of us go all at once. Of course no one else had been trained to maintain the systems and even the documentation was not done yet. I still hear from people I know who work there that they never figured out how/why things worked they way they do to this day.... Sounds like they had to spend a lot of money again to replace the one we made since they can't maintain it...
<smirk>
Couldn't happen to a nicer company. Ever since Fiorna had taken over HP, the whole strategy was "Oh if we fire all these people we'll have higher profits this quarter. Who cares about the future!" And apparently after she left they've kept on the same path.
Cut R&D -- after all that just costs money. Who cares that they'll have nothing to sell in a few years.
Cut "in-country" phone support for Indian phone support -- who cares that they had all these contracts specifically due specifically to not having outsourced support. It saved SO MUCH MONEY RIGHT NOW.
Now this. Given the astronomical failure, I wonder what the penalty is for allowing *two* mainframes to fail. Probably also astronomical. I will be amused if now that HP gutted EDS, if they pay more in contract penalties and contract cancellations than they "saved" by canning everyone. If I had a contract with them I know I sure as hell would not go with HP, knowing they fire people they have to have to meet their contractual obligations.
Hold on a sec, didn't IBM spend the last three decades telling us that NEVER happens? Oh, you mean all that was just Big Blue's marketting male-bovine-manure! I suppose maybe now RBS will look to a more modern and cost-effective solution like NonStop (what, you think hp may have wanted that all along? - shame on you for even implying that!).....
Big Blue can only notify people of microcode updated and if the customer doesn't have the knowledgeable staff to install them, then "crap" like this happens.
Now how big is HP's cost-cutting move? Hope RBS sues the eyeballs right out of HP.
Get rid of knowledge and you pay the price.
Merry Christmas HP
The were getting new offices done and moved their currency dealing operation into them first.
One day someone dropped a spanner acros 2 phases of the offices incoming 3ph supply.
High reliabilty (special lecy contract) trips out.
Building wide UPS battery room batteries were not charged so did not cut in (on the commissioning todo list)
Backup generator not fuelled yet (on the commissioning todo list).
Result. Bank Director *runs* up 4 flights of stairs to site offices. Demands for lost earnings compo
Annon because it could have happened to anyone. and smilly because I still do.
I landed a contract to install two big, garage sized, Memorex tape backup robots at a large number-crunching outfit once. Before I bid on the job, the VP of operations gave me the grand tour. He was proud of all his redundancy. He had two power lines coming in to two separate rooms, with a motor-generator, a large battery consisting of dozens of telco-style lead-acid batteries, a generator, and monitoring systems for each room-full of gear. The 48 Volts was switched by a box at the corner where the two rooms met, brought into the main building via a 5" conduit, where it was switched to two separate computer rooms. Even the links between outlying offices were redundant T-1 and T-3 lines. There was a third "data center" that was dark, to be used for spares "just in case". It was designed to provide non-stop operations, and it did a pretty good job of it. Even the Halon had built-in redundancy.
Until a semi-truck carrying some of my Memorex kit backing into the receiving dock went off course & cut the 5" conduit. The security cameras caught the sparks quite nicely :-)
Two weeks after installing the tape robots, I had a proposal for a more geographically diverse version of the same thing on the VP's desk. I didn't land the contract, alas.
I think you'll find NonStop also needs maintenance from time to time, particularly now they are little more than commodity Intel Itanic servers strapped to blade chassis. Good luck finding anyone who's actually lived down that particular technical cul-de-sac long enough to give you any support.
Pretty sure that RBS' contract assumes the system is maintained in line with manufacturers' recommendations, rather than 3rd party support outfit's budgetary targets.
I remember we broke things on a regular basis in the past . But as there were none of these locust like groups of Customer support managers and drones it didn't really matter.
The users just respectfully waited till the tech people fixed the problem and said thank you.
This is how civilized people behave.
We used to have regular 'reorganisations' in the IT section of the big company I worked for. I always insisted that any new job titles for me and mine did NOT contain the word 'Support'.
'Support' is like a red flag for the bean-counters and un-educated senior execs who perceive 'support' as an optional extra. They all bathed in the glory of launching new systems but couldn't see the cost of keeping them going. 'Maintenance' is another no-no job title for the same reason. I suspect too that Database Administrators suffer unduly from the 'Administrator' tag.
Just imagine the possible benefit from being called "Application Trouble Shooter" or "Database Controller". It would take some brass ones to cut that sort of job without sensible thought.
The headmaster - some people do not learn, they must be taught.
I got absolutely screwed over by EDS/HP on the takeover (long, boring story, but they've come back to me thrice to return and I've told them to do something anatomically impossible).
And I see it happening now on my current contract, where we are transitioning knowledge to the new, offshore support team. The Unix "specialist" had never even heard of "make", the turnover rate has been nearly 80% in three months, people with Indian accents having the names "David" and "Josh", etc. (To combat the latter, I claim my name is "Gupta" while speaking in a heavy, heavy redneck accent. When they complain that they can't understand me, I feel all warm and tingly inside.)
Anon for obvious reasons.
Here i am in work day after the works do with a horrible hangover
and this brought a big smile to my face, EDS large IT company run
pretty much entirely by laywers because they got rid most of the IT
staff which means any service you get from them is so sub-standard
because the only people left are over worked, under valued and hate
their jobs. Hope RBS takes them to the cleaner's, maybe they'll get
enough out of it to pay the Government back some of the money they
owe. :)
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
