RBS collapse details revealed: Arrow points to defective part The tech problems at the RBS banking group that left millions of people unable to access money for four days last week were caused by a failure in a piece of batch scheduling software, sources have told The Register. And at least some of the support staff for that software have been outsourced to India - as recently as …
I just saw this on another news site:
"The boss of Royal Bank of Scotland and NatWest has told Sky News that senior management will face "proper accountability" over the computer chaos when deciding on bonuses."
When popped into Google Translate (Bullshit -> English):
"We're only going to award ourselves a 1 million pound bonus instead of the 2 million we originally planned. The poor shmucks we've been running ragged with overtime this past week will receive fuck all and we'd really like our ex-IT staff to stop laughing at us now please."
[quote]"I have no evidence of that. The area, if you like, the UK backbone, has received substantial investment."[/quote]
what UK backbone ? It was all outsourced to Chennai. Unless the investment is the money they used to pay the IT staff off (or maybe the money they pay monetise to manage their mobile applications).
Here is a CEO that doesn't have a fucking clue or else his reports are lying to him. If I were him (thankfully I am not) I would be looking for a new CTO because the ones he has aren't going to finger themselves in any blame.
Utterly clueless. And very likely to be found out when reporters start digging.
The calculation we used was one good UK IT guy would need to be replaced by 2 in India plus maybe 20% input from a good UK IT guy to ensure quality. Common problem with Indian culture seems to be a kind of "can't lose face" issue, very reluctant to seek help when necessary.
Future problem for UK is how do we grow our own "good IT guys" without having them work lower down the team to build experience, skill, knowledge, insight.
Indeed. I wonder how much money outsourcing has actually saved them? If it weren't for the taxpayer backstop it'd be f*cking hilarious.
It may serve one purpose however - to highlight the risk to the financial marketplace of outsourcing your IT staff and perhaps any planning to will hold off in fear of being seen as more risky by the wholesale funding markets.
This article reads like unwarranted finger-pointing. 'Outsourced' and 'India' are not synonyms for 'incompetent', and if RBS hired someone with "4-7 years experience" of the relevant software then hopefully that person will know what s/he's doing! I just hope that insulting salary goes further over there than it does here.
No, employing staff in India isn't a synonym for incompetent, but getting rid of staff who have worked on the same systems for 20-30 years and replacing them with someone who has 4-7 years experience with the software you need doesn't compare.
Also remember: When you offer voluntary redundancy, the people who go first are the ones who are most confident of getting a new job: ie: Your most highly skilled staff.
"No, employing staff in India isn't a synonym for incompetent, but getting rid of"
Yes, that's of course a fair observation, but not one particularly emphasised by the article writer, which I believe is our fellow commenter's point above.
The article repeatedly mentions the outsourcing to India, but fails to comment on the relevance of it, if any, as regards the mishap.
"...Also remember: When you offer voluntary redundancy, the people who go first are the ones who are most confident of getting a new job: ie: Your most highly skilled staff...."
Spot on. Voluntary redundancies in this sort of case are generally a really good filter for keeping hold of the dregs and losing the cream. Doesn't stop some management doing it again and again...
As they sowed...
Lets be blunt: the best brains aren't in India. The reason? because the really excellent Indian IT people have been hired and work in the US and are being paid lots (and lots) of money. The next best are in the UK, the next best are in the Middle East or Singapore. Add onto this that IT people are paid huge salaries by Indian standards and there is an massive IT boom going on there and you get a situation where there are lots of not-very-good people in the market.
Why work in India for 10 laks (about a million rupees, or about 9K) when you can work in the US for 100K USD or in the UK for a comparable amount?
I spent weeks interviewing devs in India. I started off with 1250 CVs, did about 400 phone interviews, and 200 face-to-face interviews. I hired 15 people. And some of them were simply "warm bodies".
>'Outsourced' and 'India' are not synonyms for 'incompetent'
No they aren't but the reason for outsourcing is to cut costs so you have to add in "cheap" and when you do so you get incompetent. It's not the bods fault he's incompetent, he's probably been employed purely because he can speak reasonable Enlgish. Problems with outsourcing lay squarely at the feet of the management who instigated it and then to a lesser degree with the outsourcing supplier who recruits based on cost not ability.
Excuse me for being thick, but 11K is not cheap. In fact, if it costs 11k there, there is _NO_ real saving.
Let's factor in all factors in the equation.
1. The overlap between UK and India business hours is ~ 4h. This means that you need both a development head and a support head for every UK person you replace. In general based on experience from previous projects the "coefficient of inefficiency" here is roughly ~ 2x. Still better than India-US which tends to go into 3x because of the zero overlap between timezones.
2. You have to pay for the infrastructure there, infrastructure here and telecoms infrastructure in-between. Ballpark for that moves the number to 3x+. It is worse than India-US as you pretty much need to buy capacity on Flag/Reliance's network which is booked to the hilt and costs an arm, a leg and a prosthetic.
3. You still have to maintain a "head" in the UK either by rotating people from India or keeping PM, Arch, requirement gathering staff locally. This increases the "coefficient of inefficiency" to ~4x +
4. You now have to employ extra compliance resources to keep DP and various financial regs happy. That 4x just became 5x at the very least.
5... 6... 7....
So all in all we are talking what? The real price is at least 4-6x times the posted salary of that guy. So in reality, replacing one person in UK on a 60K salary with the one imaginatory person in India on 11K salary (in reality you are looking at 2+ and some resource on-shore to feed them) actually ends up costing 60K so no savings. At least. More like loss.
At that price you might as well go to Eastern Europe and save on compliance costs and TZ ineffiency or go even further east where labor is still cheaper and you have cheaper comms by going across USA+Pacific (at the expense of some extra compliance costs).
If you are hiring one person, that true, however, if your hiring, say, 800, Economies of scale kick in.
Also,
1), they are usually prepared to work UK hours for the money they are getting.
2), Lots of companies seem to manage fine.
3) Again, economies of scale. 1 head in the UK for every, say, 10 in India?
4) See (3)
If the numbers didnt work, people wouldnt be doing it.
"If the numbers didnt work, people wouldnt be doing it." is the worst argument I have read so far - great work, mate! It looks great on paper, but it does not work, we see it all the time. Remember: in theory, theory and practice are the same, in practice, they ain't.
Theory is, we can replace our 10y+ experienced staff by some Indian graduates, who cannot mumble proper English if you trained 'em to (no offence, guyz, the same goes for Scott's etc). You put knowledge transfer in place and hope that knowledge/1000 = knowledge + experience. Then, when you have a whole team being replaced, no amount of "knowledge transfer" is gonna work, especially when you have the guyz who get the boot delivering the knowledge transfer. Hello??, think these guyz are stupid, or what? Who really, honestly thinks a bunch of experts who have been asked to leave with a special bonus will really help those that will replace them?
Then you have the "management cretins" practice. In some companies, cretins are promoted to management positions to get them out of the way and stop them causing harm - we see this a lot in France, for example, but not only. This has a very bad effect on outsourcing.
Outsourcing will always fail, the more complex the environment, the more likely the failure AND the more complex the environment, the more costly the failure.
Note, I do not work in the UK, am not and have not directly been confronted with outsourcing ... I have been asked to clean up the mess after a failed attempt, that is all ....
Actually while I was at RBS it was widely known that each seat in India (Gurgoan, not Chennai) was costing RBS under £15,000 per year, so considerably less than the UK roles they replaced.
Oh and just because some one over here is on, say, £50k per year, does not mean they cost the employer £50k. by the time you add on their side of tax, NI, insurance, liability, medical, pension (which incidentally RBS was still ratehr good for even after ending the non-contrib final salalrey scheme), training, cost of holidays and sickness, you can add 50%+ on to the sallery. Oh and another 50% ro their first several years if they are a new starter (inductions costs, payments to agencies, cost of training up, ...)
AC because while not bitching about RBS and the fact that didn't just off-shore to India, they off-shored to cheap by local standards Indians, I am ex RBS staff.
>'Outsourced' and 'India' are not synonyms for 'incompetent'
This is true, but they have got rid of the 20-30 year 'lifers' who have invested blood sweat and tears into an organisations IT; the ones that take ownership of problems and add value in that commitment to their work and the invaluable insight they provide into complex system operations.
You cannot and never will be able to put a price on that.
If I wasn't an RBS customer with no access to my account, I would be laughing at them.
Indian staff that are based in India tend to be very junior and have a high turnover rate.
Understanding a particular software package is one thing, but knowing how it is used in a particular company environment is a very different thing. The result is that you can use them for day-to-day running of a system, providing you have an up-to-date set of docs describing in detail the process and what to do, but getting them to make all but the most basic changes is asking for trouble.
Indian staff working in the UK tend to have much better communication skills and more experience. On the other hand they are much more expensive so the bean counters (who know the price of everything and the value of nothing) don't like to employ them.
While I am sure there are plenty of talented people in India, the outsourcing companies are often terrible, I have seen companies ripped off with work so shoddy they have had to hire in local experts to fix the mess left.
You have to look at local costs before you compare salaries, the UK is very expensive, so comparing like for like is wrong.
I just know I avoid outsourcing to India now, the few times I have tried it, the results were terrible, long term its often better to hire someone locally, even if the cost is higher!
You've never tried recruiting anyone from India then?
At least 90% of the CVs that land on my desk - from organisations like Capgemini and InfoSys - are utter lies.
You read a CV that appears to be a perfect candidate, with 7 years experience working on .NET projects, but upon interviewing it very quickly becomes apparent that they haven't got a clue.
You never had the "opportunity" of working with them.
Look at landrover/jaguar, now indian owned and run and doing very nicely despite the economy. Nothing wrong with offshoring, but having seen the way it was done at RBS I can only say that it was ONLY cost that was factored in.
QUALITY went out the window.
Actually, I did... I worked there for seven or so years, I left when I was offered the opportunity of a big wedge of redundancy payment. I worked with Indian staff for the last couple of years, some of whom were pretty good, but most didn't stay long enough to become pretty good or learn the systems.
@AC 12:31 GMT
"I worked with Indian staff for the last couple of years, some of whom were pretty good, but most didn't stay long enough to become pretty good or learn the systems."
That is exactly the problem I found in other developing countries. The bright ones you train up to take over from you soon get better offers elsewhere.
Entirely correct. And who proved to be have such an entirely unsuitable experience type and length that they completely closed down the freshly purchased Indian company in <= 2 years. Of course that still leaves them with them problem of having closed the first office that had the experienced Euro product developers in it.
But the time zone was the real killer, for W. Coast USA -> India,
or UK-> India
or even better W. Coast USA + UK + India developer team meetings on videoconf.
Like, forget it.
Who wants to get up middle of the night every week maybe for 2 days every week to talk to their team?
Clue- Silicon Valley is approx 12.5 hrs behind India.
When a company makes a comment that specifically refers to "UK-based piece of software", I start smelling a rat. Why would the company's PR-drones be so specific, unless they were paranoid about media comments regarding the risks of off-shoring?
Also, it's a suitably vague PR-drone written comment - the software may well be based in the UK, but it could be written by an Indian, in Somalia, using a Brazilian keyboard and uploaded using a proxy in Thailand, after being QA'd by a kid in Poland. Where it's based is somewhat meaningless in this context.
Also, what the hell does "UK-based piece of software" mean? This is software we're talking about, it isn't "based" anywhere. If he meant "it's running on a physical server in the UK", even that means nothing, because it still could have been coded anywhere in the world, and it presumably still could be remotely executed by RBS staff at a remote location.
So it's nothing but PR-bullshitting meant for the consumption of people who don't have a clue about software.
I saw this posted by http://twitter.com/#!/YamalDodgyData over at The Telegraph, it certainly sounds plausible!
"RBS PR has specifically referred to a "UK-based piece of software" as the fault.
Why would the PR-drones at RBS be so specific, unless they were paranoid about media comments regarding the risks of cheap third world off-shoring ?
Perhaps a few questions you should ask RBS are:
Was it an Infosys employee on a Tier 2 visa at the mainframe site in Edinburgh who bungled the customer accounts batch job, or
Was it an Infosys employee working remotely from Chennai that bungled the batch job in Edinburgh ?
Does Mr Hester regret making the last British mainframe specialist at RBS redundant three weeks ago ?
and does he think the savings from sacking 1,800 British IT professionals and third world off shoring will pay off ?"
It's RBS Gurgoan, so a mix of cheap permies, contractors and some 3rd parties. RBS gave Infosys the boot ~3 years ago for being sh*t.
Also Hester just told the head of IT to reduce costs, it is they who decided to go off-shoring despite two previous failed attempts.
I knew many of the mainframe batch guys and they were a good bunch of long termers who were all showing a lack of morale when I left, but there again senior IT management had removed much of the desire to do a decent job and company loyalty with their announcements and actions.
Oh and as to sources areas to lay blame on for this fiasco. What's the bets RBS Incident management did their usual block on any repair/back out actions for amny hours while they searched their own navels and those of senior management (oh the joys of working support there).
Think there must be more to the story than this.
Whilst I've been personally responsible for scheduling issues that clobbered the Month End of a bank - I'm struggling to understand how that on its own could have caused a weeks worth of problems.
Think there is more to come on this.
Anon obviously.
It is easy. If on the first night of your batch run you miss a few input file and it isn't picked up until say the Tuesday afternoon (and let us say for example your support staff who specialise in CA7 had left for the day), you would be into Wednesday before you could be anywhere near identifying what the problem is and what caused it and what was missed the previous night. Then you have to fix the problem and work out how you are going to get the missing transaction into your batch file (they would then most likely reject because the batch header didn't have the correct day on it .. and so on..)
Of course all the time its getting escalated and people want answers, so that delays the fix because they dont want the fix screwing up something else.
One possible reason is the lack of knowledge of the RBS configuration as opposed to the product.
When the first batch failure occured, the "old" staff who knew RBS would probably have found the error quickly and could back-out the issue to the known good position.
The "new" staff however, who probably know the product, but not the RBS configuration, didn't find the issue quickly enough, further batches had completed over the top, and it was no longer possible to return to the known good position. You're then into a fix-forward scenario to apply the missing batches to an updated position which is then out of transactional order.
Think of it like an old fashioned engine - any old mechanic can probably fix it eventually, but old Bob who's looked after it for 30 years knows just where to hit it with a hammer when it stutters.
The lesson for the accounts therefore is to consider a persons salary not just as a book cost this year, but also on the basis of the investment you've already made in that resource. You can get a YTS mechanic on minimum wage, but would you really trust them to service your Aston Martin.
"Think there must be more to the story than this."
I think so as well, as sugarbear points out it should be a simple enough task to add the missing transactions to the next days run, I'm quite positive that there must also be some sort maintainance application what can be used to apply ad-hoc transaction to accounts and that the experienced maintainence programmers would know what to do to recalculate interest, refund fees etc.
wait a minute, they got rid of the experienced people didn't they
I have not used CA-7 or anything like it, god I would hate to be left alone with a mainframe.
But after a bot of googling, I assume you don't want people like this either
http://ibmmainframeforum.com/viewforum.php?f=38
"cann any one please explain the commands like PF(page forward),PB (page backward) and SR(save Replace)
Also TSO CA7cLIST"
>So ask your local MP why a UK Government owned Bank is using non-UK staff?
My MP asked a question in Parliament about fines over dog shit once, but that's her sole contribution to governance so far. May as well ask the cat.
I've seen loads of horrible tales of woe...but has anyone else noticed that international payments which normally sits in the system for a couple of days - notably Google Adsense - have arrived on the day of payment this month? I've never understood why this doesn't happen all the time - it does the other way.
Ha Ha The country continues to reap the results of citizen goodwin's ego purchase of abn-amro.
Yes CA-7 sits on the cluster of mainframes in Gogarburn, the idiots fucking it up are in India.
No, I take that back, the idiots who fucked it up were Gogarburn based as well, fred 'the pleb' goodwin, mckillop et al. 8K a year to manage CA7 Pay peanuts etc. No wonder hester thinks he earned his £1M bonus.
There are no mainframes at Gogarburn.
While Goodwin's ego over ABN-AMRO may have been the major cause of the financial crash, it was Hester et al (appointed by the LABOUR government) who aggresively outsourced the IT services to India and Accenture.
If Goodwin had backed off from ABN-AMRO, Barclays would have one the deal, which was a share swap. Come the crash, Alistair Darling would have approached RBS to take over Barclays, much as they did Lloyds for HBOS.
AC @ 12:46 - not necessarily. I worked for both during that time.
RBS was outsourcing to India way, way before the ABN farce. I know as I ran teams there (and here in the UK, Chicago, Singapore, etc). Some of those in India were good, some poor, some very bad. Sometimes the cultural issues made it worse.
As for the RBS vs. Barclays issue. When RBS "purchased" ABN they really didn't look very closely. Having seen both sides, I think I can say that Barclays would've either backed off or made a better job of the purchase.
@AC12:31 - Sorry, nothing to do with Goodwin's purchase of ABN, RBS used z and CA-7 for decades before ABN was purchased. Also, Gogaburn is the de-facto group head office (technically RBS St Andrew's Square is head office), they don't run the datacentres from a group head office.
Now take your name calling of an easy media hate figure and work out something better to do with your day.
Fred was "asset-sweating" local staff long before this kicked off and long before Gogarburn and the India-outsoucing mania. So, it is his fault, actually, for permitting this kind of behaviour to start under his watch.
The opposition across the road were much less shitty. You know, the ones without the word "Royal".
Cheers Fred, was that a fried potato of some sort on your shoulder BTW?
Here's a clue, buying things then breaking them by trashing inside processes to make this years bottom line look good isn't actually "business".
You utter twat. So having a pop at the banking establishment makes us Daily Fail readers now? Do you even know what a right-wing rag is? Have a guess who Mail readers are more likely to support; fat work-shy tax dodgers like ex-Sir Fred & Mr Pester, or hard working IT grunts like us?
You muppet.
If proven (can be, will be?), the cost of this minor (cough) glitch might offset all those fabulous offshoring / outsourcing savings they have been creaming their pants about over the last... what, 10 years?
now, how to recover this cost, so that the punters don't notice they foot the bill again....mmmm... how about bringing back the Fred?
Or maybe they could hire a "manager". You know, there are plenty of nice, honest, qualified, young folks in UK and EU. Some even have experience. Oh and they might be qualified in a thing called "banking", or even better, also in "business". Did I mention that you can get qualifications in "banking" these day at all? Apparently they don't do courses in "funny handshakes", though.
Go for Nationwide:
1) It can't play casino games with your money
2) It doesn't worry about shareholders
3) It hasn't had a penny of government cash
Is it just me who thinks if we had lost fewer building societies, we might not be quite as deep in this mess as we are ?
But you've still got the same sort of city fat cats on the board of Nationwide, with the CEO helping himself to over £2m last year. For an organisation that isn't particularly complex, supposedly works for its members, and allegedly doesn't engage in risky trading activities, I can't see much justification for that (other than of the "because I can" sort).
@12:48 How is this racist ? (which I infer from your "Daily Mail" editor line).
At no point has anyone said the chaps now looking after the NW systems are incompetent because they are Indian, which would be discriminating due to race. The point being made is that NW have outsourced to inexperienced staff to save costs, and that inexperience has possibly caused the problem and contributed to the length of time taken to fix. The same thing would happen if they outsourced to less experienced staff anywhere in the world. By screaming "Racist" over this shows what little grasp you have of the word, devalues any genuine attempts to eradicate racism and is simply an annoying and uneccessary input to this story .
Which can end up costing more than the saving. I managed 3 teams over 2 years, and by the end of it could pretty much know how to word a spec so they would deliver something that vaguely resembled it. However, this meant you had to spec so tightly you may as well have written the code yourself.
Also, they have a worrying tendency to say "yes" without understanding the issue.
"Also, they have a worrying tendency to say "yes" without understanding the issue."
It's a cultural thing, as anyone familiar with Norwich Union's cross-culture training CBTs now knows.
In my experience, Indian staff are not comfortable disagreeing with superiors, so say "yes" when they mean "not a snowball's chance in hell", out of politeness. I got caught out a few times (prior to my NU cultural training, of course!) before I realised the issue. And knowing is half the battle!
Also beware "Insh'Allah" in the Middle East. In the UK you can't get away with saying "with a bit of luck" after giving formal deadlines in senior management meetings, but in the ME, you can add a cast iron excuse, and never be taken to task for it!
"Perhaps there is a similar English phrase to inshAllah with a meaning closer to "providing all goes well" than "with luck" "
You'd still be taken to task for it not all going well, though. Whereas in the ME, several times I walked out of crash meetings scot-free, having said "well, I did say that it would only work God-willing" and everyone pretty much nodding and saying "Fair enough!".
Well the ME is not India, but I can attest through a number of adventure-laden taxi rides that Indians have a strong aversion to saying "No" (seen as a negative) to anyone they are providing a service for - regardless of truth or consequence!
That may not be incompetance at work.
One of my colleagues spent quite some time working closely with the lads from a certain large Indian outfit we use. One of the things they let slip was that they were incentivised on profit over quote. There's no money to be had in the fixed-price quote, so any gravy has to be found in the T&M Change Requests, which can be charged at "got you by the balls now" rates.
Thus, when reading a spec, any ambiguity will be read the wrong way, if the bloke reading it's got the slightest clue what the intention was, as his livelyhood depends on it. Anyone not making money tends to find themselves out the door in short order. Thus the whole system's rigged so that the idea of a quick phone call or email to resolve questions is avoided like the plague.
What you may see as incompetance, could just as easily be a deliberate and quite cunning attempt to rip you off.
When I started managing an offshore team, they were pretty much working on retainer. Mysteriously code never worked first time, things weren't tested, etc etc. It should have been apparent to anyone that we were paying for their incomptence.
First think I insisted on, was that they set up a test server (we were dealing in a web app) their end, that I could test *before* they checked it in. Much squealing over that one (in the end I just said if you can't set a simple server up, how can you be expected to be any good at coding).
Then we started using a workflow system for all communications, rather than ad hoc emails. This let us label a fix/change and track it's progress.
Thirdly, we switched to a T&M model, where *we* suggested the initial timeframe. They were free to discuss it with us - maybe we'd revise upwards if they could show why they needed more time. Otherwise, they got paid for the quoted time - regardless of how long they actually spent. I tweaked the workflow to have 2 timescales - worked time, and due time.
Once we got to that state, things were workable. However, despite the firm showering me with the cv's of their team (was there *anything* they hadn't got an MSCE in ?) I would still have to value one good UK contractor=4 offshore devs. And the firm still needed me to liaise. I dread to think how things would have gone if a BA, or worse still, customer, had to deal with them.
Having dealt with an MS employee based in India, whose English was impeccable, I think it's pretty obvious how things are. IBM had first pick. MS had second. Other big name firms third. And *then* your team got formed. So you're stuck with the developers who weren't good enough for the first 3 cuts.
The other problem people have correctly identified is churn. As soon as a dev gets some experience in your team, they'll up and off. My current employer has a 25 man team in India, who have been with the firm for 5 years now, with no one leaving. However, they are paid a *lot* more than they were 5 years ago. Couple things like that with falling UK wages, and outsourcing really doesn't have the same shine it did 10 years ago. It's a shame no one suggested that could happen 10 years ago - oh hang on, we did. But who listens to techies ?
Is sometimes done, not for reasons of cheapness, but to restructure locally and then ckaw it all back when things are working the way you want (and the incompetent/overpriced/impossible to fire staff have been shown the door)
What gets to be "interesting" in chinese proverb terms, is when the outsourcers then outsource the job to a 3rd party. The more steps removed the people doing the work are from the people wanting the work done, the greater the potential for all hell to break loose (and remember, those people doing the work answer to the middleman, not the company hiring the middleman)
An associate of mine used to work for RBS until he was offered voluntary redundancy. He graciously accepted their offer of 6 months paid gardening leave, an enhanced final salary pension which would start immediately and a big fat lump sum.
Within months they realised they actually needed his skill set so he is now working for them, full time, on a consultancy basis for even more money than he was on before.
With such compentant management I'm really surprised they got in this mess...
My experience of outsourcing things to India as both someone who has worked for banking/oil company clients and someone who's a consumer who has to deal with offshore support (eg 3, Lovefilm, Barclaycard) is that it tends to be pretty dire.
Having worked with people from India in the UK, I can say that those people have proved themselves entirely capable, so it would appear that the offshoring process itself which is at fault - usually a result of trying to do stuff on the cheap and hiring brainless morons for the lowest amount of money possible at the other end. I've also noticed a culture problem where it is sometimes seen as okay to tell you things which simply aren't true.
Outsourcing on the cheap to some poor country seems to be the quickest way to mess things up in my experience - though I'm sure outsourcing companies would disagree. As long as there are enough stupid people to put up with bad customer service and MDs who think they can save a packet on their IT budgets, mistakenly thinking a developer can be easily swapped out for another one, this practice will long continue.
I would aggree but it is worse than you think. I worked along side a few Indian engineers and got an inside look at the out sourcing companies. The biggest problem was staff churn, the companies pay them peanuts treat them badly and like to turn over all their staff within 2 years to keep the wage bills down.
The only way to make out sourcing work is out source some incidental part of you business build in punishing SLA so when the make a mess up you never have to pay them. After a couple of years bring it back in house. Result 2 years free IT support.
I have to agree, having worked with outsourcing whilst working in a foreign-owned bank that used to be an English building society. (work it out!)
One of the biggest problems I had was knowing whether or not they understood what needed done or whether or not they thought they could do it, because the answer was "yes" to just about anything.
More than once during conversations I'd ask them "can you do such-and-such" and get the answer "yes". Unconvinced, I'd start to expand it to things that I knew damn well they couldn't do, e.g. "These alerts we need to send out through the browser - if the customer isn't using their browser at the time of the alert, can you start up their browser, log them in, and show the alert? It's all got to be through the browser, we can't install anything on their machine." and I'd get a "yes".
So I'd basically say "No you can't. Now, we'll go back and start at the beginning again and be realistic this time". They were just as happy to put it down in writing too.
I'd really have to work hard at convincing them it was ok to tell me they couldn't or didn't know how to do something.
I don't think it's being racist - and I'm not sure I can express sufficiently that it's absolutely not intended as such - but there is definitely a cultural issue where it would appear they wish to please to the extent that they are very reluctant to say "No" to just about anything.
I was chatting to a south african a few years back who'd spent three years working in India. Keen to make friends, he kept on inviting people over to dinner. They graciously said yes but never turned up. Happened four or five times until someone told him that they had no intention of going but didn't want to upset him by saying no.
Helped me a great deal at the time, because I started phrasing questions open ended, rather as yes/no responses when dealing with Indian clients.
It's a cultural thing and it happens in a lot of Asian cultures where the concept of 'losing face' operates. They lie to you so they are not seen as losing face which is a much worse thing, for them, than being caught in a lie. I've seen it when working with Japanese, Chinese and Indian colleagues and face to face you can learn to read the signs of when they are saying 'yes' out of politeness/fear. On the phone or via email/skype that is harder to do.
Understanding that it's a cultural thing makes it easier to deal with, unless someone who doesn't understand the difference through being too PC intervenes.
HSBC has got all this stuff covered.
They call it Unscheduled Maintenance.
Then, when their long suffering customers actually try calling their branch, they get shuffled off to Sweatshop India where totally inept people, who speak a form of English that is totally incomprehensible to the average English person, try to confuse you.
By the time you have wasted your time trying to communicate with Customer Support (India) for ten or fifteen minutes you will find the Unscheduled Maintenance is over.
Can't beat exporting British jobs.
Thats a bit unfair
Every time I've called the HSBC customer support , I've been put over to their India based support centers and had the problems resolved in no time.
Because they dont understand my accent when they fail to understand my password for the 5th time and put a security lock on the account .... which is usually what I'm after since I wont use their on-line banking system :-)
Boris the luddite
Cron, really?
Have you ever had to use a pukka enterprise grade scheduler such as CA7, Autosys, UC4 or Control-M in a production landscape? I could be taking a stab in the dark here but I'm guessing not.
Cron bears as much resemblance to any of these as does say Windows Task scheduler.
They are very different in their modes of operation, resilience (if configured properly) and capabilities.
For one thing, they support the concept of dependencies (eg only run this job once the previous one has completed successfully) and alerting when things don't go to plan.
Hi AC,
Have you got a link for any of these fancy new features in Windows Scheduled Tasks? Last time I looked the Windows scheduling facilities were no more advanced than those back in Windows 2000 as detailed here:
http://technet.microsoft.com/en-us/library/bb726974.aspx
Sure the snap in under Win 7 and the latest server versions for managing the tasks looks snazzier, but I bet if you export the tasks via the command line to .CSV they'll look the same, eg:
schtasks /query /FO CSV /V > schtasklist.csv
Where the software is and where the staff are is irrelevant.
Experience and competence are essential.
The head honchos at all companies are paid inflated salaries because you have to have "World Class People" to run the business.
All board members should only ever be bonussed on the customer experience and NOTHING ELSE!
They think they are running the company, but it is everyone below them that are doing it.
When I was working I always said, I'd settle for annual salary increase on the same basis as the chairman in either cash or percentage terms.
This is a massive fail. Hester and the person responsible for the Banks IT should fall on their swords.
Customer Experience = No bonus for any of the board this year!
[The person responsible for the Banks IT or may not be Ron Teerlink. He is Chief Administrative Officer of Royal Bank of Scotland Group Plc. He joined the RBS Group in April 2008 as Chief Executive of Business Services, becoming the Group Chief Administrative Officer in February 2009. At the same time he was re-appointed to the Managing Board of ABN AMRO to oversee the integration programme. Ron started his career with ABN Bank in 1986 as an IT/Systems analyst and held various functional positions before becoming Chief Operating Officer of the Wholesale Clients Business in 2002. He was appointed Chief Executive Officer of Group Shared Services in 2004 and joined ABN AMRO’s Managing Board in January 2006, where he was responsible for Services and Market Infrastructure. Ron holds a Masters degree in Economics from Amsterdam’s Vrije Universiteit.]
Anyway, Teerlink is the only board member that has any IT experience listed.
I used to contract for a big OS and Office software company - they sent loads of work to India. After 2 years they brought a lot of it back to Europe - offshoring was no cheaper (in fact it worked out quite similar cost-wise once you factor in everything) and it's not that the quality was bad, it was more that it meant trying to discuss complex issues over VOIP when having 6 people in a room together with a printed schema and some sharpies was just more efficient.
Installation instruction scripts were followed to the letter, everything needed to be spelled out as you never knew who you would get doing an install or update - so you had to try to pre-empt all the failure modes and explain in bullet points how to roll back etc for each case. Sure, this is good practice by and large, but it meant UK engineers were spending hours writing these instructions each week when the patch was 2 minutes work and could only really be rolled back by those same engineers anyway as it required deep business knowledge.
They were also hit by the vendor creating and billing phantom employees, which really did not help, but is very hard to monitor.
...and having issues with understanding the person on the other end of the phone.
Turned out the person I'd been speaking to in Bangalore (whom I understood perfectly) had put me through to a call centre in Merseyside for "local service" - I couldn't understand a word...
So the bank 80% owned by the UK goverment to save cash offshored work to India. If they had kept the work in the UK the money spent atleast would have stayed in the UK to help the UK economy get going rather than India's. In the long run that would have helped the UK more ( and cost the UK less). If all the banks that have taken government cash offshore then all we have done is help everyone esle out at the expense of the UK economy.
"The RBS spokesman stated that "the software error occurred on a UK-based piece of software" but declined to give details on where the staff overseeing the software were based."
"UK-based piece of software"
FFS!
I obviously don't work in marketing because only a Spin Doctor could come up with nonsense like that.
Because the long-tem costs don't matter in the slightest to them. Someone (let's call them an advisor - the government use loads of these) has told them that they will save a load of dosh by not having to pay loads of people any more. The long term costs will hopefully be someone else's problem to deal with.
And it's not just limited to government. It happens everywhere, and not just in IT. Quarter-end profit and share prices matter more than anything, even if it means running the company/country into the ground. As long as it happens after those reaping the benefits have retired/are dead then it doesn't matter.
Running a company should come with responsibility. If you fuck it up you should lose everything.
Every person who comments on dealing with offshore co-workers seems to have the same experience? Either everyone commenting here works for the same company as me, or the issues are consistent across all offshored companies. The other bit we were fobbed off with is that the offshore staff will work shifts and do all the on-call work and work that needs to be done outside UK business hours. Well all the bits they want to do, which is basically nothing beyond a few very simple menial tasks, and as soon as they have to make a decision or find out why the documented procedure didn't work then they're screwed and leave it for us to sort out the mess they've left behind. Oh wait... hang on... that's where we started with this mess wasn't it?
I think they are. Terrible, terrible time at Logica working with outsourced pieces of the business (won't say which sector) except for one awesome guy who Bangalore sent to us for hands-on training. Sharp as a knife, brilliant ASP.Net skills and a thoroughly decent chap. All the rest: rubbish.
This post has been deleted by its author
The problem is most of these big companies don't outsource because they need talent but they do it to cut costs and in some cases they don't even pay well in the country they're outsourcing too so save even more money so it's no surprise things go wrong with outsourcing for some companies.
Bad management always leads to a bad job no matter where the developers are from.
(which may or may not be offshore) is that it may be a good idea if some or all of the following criteria apply:
(a) your internal operation isn't big enough to gain economies of scale;
(b) the activity being outsourced is not a core part of your business;
(c) your requirements are quite similar to many other businesses;
(d) you can produce a clear and unambiguous specification for the services you need that is unlikely to change for the duration of the contract.
So catering and night-watchmen - OK to outsource. IT may be a possibility if you're a small to mid-size manufacturer (say) using COTS systems. But if you're a high street bank whose product is (to a first approximation) your bespoke IT systems, it doesn't take a genius to realise it will all end in tears.
I wonder if we met the same bank IT manager; I've heard the same argument.
My reply was, essentially, "No, you're in the IT business, you just happen to work at a bank."
I guess we now have a powerful example of what happens when the IT folks lose sight of what business they're in...
I just heard Stephen Hester on Radio four news, being very careful to say that this wasn't an outsourcing problem. He's right, the staff in India are technically RBS employees, the problem is employing cheap labour in a foreign country who don't have the experience of the staff they are replacing.
Beware of semantic, qualified answers to questions about this situation...
If anyone is to blame its the man named above who started the whole process about five years ago.
Wouldn't listen, didn't care, you would think he was only interested in securing his own bonus (or he had shares in infosys!)
I wonder if he is feeling the heat now....
This seems to be a ISV (independent software vendor) Product (CA7) upgrade issue, nothing to do with the person's job role you 'googled' and posted in the news!!! or even his nationality. Any Tom or any Ram might run to this issue if your systems are way too 'obsolete' and refuse to pay for updated s/w.
Please have your facts clear before you post something like this, shame on you.
http://www.computerworlduk.com/news/it-business/3366112/ca-helps-rbs-resolve-tech-problem-that-led-massive-outage/
"This seems to be a ISV (independent software vendor) Product (CA7) upgrade issue, ... Any Tom or any Ram might run to this issue if your systems are way too 'obsolete' and refuse to pay for updated s/w."
This seems to be a failed s/w upgrade - which happens all the time - followed by a botched backout to a known good restore point.
Given the systematic incompetence that RBS have demonstrated, their cavalier disregard for either private or state shareholder, their snivelling unfelt apologies for pissing off millions of customers, their well dodgy corporate banking arm, isn't it about time that the government did what they did for Fred?
Sir Fred is now plain Fred. Let's follow the pattern, and whip the "Royal" out of Royal Bank of Scotland. Obviously HBOS own the name "Bank of Scotland", but RBS could perhaps investigate other possible names. A quick check on Companies House shows that "Bank of Thieving Bungling Twats" doesn't yet appear to be taken.
Anyone else find it highly suspect that the advert for the job, shows a Gmail address? Perhaps because I've been investigating Indian based scammers for years, I've be come overly skeptical, but no bank or financial company, should be dealing with ANY company that uses a Gmail address as a contact spot, especially for something as important as this.
Makes you wonder how many ex employees of other large companies are saying "that could be XXXX". I know I am.
Skilled UK people bailing at the sight of a reasonable redundancy - check
Skilled Indians leaving almost before they had finished company specific training - check
Morale diving in the UK as what was once a relatively smooth efficient machine starts making grinding noises - check
Management proclaiming how successful it all is while everyone else wonders what they are smoking - check
Cultural differences causing problems as the two ways of working clash - check
The cost savings justifications being eroded over time as the cost of the Indian work force rises - check
The cost savings justifications being destroyed as one or more easily avoidable blunders eat up cash like crazy - check
My own poison was Tesco and while I have many fond memories of the company I still shake my head at how gung-ho they were about the off-shoring process. After the accidental deletion of the entire UK user tree in AD followed by the not so accidental loss of the relevant log files and the discovery that the backup procedures hadn't been followed that THEN they might actually have learned something but no..... that was the signal for me that I'd be better off elsewhere.
CA7 has been around for many years - i was using it in the 90's. It works and performs very well, provided you know the tool and you know the environment. it is uniquely mainframe.
Not only did they give it to people who dont know how to use it nor do they know what their environment is, but they scheduled this upgrade on a tuesday. Had they done this on a saturday after midnight they would have at least had the sunday to back it out.
This is incompetence at its grandest.
It also proves... you get what you pay for.
I was at Direct Line, Croydon when RBS shifted it all off up to Gogarburn, c2003/4
Relocate to Edinburgh? No thanks, next stop probably Mumbai, could see it coming even back then.
Initial redundancy offer was the state minimum - from a company (DL) making over £1M a day in profit!
Whereas RBS staff T+Cs = far more lucrative.
Make us RBS staff first then? NO.
And they wondered why we had no enthusiasm for knowledge transfer.
No disrespect to the Edinburgh lads - they were a decent bunch and our situation wasn't of their making.
And re RBS hardware investment - Can we have STK tape robot/silos please? NO. But do have this unreliable hunk of junk instead. Whatever it was, nasty thing. Probably cheap though.
TSB
Mid 1990's, some mid-management fool asked why we have 24 shift ops staff.
Well, dimwit, it's 4 shifts of six staff.
But why four shifts?
Well, dimwit, would you want to work more than an average of 8hrs a day, covering 24x365?
24hrs = three 8 hour shifts, plus 8 hours off, which = 4 shifts.
>> Well, dimwit, would you want to work more than an average of 8hrs a day, covering 24x365?
>>
>> 24hrs = three 8 hour shifts, plus 8 hours off, which = 4 shifts.
What pattern did you work 3 on 1 off? I work 12 hour shifts, so we have 2 x 12 hour shifts plus 2 x 12 off - still works out as 4 shifts. We work for days (or nights) on and 4 off. Means longer days, but fewer trips to work (so less wasted on travel), also we avoid rush hour traffic.
BTW, I'm not with NatWest or RBS, however my wife's employer is - and she was due to have been paid her monthly salary last Friday.
What pattern did you work 3 on 1 off?
Eight week pattern, meaning, by average, only one weekend a month.
WK1
Sat + Sun 20:00 - 08:00
Mon-Fri 00:00 - 08:00
WK2
Sat+Sun OFF
Mon-Fri 16:00 - 20:00
WK3
Sat+Sun OFF
Mon + Tue OFF
Wed-Fri 08:00 - 1600
WK4
Sat+Sun 08:00 - 20:00
Mon+Tue 08:00 - 16:00
Wed-Fri OFF
WK5
Sat+Sun OFF
Mon-Fri 00:00 - 08:00
WK6
Sat+Sun OFF
Mon-Fri 16:00 - 20:00
WK7
Sat+Sun OFF
Mon + Tue OFF
Wed-Fri 08:00 - 1600
WK8
Sat+Sun OFF
Mon+Tue 08:00 - 16:00
Wed-Fri OFF
....and repeat, for very nearly a decade until Lloyds took it all away, redundancy #1
How many on you on this discussion love calling a overseas call centre love giving your details or knowing in this case your sensitive banking information is somewhere thousands of miles away.
The Lord of the Banks say we are saving money, yes you are but now this will cost you millions of pounds to fix the potential loss of customers could damage the bank for years. The common sense approach is to bring back all IT services back to the UK were they belong, bring back the jobs that were taken,
I'm with ulsterbank (natwest/RBS) and my wife's pay has not arrived at all.
My pay is supposed to arrive on the 28th but I'm not convinced it will, as none of te other transactions are working. I have four accounts with ulsterbank and none of them seem to be working as before this fiasco.
I bet the feckers charge me for late fees on my ulsterbank credit card, let them try.... I've never failed to pay it in full before!
You know, it is better to use one brand of software in scheduling across the landscape, especially if you can have one central point of control. I guess the sales guyz of the other schedulers will be knocking on RBS's door soon. I want run my jobs, total control, no maintenance costs, no downtime ... one of them must have the solution ...
Whilst it's all very fascinating hearing about peoples issues with outsourcing (& I have my own experiences...) nobody seems to have touched on the other major show of incompetence, namely - who, in the name of all that's holy, schedules an upgrade to a major finance system in the middle of the week?
I'd love to hear the reasoning behind that!
I didn't believe they could be so dumb as to do an upgrade midweek. These comments tally with what their spokeperson said on Moneybox last Saturday: basically it was the backout on Tuesday that caused the problems. These posts spell out why they needed to do the backout, following an upgrade at the weekend that caused problems on Monday.
The following is taken from a number of posts on the westhamonline dot net forum by a poster called CTM who is currently working on the Natwest/RBS fiasco and who was due to be made redundant 2 weeks ago. The most worrying comment is that 'The backout caused the "queue" of jobs for that batch to be unexpectedly deleted. '
"Ok, since I work on the mainframe at RBSG, I'll let you know some facts - I'm currently on my 5th 12 hour shift in 4 days (and am quite busy so won't be back here for a while)
It is a mainframe problem and not a virus It was not deliberate by the bank
It was a third party product that was upgrading that caused a big whole in the batch schedule
RBS wasn't actually affected directly as the schedule is on a different LPAR - however focus was on NWB & UB and normal everyday errors on RBS side were not getting dealt with
COBOL and Assembler on the Mainframe, like most big banks. But it's not the application code that is the problem as I stated below. CA7 was updated to a new level last weekend. It was causing problems with the schedules on Monday so was backed out on Tuesday. The backout caused the "queue" of jobs for that batch to be unexpectedly deleted.
Not sure why DR wasn't invoked to be honest, but it wouldn't have made much difference as we'd still have been hours behind on our batches.
All transactions made on Tuesday and since WILL be processed eventually.
As for the account thats being sold to Santander - you can just phone the number on your letter and you won't be moved, so stop fucking whining like you've been made to do it. We could have done without the work and expense that was foisted on us! Anyway, there's a pot of gold at the end of the RAINBOW. SWT
Also, in my previous posts, the "not sure" was my own, not the banks - I'm sure that the discussions were had about DR and there must have been a good reason why it wasn't invoked. In truth DR is great in certain circumstances if it works, but in this one it wouldn't have. Obviously i can't go into too much detail here, but although the outsourcing to India did not cause the intial problem, the recovery has prompted questions to be asked as to who thought dumping over 50% of the UK support staff was a good idea - indeed people made redundant less tha a month ago have now been offered short term contracts to come back and help. Unfortunately, the balance sheet and pressure from the government to become independent again hasn't helped in the outsourcing/offshoring decision and I speak as a someone who was due to have been made redundant two weeks ago."
Of everything that I have read so far, this seems to be by far the most plausible explanation. Combination of upgrade not working and then the decision to roll back causing a serious error.
From (admittedly back in the 90's) experience, you can recover if you lose the CA7 data. But only if you have a bunch of guys in the computer room who have worked on those batch schedules for years and know them off by heart. You can't simply re-create without them. However, they are generally the first in the chopping line as on a resourcing spreadsheet you simply see very expensive resources doing something that the management consultant doesn't understand.
Again from experience of working in IT in retail banks, the business / management neither understand nor respect IT. As such they are dealt a crap hand every budgetary cycle and simply have to make do. Until the business understand that they are dead in the water without IT and start giving it the money it requires then this event is going to be repeated over and over again.
A few years ago I was made redundant when my job role was offshored to Chennai, India. It was a nice investment management company with circa 500 people and visions of an IPO. I had been at this company for 14 years. The knowledge transfer guys were pleasent and nice guys with a little, but not too much knowledge on the subject (the subject being, for me Information Security). Their production of Run Books for their offshore team, however, wasnt.
This is my point - the high turnover of the engineers assigned to a customer contract means they have the poorest of documentation that they themselves have created as their only knowledge of context of product use. It's not the technology so much, as how it's used in the environment. That, is of course when things are working. Like outsourcing, it's only great when it actually works. And when it doesnt then you're given the scenario that has RBS/Natwest without transaction services - the absolute core of their multi-million pound establishment, in ruins.
I'm thinking about creating a meme on this, with that guy from Anchorman:
Outsourcing:
<insert jpg>
60% of the time, it works everytime
Why AC? Moral of the story I guess. The company in question expected a launch at 300p, launched at 220p and dive bombed to 70p... all a result of mis-management by those same people who thought outsourcing 25 people to India the great company I worked half my life for and had existed since the early 60s is no more.
Once Upon A Time, in a land Far Far Away, there was a publicly-listed networking device company that closed a sh*t-ton of offices globally including one or two newly-built and taken over from competitors. They outsourced quite a bit of development to India and the stock price promptly nosedived, the only thing that saved them from circling the drain quite so closely was canning the India office immediately and taking themselves OFF the stock market.
Thus ends the Parable of The Stupid Fat Hippy Buddhist Boomer and his New Office.
This parable may or may not be related to the Parable of The Brown Envelopes and the Parable of The Developer That Wasn't and The Stockholder Who Was. All rights reserved. The names of people and places may have been changed to protect the innocent, if there were any.
may I respectfully suggest a better way to vent your ire, is not to close your bank account, but to simply open another, and reduce the balance in your NatWest account to (say) £1.01. Make sure you are signed up for monthly statements. By post.
I did this 16 years ago when Barclays ****ed me off. Every month I get a statement. Last year they almost twigged that the account may be dormant - I got a warning letter. So I shifted £5.00 in, and then a day later £5.00 out.
Just make sure there's no overdraft availale on the account.
If your outsourcing governance is solid which for a mutlinational bank it would have to be then your outsourcers would not be in a position to make any errors
OR........
You could hawk it out to the cheapest bidder and use the money you "saved!?!?!?!" to bonus up your senior management
I wonder which of the two approaches RBS is using ?
Also I wonder what will become of the onshorebased management team for the outsourced departments. (New yachts for everyone from what I can gather).
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
