RBS: June's tech enormo-cock-up cost us £125m

RBS's software upgrade blunder in June has cost the banking group £125m so far this year, with further costs expected. The bulk of that money was spent on compensating customers, with the cost of the actual technical fix "unlikely to cost a meaningful amount", according to group finance director Bruce Van Saun. RBS also …

[Chris Miller]

And how much did you save by sacking most of your experienced IT staff? A few million was it?

Experience keeps a dear school, but fools will learn in no other. - Benjamin Franklin

[Anonymous Coward]

And (dodgy el Reg article aside) you know this was a consequence of sacking staff... how?

[Anonymous Coward]

Anon, for commercial reasons, but whilst running a training course for serious RBS IT staff had to break off at one point so's the guys in the room could dial in to a conf call to find out how many of them were being made redundant. Jobs off-shored with no real handover. That'll tell you where RBS's priorities lie.

[Shagbag]

Management will do their damnedest not to admit that

...they'll throw every possible excuse but they won't recognise they got it wrong.

"I should be clear that over the first 3.5 years of our recovery plan, we've increased tech spend quite significantly and we've increased spend on the infrastructure as well as the fancy new stuff."

As an accountant, there is no clear definition of what "spend" means. It can mean many things. For example, it could include depreciation/amortisation of assets, ie. a non-cash "spend". Accelerate your depreciation - ie. write it off faster/over a shorter period - and your "spend" has increased without actually "spending" any cash. It could also include the cash spent on redundancy. Those costs could be allocated internally to the IT Department at RBS. If they were, then they could account for some of the increased "spend".

I'm not saying this is what happened. I'm saying this could be what happened.

[Anonymous Coward]

Re: Management will do their damnedest not to admit that

'... increased tech spend... significantly...'

From what to what?

From $1 to $10? That's a 1000-fold increase (significant) of a piddly amount (insignificant).

From $100,000,000 to $101,000,000? That's a $1M increase (significant) or a 1% increase (insignificant).

[Anonymous Coward]

Re: Management will do their damnedest not to admit that

my bad, % not fold.

[mrfill]

Re: Management will do their damnedest not to admit that

$1 to $10 is a 900% increase

[Richard Wharram]

Reputational damage?

How much is that worth?

[Yet Another Anonymous coward]

Re: Reputational damage?

They went bust (sort of) and were bailed out by the taxpayer

They then managed to lose a billion quid/year while being lent money for free by the Bank of England

What reputation was there to damage?

[theblackhand]

Re: Reputational damage?

Re: They then managed to lose a billion quid/year while being lent money for free by the Bank of England

RBS loaned money for items that were worth X during the boom years and Y during the recession. Once a borrower defaulted, RBS had to reassess the value of the asset at some point, hence the loss. The loss may have been stated when receiving free loans, but the loss had occurred some years prior short of miraculous, never ending boom. (I admit some politicians may have promised this....)

i.e. the loss was a combination of the initial loan (the asset was over valued) and the recession (the borrower defaulting). I would consider a business that wasn't able to correctly assess the value of the assets when it's business depended on those assets to be valued correctly the larger issue than the recession. Particularly when the majority of it's peers didn't suffer the same fate.

[Yet Another Anonymous coward]

Re: Reputational damage?

I thought all those bad loans got magic-ed away when they were bailed out by the quantitative easing fairy?

Seriously it's getting to the point where the banks welcome people with sawn-off shotguns because it's less costly than letting their own people near the money.

[mark 63]

"refunding late payment fines "

I wonder how many of those fines were their own?

"We fined you £30 because we were unable to move £xxx from your RBS current account to your RBS mortgage"

[jonathanb]

None of the £125m would be related to their own fines. That is just them not making additional profit from fines.

[sugerbear]

[quote]We'll be simplifying the batches and tightening procedures a bit.[/quote]

I cant see that ending well.

See my previous posts. No change, no blame and more red tape ;-)

[squilookle]

"meaningful amount"

Here lies the problem. They don't understand the value of money. Any money spent on something avoidable is meaningful.

Also, I have no doubts that this amount of money that is "not meaningful" is more than my annual income by a factor of many, and would more than cover the cost of wages of many staff they could employ and/or not make redundant to make sure this kind of thing doesn't happen.

[Anonymous Coward]

Not really. The probability of this type of incident is inordinately low, and there will be several other "could happen" scenarios. To resolve (not mitigate) every conceivable risk is usually considerably more than the costs of the risk. It's only hindsight that allows you to say "you should have fixed that one". And how long do you measure this over?

There will be two separate data centres for RBS (minimum). The cost exposure if they both fail would be enormous and considerably less than the cost of building a third - doesn't mean you'd do it.

[Anonymous Coward]

No you can't have coverage for very low probability events but if you want to be prepared for handling "this kind of major fault" you can plan and conduct drills once in a while to keep thinking sharp and to spot any easy-wins in preparedness (some pre-canned queries and scripts plus tuning the database to support them, how to run some critical batch manually if you really had to, business timeline monitoring etc).

But then with that foresight and investment, would you offshore the whole lot to some very enthusiastic but utterly inexperienced grads in a far away country.

We really need a Private Fraiser icon. I'd better post this anon...

[Johan Bastiaansen]

"The probability of this type of incident is inordinately low."

Yes you're right, but only if people are sane and work to avoid this risk. If they're the mad management type, and they know "The probability of this type of incident is inordinately low" meaning "we can f*ck around like we want, this is never going to happen", than of course the risk rises astronomically. And this is upgraded to "a disaster waiting to happen".

[jonathanb]

Hiring a British person at c£50k a year who actually knows how to operate this software isn't a "meaningful" cost in the overall scheme of things. That is probably all they need to do to sort this out.

[Tom 38]

…RBS missold them insurance (interest rate swaps)

This little shit nugget has been dropped into this news without any explanation of why or how they mis-sold interest rate swaps.

One of the things that Barclay's may have done by conspiring to manipulate LIBOR is that they would have directly affected people who purchased financial instruments linked to LIBOR - in other words, they mis-sold interest rate swaps, which are explicitly linked to LIBOR.

Is this RBS slipping in that they also conspired to manipulate LIBOR, and are now putting away £25m in restitution? What about the companies that went bust because they couldn't access credit because banks were lying about the cost of LIBOR?

[jonathanb]

Re: …RBS missold them insurance (interest rate swaps)

Manipulating LIBOR is part of the story, but not the whole story.

They manipulated the rate downwards so that they could make more money from the swaps they issued, but that is not what the mis-selling scandal is about. They sold swaps that were not appropriate for the customers, for example they sold a 15 year swap to insure against a 5 year loan, so the customer had a lot of expenditure after they thought the loan was paid off.

[Anonymous Coward]

Re: …RBS missold them insurance (interest rate swaps)

But that's even better. They mis-sold insurance - no doubt fix/float IR swaps so the client pays fixed and receives LIBOR +- spread and then were likely part of the LIBOR fixing ring that would have seen said clients receive less than they should by holding down LIBOR. Fantastic. This fine will merely be a first installment. The perverse part is it's really the tax payer paying the fines and restitution to the tax payer. You couldn't make it up.

[regprentice]

£125mn... 17.5mn customers..... so just over 7 quid each for customers who, in some cases still cannot reconcile all of the transactions on their accounts 6 weeks later.

Does pose the question 'how did you manage to prepare a set of accounts?

That's £2.50 per head less than o2 voluntarily paid out to customers impacted by 24 hre outage.

As an employee of Ulster bank I'm absolutely ashamed to work for rbs. Frankly the devestation they have caused in eire should be subject to a criminal prosecution.

[Anonymous Cowerd]

"RBS's software glitch"?

As far as I know, the glitch occurred in a piece of wetware.

[Dodgy Geezer]

@sugerbear

We'll be simplifying the batches and tightening procedures a bit.

In other words, the whippings will continue until morale improves......

[Anonymous Coward]

bolshy bankers

The trouble is they have turned into a bit of a british leyland. The public pretty much own it and foot the bill for all the balls ups. I doubt they will ever post a profit but we will keep shovelling our money into the bonfire.

In fact we would probably be better off if they went out on strike for weeks on end. lol

[Ron Christian]

refunding late charges?

Something I don't understand -- if a substantial part of that giant gob of money was refunding late charges to customers, how is this a cost? Shouldn't it be considered money returned that the bank should not have received?

[Annihilator]

Re: refunding late charges?

Couple of things, it will relate to charges from other companies (RBS errors will have prevented payments to pay third party loans, council tax, rent, etc which will/can all incur charges that RBS will be obliged to pay) but also some of the charges would have normally been (morals aside) legitimately received by RBS in that period that the outages could be held responsible for, even if they would have happened anyway.

[Sim]

How Much did RBS make

Did they have their customers money earning interest on the money markets for a week or so ?

[Arctic fox]

RE: "unlikely to cost a meaningful amount" So the fix is an easy one then.......

.........if it's so cheap to fix, no? Correct my logic here if I have missed something but does that not rather imply that the glitch was rather straightforward/obvious and should have been detected before it did damage? Given that it was not, what does this say about what RBS have done to their IT support? The only conclusion I can draw is that RBS have qualified themselves to be awarded the prestigious title of "Knobheads of the Year" by public acclamation.

[AJC]

Another risk

I have seen scant reference to the possible impact of this weeks power grid failure in India on outsources IT.

Did it impact any outsources IT operation? I presume that these operations (including help desks) have adequate backup power generation (how many days fuel supply on-site?). Was there any disruption to (failure of) the communications paths?

Perhaps this particular failure didn't affect any of the outsourcing areas on this occasion?

What (now) about the risk of it happening again and bringing down other regions next occasion?

[Martin 63]

Probability

"The probability of this type of incident is inordinately low."

And happens nine times out of ten. Gotta love Terry Pratchett

[despairing citizen]

£125m, Ah, and the rest .............

£125m is an exceedingly low figure, and even a "bag of a fag packet" analisys by a BA would show that their indemnity costs are like to be double that.

Now add on the costs of customers walking (given recent survey RBS/FailWest/StillNotWorkUlster), and deduct that from the revenue stream.

The cost of managers being hauled up in front of the Irish government to answer questions (the ClownServatives, LibDumbs and LieBour parties don't see an issue with 25% of the UK ability to bank being offlined)

Brand damage is going to cost vast quantities of advertising budget and account managers time convinsing customer to rely on our banking systems.

personally I would be amazed if the damage is less than £300m, and would be unsurprised if it was north of £600m

So that half the IT team in scotland saving (approx £10m over 3 years), looks real good (sic)

If I was a majority share holder in the bank, I would start by firing the entire executive board, and the NEDs on the Audit committee, and appoint a brand new board with job one being to stablise the administrative systems of the bank, and job 2 to review the sign off of Risk on the downsize, outsource and offshore programmes, with a view to sacking on grounds of competance, or reassigning to less demanding roles.

[John Smith 19]

It appears no one looked at the *consequences* of failure.

I don't mean "if this little bit fails," I mean the big picture. If the *process* (or any other for that matter) fails.

For an hour.

For a day

For a week

Or as it's turned out about 3 weeks.

*something* that just sets a bar which says "Beyond this time we be f**ked" and allows peoples thoughts to *prioritize* the mildly annoying form the business threatening.

And it looks like I'm not alone in thinking "125m, that's *way* too low a figure to *fully* account for this f***up running for 3 weeks"

[Anonymous Coward]

Re: It appears no one looked at the *consequences* of failure.

@John Smith 19

"For an hour.

For a day

For a week

Or as it's turned out about 3 weeks."

The (non-UK) bank I once worked for had a power outage lasting several hours which among other things cost them compensation for the delay in processing large settlements. The settlement processing system was upgraded sharpish to redundant kit in two separate buildings, with a third location planned. That wasn't cheap but it did cover the risk.