Let's talk about the RBS IT cock-up

One month ago a rookie IT mistake crippled the banking network of the RBS Group. Sixteen million customers – individuals and companies - of RBS, NatWest and the Bank of Ulster were locked out of accounts, unable to withdraw cash or pay into accounts for days. The cause? An “inexperienced operative” pressing the wrong button …

[Anonymous Coward]

Something to look at:

Recall ABN AMRO? First slurped up then spit out by RBS, or whatever really happened? Dutch punters have had various troubles accessing their accounts with them too. Coincidence?

[Anonymous Coward]

Re: Something to look at:

No... ABN Amro seriously troubled Dutch bank, split into three, one of the chunks was purchased by RBS and was (is still being) integrated into their systems.

[Anonymous Coward]

Re: Something to look at:

The Dutch part of ABN-AMRO seems to be working fine, or at least, I've never had any trouble with my accounts nor heard of anyone who has had trouble with theirs

[Anonymous Coward]

"I've never [...] heard of anyone who has had trouble with theirs"

Then you haven't been paying attention lately.

Three times trouble in last two weeks (google translated).

[Anonymous Coward]

Hmm

"And although RBS denied it, The Reg uncovered that the person responsible worked offshore – in one of many positions sent abroad as part of wide-ranging outsourcing to cut costs and so-called efficiencies."

No you didn't, you uncovered that other people who work on CA-7 are based offshore. You've got no actual proof that the person who buggered this particular change was offshore, onshore, or on the moon.

[Annihilator]

Re: Hmm

Plus, the thing that was "uncovered" (and as Volvic says, you didn't uncover anything) was that *RBS* were looking to hire someone in India. This isn't outsourcing, this is offshoring. Please don't take a simplistic/erroneous Daily Mail approach to this. If it were outsourcing, it would be a different company involved.

[Anonymous Coward]

Re: Hmm

The whole approach to this story from El Reg has stunk, unfortunately. Very Daily Mailesque, from the immediate finger pointing to India to the vindictive move of printing entire extracts of some poor sod's CV - who for all they know had absolutely nothing to do with the incident whatsoever.

Definitely not the vulture's finest hour.

[frankfrankerton]

Re: Hmm

Agreed, the whole reporting of this story is very tabloid. AKA - based on nothing but google searches and a questionable "source"

[Anonymous Coward]

The incident whatsoever?

I do feel some sympathy for the chap, but why would he claim experience of working on overnight batch runs for RBS, and then delete the record from Linkedin.

http://cantankerous.co.uk/?p=747

[Anonymous Coward]

Re: The incident whatsoever?

Because a clueless blogger started posting screenshots of his LinkedIn account, with his full name, claiming that he had something to do with the incident.

Followed up by a supposedly reputable IT news website posting screenshots of his account.

[Anonymous Coward]

Too specific. This issue does not just affect banks, it's everybody who has outsourced.

Will/should heads roll?

Will heads roll? No. However, the head of whomever decided to ditch the experianced staff and thousands of man years worth of experiance simply to cut costs without dealing with the risks caused by his actions should be out of a job. He's clearly proven himself to be incompetent, and people regually get sacked for far smaller issues with impacts that don't stay in the national news for weeks.

Maybe they could outsource their CEO position.

What does this mean for your job?

This particular incident doesn't mean anything, however I have my current job as a result of the company I am working for taking IT back in house due to poor experiances of outsourcing. And they did get fed up and shift suppliers several times as well so it's not an issue with a single supplier.

Does this spell the end of cheapo outsourcing deals?

Probably not, but it should do. Outsourcing has it's place (small orginisations often simply can't hold all of the required experiance in house) but I think you need to have your own staff inhouse.

[Scarborough Dave]

You can outsource or offshore for all I care!

You can delegate the thereby:

1. Offshore the process

2. Outsource the Process

But you cannot and should not try to

Delegate your responsibility if things go wrong unless you have lots of legal papers signed and an independent third party checking what you are proposing is the best LONGTERM decision.

I still see this in some management I meet; just because they have delegated a job or task that they are somehow devoid of responsibility when it goes wrong.

This trait is often proportional to how high up the tree the manager is!

I still meet some Directors who have no responsibilities at all as they have delegated them all to middle managment or worse an outside consultant!

[phlashbios]

It's still not fixed...

The press is currently reporting this as if the problem is now fixed. It most certainly isn't.

As an Ulster Bank customer, we were last in the queue to be sorted. At the time I write this, I still cannot see what direct debits have gone out of my account for June (in fact no direct debits at all are showing for the last week in June) and I currently have no idea what has been paid and what hasn't or how much money I really have available, given that I don't actually know what has gone out.

At this point in time, I am due to be paid again on Monday, still having no idea what my financial position is other than a statement from Ulster Bank saying "customers accounts will need some reconciliation over the coming days and weeks".

It is an absolutely ludicrous situation, and beggars belief that RBS group are a month into this problem and have still not resolved it.

[Aitor 1]

Re: It's still not fixed...

You should open a bank account in another bank, get all your money (the indicated money you have) and close the account.

[Anonymous Coward]

Re: It's still not fixed...

Sorry for the anon posting, but there are work reasons for that.

I am also an Ulcer (sic) Bank customer and my account is mostly OK apart from my mortgage still hasn't went out of my account.

From what I have been told when things went belly up in the RBS :

It may have coincided with a server upgrade which the UB where carrying out, and apparently no backups where taken, coupled with the immediate scrapping of the old server

OR

The backups could not be restored

I have heard both stories multiple times from widely varying sources, like the RBS they had got rid of as many experienced IT staff as possible (after all every decent director knows that IT staff are simply a drain as they contribute nothing)

In any case they eventually started processing all transactions manually (including recalling staff who had retired in the last couple of years)

Of course one retired staff mentioned to me that when they where being trained on the new systems a couple of years ago that there where issues with the system, but that there was no money to fix the faults

In my case I was without access to my pay for 23 days (an Illuminati conspiracy obviously) and things are still incorrect.

Currently I will limp along until I can get some sort of compensation from the idiots (had to take a day of work to pay several bills in cash and having to take time out of work to stand in queues to get an 'advance' on my wages)

[Anonymous Coward]

Re: It's still not fixed...

Yeap... my Ulster Bank creidt card shows my payment from the 5th reaching the credit card account late on the 16th, then being removed from the account yesterday resulting in a late payment charge and an interest charge today.

So the initial cause has been fixed but issues are still on going.....

[Ross K]

Bank Of Ulster?

Ulster Bank, you mean?

[Anonymous Coward]

knew this would happen at some point... im one of the IT staff they allowed to leave and worked with these "offshore" guys in india.. not there fault but the word useless come to mind often when in contact with them

[Wanda Lust]

Culture

See the problem is that the offshore wallahs don't have the experience of what "sh1t hitting the fan" means. I've been there, in a professional/business capacity, the offshore culture simply doesn't have the experience to deal with operations of this importance.

This is a terse post, it may appear politically incorrect but it's Friday evening & I'm tired.

[Eponymous Howard]

From a very reliable source...

***The Reg uncovered that the person responsible worked offshore***

Nope.

-The "subtle" error in updating batch scheduling software was made by an experienced, UK-based person.

-But with no-one in the UK left with a proper overview of the entire system, when the shit hit the fan India was instantly and completely overwhelmed, and had no idea where to start or who to contact.

-RBS Group pulled everyone in and stopped all other project work. People called out of retirement, past contractors etc were all contacted and many are still working 24 hour shifts. Triage was needed, with NatWest prioritised (most customers)

-But here is the scary bit.

Only three of the 4 main banks had had the update applied (NatWest, Ulster North and Ulster South). Had RBS itself also had the software, then the situation would in all likelihood have not been recoverable. Think about that.

This will. I am assured, rumble on for months and the opportunities for fraud are gigantic.

[Anonymous Coward]

Current Employee

I'm a current IT employee for Citizens (A RBS owned bank in the US) and I can see first hand what a mess RBS has made of our Bank. Lots of hard working, talented people being let go because their jobs are being shipped to India and promotions are more based on politics than experience. One good this has come out of this however, they are halting any significant internal group changes until 2013 so this can be sorted out.

[Georgski]

Transcript

I presume El Reg will post a transcript after the chat?

(- Ex RBS staffer, now in a really bad timezone)

[Anonymous Coward]

Re: Transcript

Yep.

[Dominic Connor, Quant Headhunter]

Re:Transcript

The question is when the Transcript is made into a movie, who will play me ?

[James 47]

Re: Re:Transcript

A Legoman of course

[J. R. Hartley]

Bank Of Ulster

What's that then?

[Neoc]

Remember: Offshoring makes perfect sense to management

Because their yearly bonuses are based on how much they saved *this* year, not how much their idiotic move will cost the company 5 years from now.

[Anonymous Coward]

More interestingly:

Who's next?

All the banks are in the same boat. They've all been laying off experienced IT people left, right and centre. This isn't the last "noob with un-reviewed access to live systems" problem we're going to see in the banking sector.

[Anonymous Coward]

Re: More interestingly:

It isn't just the banks you know.

The Ministry of Silly Walks^H^H^H^H^H Ideas has been working hard in the public sector as well.

You really don't want to know how much of your defence, policing, healthcare and prison service has been sent to external companies. That's what the PFI was all about you know.

[Anonymous Coward]

customer view

From my perspective - as a business and private customer - RBS is a complete and utter disaster - considering the idiots they have working for them in the UK off shoring could only be better.

Planning on changing soon - nothing to do with the recent cock-up just the ongoing incompetence.