at 13:24 they sent me a text apologising
On the fourth day of a IT systems choke-up that has left customers unable to access money and in some cases unable to buy food or travel, Natwest and RBS – which both belong to the RBS group – still have no idea when the issues will be fixed. A spokesperson said the banking group had been working overnight to fix the problems …
Friday 22nd June 2012 13:28 GMT Anonymous Coward
Friday 22nd June 2012 13:38 GMT Anonymous Coward
Friday 22nd June 2012 13:39 GMT Anonymous Coward
Friday 22nd June 2012 13:51 GMT Anonymous Coward
Their backend systems are a joke
My girlfriend has just left Natwest as a business manager mainly due to the horror of the backend IT systems - they are numerous, broken and useless - which means you can't reliably (real examples) : send out checkbooks (businesses still use them), close accounts, take signing people off accounts (led to massive fraud), change addresses(!), or do many other normal things.
Her advice to everyone ,whether business or normal customer, is to leave Natwest/RBS. She has just seen far too many people screwed over by the outsourced control centre's actions or inactions. It even took her 6 months and 4 tries to get her own name changed back from her old married name.
My favourite was that she was locked out of her bank computer account - and couldn't get anyone to unlock her account since the "two secretaries who still know how and have permissions to unlock accounts (across all of England) were on holiday". So she couldn't do anything for 4 days until one of them came back.
The IT systems have been slowly breaking, system by system for a couple of years now.
Friday 22nd June 2012 14:07 GMT Anonymous Coward
Sorry calling BS on this one. As an ex-RBS employee who was caught up in the off-shoring to India I have a right to bitch about them, but for a lcoked out account a simple call to the help desk and they would have just got her to get a full time employee who could log on to fill out an electronic form and then give a second person's name for confirmation of who we was. These two would get the two halves of her new temp password.
Longest myself or any of my colleagues were lcoked out was only a couple of hours. Normally it would take just 10-20 minutes
Friday 22nd June 2012 17:38 GMT Anonymous Coward
Re: Their backend systems are a joke
Wow. I'm not sure if it's "2 secretaries for the whole UK". But it may be a local problem. IE, if the manager is on holiday, you cannot get their sign off on a password reset/IT assistance. :D
I know someone in an insurance company who did similar to get himself 2 weeks off work.
Friday 22nd June 2012 14:20 GMT digdilem
Am I alone in being very suspicious that this is not a security compromise? I don't doubt the it teams are working hard to fix and won't allow credits to go ahead until they are sure the system is safe.
I had to tell our workforce today, wage day, that if they bank with Natwest (as I do) they won't be getting paid today as usual. Not fun.
Friday 22nd June 2012 14:27 GMT Caff
I would be very suprised if it was a security issue, there are many more ways to mess up a batch and corrupt a database. However most of these are quickly recoverable within a few hours and happen more often than banks would admit. As long as mainframe ops and support do their job noone knows they do it at all.
Friday 22nd June 2012 16:10 GMT JimmyPage
So serious they are opening on Sundays
The problem is (as I have stated before) is they will have to field legions of claims where their snafu has cost people money. There was a story on the BBC of a guy who hadn't recieved his wages, and couldn't afford to pay for a headstone for his stillborn daughter. I wish them the best of luck trying to argue that down in the "court of public opinion"(c)
(c)Harriert Harman 2009.
Friday 22nd June 2012 16:19 GMT Grivas Bo Diddly Harm
I opened an account with Williams and Glyns in 1981, and they were excellent; after a while it transformed into The Royal Bank of Scotland before truncating itself into RBS, with the emphasis on the last two letters. Even after [Sir Fred's] Mr Goodwin's departure the branch network was still solid and, from a user's point of view, the online services were good.
Then I heard that RBS England was going to be passed over to Santander and alarm bells rang good 'n' proper - they'd done me over years before when they took over/rebranded Abbey [National].
I moved (hence the coat icon) over to Co-op - on the basis that they seemed the least worst and owned, at least notionally, by its customers - and completed the transfer of D/Ds, S/Os, funds and notified anyone paying in last month. Just in time, it seems. Now maintaining the accounts with zero balances just to piss off RBS/Santander, though must keep an eye on sneaky account charges being introduced.
In all my dealings with the phone banking staff over the years I unfailingly received good, polite, friendly service and it saddens me to leave. Good luck guys.
And I work for an organisation that itself is still worshipping the false god of outsourcing, so my sympathies go out to all those former RBS IT staff who have been dumped.
PS, Listening to an RBS Spokeswoman, Susan Allen, on the PM programme. Apparently this failure wasn't expected. That''s OK then...
Friday 22nd June 2012 16:36 GMT avatar111
Still at it
Strangely enough only last week Lloyds Bank signed a 'no regrets' contract with Wipro to replace skilled Open Systems contractors with off-shored staff.
I guess those contractors might be able to amble over to NatWest and amble over to Lloyds when it goes pear-shaped there.
I wonder if NatWest has got around to ringing-up the newly-redundant skilled staff and offering them lots of dosh to come into the office this weekend?
Friday 22nd June 2012 19:31 GMT Banker
They've outsourced much more than that!
I worked for RBS during and after the merger with Natwest, I left their Global Financial Markets Department in 2004 after a 5 year stint. They had already moved some IT functions to India at that point and have continued to do so year on year since. The numbers some people are quoting 1600/800 are possibly the more recent figures, the total is way way beyond this.
The comments on documentation are comical, as if a document is the thing you turn to at a time of crisis.
The fact is, when you work closely with systems and the business users, you understand not only the quirks of the systems, but the risks and consequences of failure. You work with those users on the work around solutions that will get the banking day complete.
They haven't just outsourced the IT staff, but the very experienced and valuable back office / operations staff that would work with IT staff to solve the serious issues. I beleive these guys are mostly posted out in Singapore, who probably have never met the IT staff in India. The unseen cost of outsourcing is a compounding loss of shared experience and commitment, which becomes accutely apparent when the sh!t hits the ... cash machines
The chaps I trained out in India were nice enough, but they simply lacked the knowledge and experience of Finacial Markets trading, trade and settlement processing, Swift messaging blah blah and the risks involved.
I'll be drinking with a bunch of ex RBS/Natwesties soon enough, where we'll all be saying.....
"WE TOLD YOU SO!!!!!!!"
I doubt any senior manager in RBS/NW is going to say the outsourcing was to blame and change anything significant.
Over the next 10 years, I would expect this sort of failure to happen to vertually all the big UK banks. Hopefully it will become such an inconvenience to the UK public, the government will have to step in and regulate service expectations, a kind of SLA for the public......... then maybe, the management will be forced to value to kind of 'Support Culture' I enjoy to provide. Then maybe.... the jobs will start driffting back to the UK once more.
Friday 22nd June 2012 22:14 GMT Keith 17
Re: They've outsourced much more than that!
The comments on documentation are comical, as if a document is the thing you turn to at a time of crisis.
Well said sir, you can tell on this thread that there are some experienced devs.
Documentation is _the_ last thing I would go near at a time like that. In fact, I'd go as far as to keep _all_ my devs well away from documentation at such a time. Why? Because I have no guarantee of the accuracy of any given document. You can't unit test a document, it wasn't compiled, it likely wasn't reviewed, was it even finished? Was it updated when changes came along? I don't know.
Innacurate documentation leads to bum steers, it tells you to look in the wrong place, it describes how things work when it's not the way they work at all. Code, on the other hand, is harder to read but tells no lies. Commented code is more valuable in times of troubleshooting than documentation because it's concise and granular.
I've been a dev for over 25 years and I firmly believe that bad documentation is far worse than no documentation at all.
If the devs in this case were frantically grasping for the documentation, I'd be very worried indeed.
Friday 22nd June 2012 23:36 GMT Anonymous Coward
Friday 22nd June 2012 20:09 GMT All names Taken
Friday 22nd June 2012 20:22 GMT mr33
I used to run NatWest IT 20 years ago and I suspect many of the back end systems are similar to then, but with a multitude of front end systems added for online and mobile banking etc. For all the comments on documentation and outsourcing I'd have to say:
This feels like a major batch scheduling cock up and/or Db corruption to me. Given the length of time it is taking it would seem like they've also screwed up what might have been a straightforward rollback or recovery process. However good your documentation or ITIL processes or scheduling automation it's still possible for a production control issue to happen, but you do need people who understand how it fits together not to make it worse... In my experience that's not the people that are outsourced, at least, I hope they kept a few people who know which batch job needs to go after which...
So I think this is less about app programming and more about production control and operations...
In fact outsourcing the ordinary application programming makes sense and it's naive to blame this on outsourcing. It's often the case that the so called 'experts' that have been around for ever are the worst at documentation or training others because, hey, they wouldn't be the 'guru' anymore. I usually get rid of the guru's as they are often the blockers in getting the applications updated and written properly.
Friday 22nd June 2012 22:31 GMT Keith 17
Saturday 23rd June 2012 02:01 GMT mr33
Well I meant 'ordinary' as opposed to 'a bit special'. There is a difference - for a company with the diversity and scale of NatWest it makes sense to differentiate what it decides to outsource. I might decide to have different level of expertise and experience coding my fraud and risk algorithms than I do doing some simple customer service screens. Outsourcing simple stuff is obvious, outsourcing core critical stuff is riskier. Anyone that says outsource nothing is being as naive as someone that outsources everything.
Saturday 23rd June 2012 22:38 GMT Grivas Bo Diddly Harm
..." 'ordinary' as opposed to 'a bit special' ".
And there you have a neat encapsulation of the outsourcing problem. It's the 'ordinary', less experienced, programmers and developers of today who, with experience and training and identified talent, become the 'special' programmers, maintainers and developers of tomorrow. By cutting that link you might make immediate savings but you've bolloxed yourself for the future.
But what do I know, after all I only work for an organisation that has done exactly that and then finds itself over a barrel when it does need people to jump through hoops in double quick time.
Sunday 24th June 2012 06:53 GMT Anonymous Coward
"It makes sense to outsource", eh?
Mmmm. Tell that to the FD or TUI Travel plc, who was shown the door when they cocked up their IT and Finance after they outsourced and offshored it all. I very much doubt they saved the £117m they had to write off after over-stating their results.
Offshore activity at minimum cost is absolutely fine when the "value at risk" is a T-shirt, or a cheap plastic toy. When it is the integrity of a company's core financial systems and data, maybe the relatively modest costs savings need to be placed against the reputational damage, and the costs of rectification and compensation. Look at the Cable & Wireless/IBM spat back in, when was it 2001? C&W thought they'd save money, and in fact they found that they were getting a worse service and paying (IIRC) about £120m a year more than they were before - all went to the high court, and on the day of the case IBM came up with an out of court settlement. My own employers have outsourced IT to HP, and the like for like operating costs have gone up; we've cut the projects budget to keep costs within budget, and as a result we get less for more. Meanwhile the directors think that the outsource has been a huge success because total costs have gone down - they're too far from reality to understand that we pay more for the basics, and now have less being invested in new systems and enhancements than before.
In terms of "ordinary application development", that's what British Gas did with Accenture for a utility billing system, resulting in a huge project failure, regulatory intervention, and a £180m dispute that went all the way to the high court, and resulted in BG cleaning the floor with Accenture. Was that the sort of "ordinary application development" that you think should be outsourced?
I'm sure you could come back with many examples that have worked, but the risks undoubtedly are far greater than the clowns signing the deals will recognise, and the savings more modest.
Friday 22nd June 2012 20:42 GMT Anonymous Coward
I'm a security consultant who works for a rival but banks with Natwest. We all know that IT systems fail from time to time and sometimes these make the news but the duration of this is extremely worrying to me.
I've been impressed with the Natwest mobile app and thought that they must have some top notch security if they were offering 3rd party payments and cardless cash withdrawals. So I decided to take a look and it took me 20 minutes to reverse engineer it and patch so that it captured my passcode as I entered it ! I was shocked that it was so sloppy and had no discernable security measures at all, this can't have been pen tested by anybody remotely competent. This is bordering on the criminally negligent, so I've sent my findings to the FSA and needless to say I'll be moving all of my accounts on Monday.
Obviously they have nobody skilled in risk and security left either.
Sunday 24th June 2012 13:07 GMT Anonymous Coward
Monday 25th June 2012 17:51 GMT Anonymous Coward
OK, let say this patch captured the GUID, Passcode, TWK, IssuerName (terms meaningless to you or I but the people that wrote the app would recognise them as everything that is required for logon). Now let's say it then transmitted these details to a criminal's phone and now the criminal has complete takeover of your account including the facility to take cash out without a debit card. To easily install my patch I need to be on a rooted (Android) or jailbroken (iOS) phone so the first line of defence would be to ensure that the app doesn't install or run on either of those. No such detection, so I can install my 'malware' by hiding it inside a cydia or android app or even with a well-crafted drive-by download via a well-know web exploit.
There are then extra levels of defence that I would expect any competent banking app to deploy but this has none - sheer negligence.
Friday 22nd June 2012 20:48 GMT Anonymous Coward
Friday 22nd June 2012 22:41 GMT Jon182
So pleased I left them. Now with First Direct i.e. HSBC. Would not touch any of the nationalised banks with a barge pole. This on a Blackberry scale in terms of screw ups. Used to support payment processing systems at a mobile operator but now look after mobile data. No glacial batch processes to worry about.
Friday 22nd June 2012 23:05 GMT Anonymous Coward
Due Process or Fraud
This is either down to a process similar to what is detailed here:
Causing a massive tailspin when it collided with the next process and the dominoes ended up in a heap on the floor.
Mahoooooosive fraud attempt in the BACS or CHAPS process, which is why fast payments are working but all else isn't, meaning that all BACS CHAPS Batches are having to be manually checked with real eyeballs in triplicate before being allowed to pass into reality.
FirstDirect is the home of my money thank (insert deity of your choice), but my employer uses GnatWest so I fear for my salary arriving there this month.
Saturday 23rd June 2012 02:00 GMT zoodle
Unhappy RBS/NatWest customer? Vote. Feet. Now! Plenty of competition. Many banks now iron-clad by national governments.
Unhappy creditor. Meet and commiserate with fellow creditors, then serve class action suit.
Unhappy debtor? Wheee! RBS/NatWest will cover all your accidental failed transactions and penalty charges. Go spend.
Unhappy RBS/NatWest staff? Whinge about rejected DR plans. Find new job. Now.
Saturday 23rd June 2012 02:00 GMT Anonymous Coward
I bank with NatWest and went to look at the update on their website (http://www.natwest.com/personal.ashx). My eye was caught by this comment from another customer:
"You bunch of s**theads , I want my wages now , I'm fed up waiting . Need my money. And no I didn't choose to be paid by natwest sue from south of England
by Pi**edrightoff from Kent on June 22, 2012 at 9:59 pm "
I have the screenshot :-)
Saturday 23rd June 2012 06:43 GMT Spoonsinger
Nice, but following looked more proactive....
"I'm going to take a poo on your doorstep you inbred mugs. If I don't get my money in my account within 24 hours I'm going to chain myself to your qc naked and play with my todger. I think your employes should work naked for compensation #SayNoMore
by I hate natwest from Hmp Bristol on June 22, 2012 at 11:12 pm 0 comments "
(also have screenshot :-)