back to article Natwest, RBS: When will bank glitch be fixed? Probably not today

On the fourth day of a IT systems choke-up that has left customers unable to access money and in some cases unable to buy food or travel, Natwest and RBS – which both belong to the RBS group – still have no idea when the issues will be fixed. A spokesperson said the banking group had been working overnight to fix the problems …

COMMENTS

This topic is closed for new posts.
  1. nsld

    that's what happens

    When you slash a workforce, heads should roll, but they won't, of course "lessons will be learnt"!

    Can customers charge 30 quid a day for this? Must be worth a shot

    1. Pete 2 Silver badge

      Re: that's what happens

      > of course "lessons will be learnt"

      Generally the lesson that is learnt is that the bank in question can futz around for this, particular, length of time without anything bad happening to it's senior staffs' employment prospects, the bank's long-term reputation or in regard to shareholder backlash.

      No doubt when RBS carry out a post-mortem, they won't actually find the root cause of the problem (it's the network, stoopid!) but will blame some third-party: either outsourced, software supplier or infrastructure. They will then issue a suitably smug contrite press release about how they've "taken steps to make sure this never happens again", award themselves large bonuses for the successful cost-savings, take the regulator out for a very good lunch and prepare their CVs to move on and stick it to the next financial institution on the list.

    2. Anonymous Coward
      Anonymous Coward

      Re: that's what happens

      As somebody who is facing redundancy after 20 years due to his job being outsourced to India, I can't help but hope that this kind of shit happens to my current employers.

      I won't laugh.

      Much.

    3. nsld
      Mushroom

      Re: that's what happens

      nice thumbs down from the PR troll at RBS

      Perhaps if you useless cunts invested in making your systems work and not trolling forums looking for negative comments about the monumental goat fuck perpetuated by the bean counting fuckwitts at the top of the steaming heap that is your dying business you might not be ankle deep in shit whilst doing a headstand.

      Hurry up and just die, you only got bailed out because Gordon Brown is a jock and we all know how good he was with the economy, he should have let your toxic, debt ridden whore of a bank go bust at the time.

  2. John70

    Out-sourcing is bad

    "The problem is that IT systems have become vastly more complex. Delivering an e-banking service could be reliant on 20 different IT systems. If even a small change is made to one of these systems, it can cause major problems for the whole banking service, which could be what's happened at NatWest. Finding the root cause of the problem is probably something NatWest is struggling with because of the complexity of the IT systems in any bank."

    This is why out-sourcing IT is bad. You fire the permanent staff who knew all the quirks of the system and would have pinpointed the problem in no time at all.

    And now the decision to fire IT staff has come back to bite them in the arse.

    1. Crisp
      Coat

      Re: Out-sourcing is bad

      "This is why out-sourcing IT is bad. You fire the permanent staff who knew all the quirks of the system and would have pinpointed the problem in no time at all."

      I'm sure that the project manager made absolutely sure that there was adequate time to complete all documentation and each quirk and bug is accurately recorded so that the next guy to come along would have a fighting chance.

      1. Anonymous Coward
        Anonymous Coward

        Re: Re: Out-sourcing is bad

        Not, sure, if, joking....

        1. Crisp

          Re: Out-sourcing is bad

          Do I need to add /sarcasm to the end of that post? :)

          Good documentation helps prevent future disasters, and speeds up the resolution of problems. Good documentation should save you time in the future. The problem is that few people seem to realise the importance of good documentation until they don't have any.

          1. sugerbear

            Re: Out-sourcing is bad

            [quote]Good documentation helps prevent future disasters, and speeds up the resolution of problems. Good documentation should save you time in the future. The problem is that few people seem to realise the importance of good documentation until they don't have any.[/quote]

            Haha.. 3..2..1 you're back in the room :)

            Honestly, you have never worked in a real IT department then. People want to do the minimum in terms of effort and time, and documentation (updating it, managing it, reviewing it) will be the first thing that gets chopped and dropped when deadlines approach or the project finishes. IT Departments ignore the need for decent documentation or its control because it is expensive to manage.

            You can probably rely on the original spec being right, but any subsequent updates will always be poorly document. In fact a decent programmer using a decent language should produce decent self documenting code anyway.

            I worked in application support and trust me the last place I would look when fixing things was the documentation. Better to see what the source code is actually doing.

            1. Peter Gathercole Silver badge
              Coat

              @sugerbear

              It has not always been like this. I've been working in Data Processing (remember that!) for over 30 years, and there was a time when good best practice, BS5750 and it's follow-ups like ISO9001 were actually valued. But this was back in the days when computers were expensive, and it was seen as valuable to invent in people and process to get the maximum value from your high outlay.

              Of course, everybody bitched about having to write the documentation, but at least the management bought in to the overall need for it, and factored time into the project plans, because these standards said it had to be done. Sometimes the docs were junk, but often they contained useful information. And the more documentation you wrote, the better at writing it you became.

              Nowadays it's all about trimming the fat, over and over again, and if the managers complain, they get trimmed themselves and replaced by others who are happy to comply. This means that the barest minimum is done to get a service kicked over the wall to support, the support teams have no way of pushing back against a poor service, and then this happens.........

              I'm now seen as a boring old fart, locked in the past, so I'll just go and get my Snorkel Parka and go.

            2. Anonymous Coward
              Go

              Re: Out-sourcing is bad

              I like writing code comments, ensuring system/unit test coverage and getting basic things like Interface Communication Documents put together.

              Sure I will avoid writing a user guide, but the first thing I do when I join a project is find out about their build process. I then work to automate that process (projects rarely do until you point out the wasted time).

              Then Once I have Jenkins/Cruise Control up and running I get tools like Cobertura (Unit Coverage), JavaNCSS (Code commenting), Checkstyle, JSLint, PMD, FXCop, StyleCop, Lint, etc... (build scripts tend to be written in ANT or Maven and I have done it so many times I add this over a lunch).

              Then all you have to do is show the system to your team /technical lead and wait. I've yet to see one who can resist all of those lovely build metrics. Once you have their buy in most of the rest of the documentation comes as a matter of course.

          2. Nigel 11

            Re: Out-sourcing is bad

            Good documentation frequently requires the benefit of hindsight. A person with experience of a system will often quickly work out what's gone wrong with it after the problem arises. His knowledge is implicit, not explicit, and couldn't have gone in the documentation.

            Outsourcing, bad. Check. All other forms of de-skilling, likewise. Monkey see, monkey do, doesn't work for what the monkey's never seen before.

          3. Lusty

            Re: Out-sourcing is bad

            "Good documentation helps prevent future disasters"

            ROFL - who are these IT people that read the docs then? I've certainly never met one!

            1. Wensleydale Cheese

              Re: Out-sourcing is bad

              @Lusty

              "ROFL - who are these IT people that read the docs then? I've certainly never met one!"

              Those will be the people who start a new job and then have to wait for three weeks to get login credentials, security passes etc. :-)

          4. Anonymous Coward
            Anonymous Coward

            Re: Out-sourcing is bad

            "Good documentation helps prevent future disasters, and speeds up the resolution of problems. Good documentation should save you time in the future. The problem is that few people seem to realise the importance of good documentation until they don't have any."

            Completely true. Please can you be on my next programme board.

            Unfortunately when explaining to your project/programme sponsors, before the disaster, that you would need to have an analisys team with the following skill sets (x..z) go over the complex and byztanine structured delivery environment to assess the risks and any project "tuning" that needs to be done, you get shot at for delay and bumping up the cost of the deliery. A decision based upon their profound understanding of IT and system development issues, as demonstrated by their ability to use twitter!

            After the disaster, the sponsors then all forget the warnings and advise you gave, that they over-ruled, and then blame the PM and his delivery team.

            Yep...got the T-Shirt

        2. Ian 31
          Happy

          Re: Out-sourcing is bad

          Of course he's joking, he used the term "project manager".

          1. Anonymous Coward
            Anonymous Coward

            Re: Out-sourcing is bad

            Think you guys are thinking of a SDM - Service Delivery Manager. Its not the PMs issue, once the deliverables have been produced as the SDM required and its been agreed and signed off as part of going into Live service, its not the project team's responsibility any more.

            Otherwise you would have projects that never end! Now if the SDM had no idea about the Service they are responsible for delivering, and didnt think about the training required and protection of said service, well thats a different story.

            Enough of the Alan Sugar mentality please.. thanks

            1. Stuart Castle Silver badge

              Re: Out-sourcing is bad

              AC 13:12, I take it you know ITIL?

            2. Anonymous Coward
              Anonymous Coward

              Re: Out-sourcing is bad

              Nope, this is failed change and should be handled under warranty by the project.

              I doubt the requirements included "bring the bank to its knees".

              That said, I'd not want to be an SDM who signed up to having no blackout plan...

              1. Anonymous Coward
                Anonymous Coward

                Re: Out-sourcing is bad

                Erm I meant backout plan.

                How cool was that typo :-)

      2. edge_e
        Happy

        Re: Out-sourcing is bad

        It's ok crisp, some of us can still spot sarcasm.

      3. Andus McCoatover
        FAIL

        ...project manager made absolutely sure that there was adequate time to complete all documentation

        ISO 9001.

        'nuff said.

        1. Duffaboy
          FAIL

          Re: ...ISO 9001.

          LOL, LOL, LOL... Yeah right

      4. Anonymous Coward
        Anonymous Coward

        Re: Out-sourcing is bad

        Besides outsourcing, that other over used IT fad of the moment "agile development" could be in play here. Whilst a good technique for most green field sites and systems, I have come across it being used in complex multi-system environments, where it struck me as something similar to playing russian roulette, especially when being carried out by teams that did not a have deep knowledge of the overall delivery environment.

        1. FluffyERug
          Happy

          Re: Out-sourcing is bad

          Couldn't agree more, Agile === Development by the seat of your pants, with at best a one week ahead view of the project.

          I quote a contractor friend of mine "I love agile, its just like I've always coded a bit at a time and fuck the rest until I need to know!"

    2. Tom 38

      Re: Out-sourcing is bad

      Sacking 1.8k permanent IT workers, and replacing them with 800 offshore workers was always going to end like this. Roughly 9 months after they did that, this happened.

      I expect that until now, they have simply been managing the existing systems, and now they have put some changes live, utterly breaking everything, and no-one left has a clue how to fix it.

      The golden rule when you shed staff like this and cut wages, you lose the good people - who will always be able to find a job - and keep the dross.

      1. Anonymous Coward
        Anonymous Coward

        Re: Out-sourcing is bad

        The 800 "offshored" roles have been phased over the last 18 months. The last few dozens of UK staff actually finished up at RBS in May, just four weeks ago.

        1. Matt2012
          Trollface

          Re: Out-sourcing is bad

          Sounds like the person switched off the lights..

      2. Nigel 11
        Alert

        Re: Out-sourcing is bad

        Oh, as little as nine months? These things usually manage to run on auto-pilot for a bit longer than that. Long enough for the seagull-managers to trouser their bonuses and find their next jobs!

      3. Field Marshal Von Krakenfart
        Coat

        Re: Out-sourcing is bad

        Maybe the 800 Indians can't read German.....

        Alles touristen und non-technischen looken peepers! Das computermachine ist nicht fuer gefingerpoken und mittengrabben. Ist easy schnappen der springenwerk, blowenfusen und poppencorken mit spitzensparken. Ist nicht fuer gewerken bei das dumpkopfen. Das rubbernecken sichtseeren keepen das cotten-pickenen hans in das pockets muss; relaxen und watchen das blinkenlights.

        1. Primus Secundus Tertius

          Re: Spitzensparken

          Good to see this one is still around - is at least 50 years old.

      4. MH Media

        Re: Out-sourcing is bad

        From experience, up until now they'll have been able to read through prepared sheets of instructions created by the people they replaced. Now they've had to make a (probably small) change on their own and have come horribly unstuck. Lessons will be learned..

        1. Anonymous Coward
          Anonymous Coward

          Re: Out-sourcing is bad

          @ MH Media.

          Sorry, no they won't, nor will it affect the bonuses of the Upper Manglement.

      5. geekguy
        Pirate

        Re: Out-sourcing is bad

        Sharpen your daily rates ex-employees and offer "assistance"

    3. Anonymous Coward
      Anonymous Coward

      Re: Out-sourcing is bad

      Also shows that potentially documentation and config management was gotten out of date because of lack of staff. Can't just sweep in contractors and say "here read this, sort sh*t out".

      Hopefully the former permies are all being contacted and offered awesome rates to go and help out.

    4. Anonymous Coward
      Anonymous Coward

      Re: Out-sourcing is bad

      Isn't NatWest the bank who makes all the fuss about having tried outsourcing, and found it lacking, they brought it all back to the UK (or was that just the call centres)

  3. Anonymous Coward
    Alert

    "The Queen has been unaffected by the glitches,.."

    Phew, what a relief, haven't slept a wink worrying how she was going to manage.

    As a Natwest customer I can say the at the time of writing this I've had money credited to my account but am unable to transfer/make payments online. Plus direct debits that should have been gone out today, the 22nd, have not been paid.

    1. Crisp

      The Queen has been unaffected

      That's a load off my mind too :)

      Can you still walk into a branch and get money out over the counter? Lets hope you don't get charged for each failed direct debit.

      1. I ain't Spartacus Gold badge

        Re: The Queen has been unaffected

        I took £50 quid out of the cash machine yesterday from NatWest. My salary wasn't showing on the balance though. Oh, and they were unable to give me a mortgage quote or start an application, but weirdly were able to do their internal credit check and decide I was probably a better risk than the Greek government.

    2. Andrew James

      A customer who sent us confirmation that BACS payments had been made on Tuesday has still not had those payments arrive in our account from their natwest account. Its ridiculous.

      All the press is about people not receiving money into their accounts with natwest but there must be a huge amount of people affected by payments not going out from natwest accounts too.

  4. Crisp

    IT People

    Management think they don't need us until the magic box of blinking lights doesn't work any more.

    Then they *really* need us.

    1. Anonymous Coward
      Anonymous Coward

      Re: IT People

      The Elders of the Internet have given me permission to allow you the users of the internet to see the Internet running.'

    2. Rob Beard

      Re: IT People

      I wonder if some of the poor staff they laid off a few months ago will come back in as consultants to fix the issue at something like £10000 a day rates :-)

      Still really feel bad for the poor folks who are Natwest customers, I guess screw ups like this could happen to any bank too. :-s

      Rob

  5. Gordon 10
    Happy

    If one of their competitors had any sense

    They would be offering a free £100 1 month overdraft to anyone with a Natwest debit card who switches their account in the next week.

    Although I suspect most CIO's are just thinking "there but for the grace of god go I". Thats even if they have a CIO and he's not been gobbled up by the CFO's Office.

  6. dotdavid
    Facepalm

    Senior Manglement

    "That loud room with all the boxes in was awfully cold this morning, so I turned all those fans off. It is much quieter now...."

    1. Sir Runcible Spoon

      Re: Senior Manglement

      Unless the cleaners have been in and needed the socket for their vacuum cleaner :)

    2. Anonymous Coward
      Anonymous Coward

      Re: Senior Manglement

      lol - nearly fell off my chair laughing at this. Sooo true. Just the other day I was in a server room and the IT manager said, "need to shut the door quickly, the fans annoy the marketing girls".

      Still part of my thought, nice deployment of server room so that the lonely IT crew get to see the other parts of the business.......

    3. Dapprman

      Re: Senior Manglement @Dotdavid

      Don't joke about it, the first company i worked for the CEO did similar as he felt the aircon cost too much money to run .....

      Fortunately I don't have to go AC for this one as the company is long gone (was bought out rather than senior management stupidity - the CEO actually used to keep profits back (rather than rewarding them to himself) so he could keep the company going with out laying staff off during lean times)

      1. Anonymous Coward
        Facepalm

        Re: Senior Manglement @Dotdavid

        They did the same at my college. The old Apple room was fully air conned and security protected (smoke system, alarm etc). Run a dream except the Admin tools crashed the PCs. :(

        But then they decided to get a fancy PC room next door with 15+ top of the range pcs. Oh, but blew the budget on the PCs only. The room had 1 wall with full windows, facing the sun and they barely opened. I think one or two PCs blew each week. But at least the replacement boards came in quick. :P

  7. thesykes

    Train of thought

    To me, there are two trains of thought that exist in management.

    Train A goes something like: We employ 1,000 IT staff and, because of this, all our systems run smoothly.

    Train B, however, goes: All our systems run smoothly, why do we need 1,000 IT staff?

    Unfortunately, it's standing room only on Train B, whereas Train A has been cancelled due to lack of demand and a bus replacement service is now in operation.

    1. Silverburn
      Thumb Up

      Re: Train of thought

      Comment of the week!

      1. Anonymous Coward
        Anonymous Coward

        Comment of the week?

        More like the year, don't remember seeing three-digit "like"s before... good stuff sir!

    2. NoHttpsHere :(
      Thumb Up

      Re: Train of thought

      Posted Friday 22nd June 2012 11:22 GMT

      thesykes:

      'To me, there are two trains of thought that exist in management.'

      This post nailed it...........

      1. Field Marshal Von Krakenfart

        Re: Train of thought

        That's news to me.... managers think!!!!

  8. JimmyPage Silver badge

    Wait till the lawsuits start

    whilst an hour or two outage would be dismissed by the courts as "reasonable" within the banks T&Cs (which will try to remove any liability from the bank for any loss due to it's actions), I suspect a 4 day outage would be considered unreasonable, and consequential losses *will* be awarded.

    Just the thought of the hundreds of claims through the courts, and having to argue each one on merit should keep the lawyers on the toilets for the next decade.

    Once the final liablity has been costed, I wonder it IT staff will seem quite so expendable on a risk/return assesment. We need a bewigged icon with the caption "wait till the lawyers get there"

    1. Anonymous Coward 101
      FAIL

      Re: Wait till the lawsuits start

      But they have already said that nobody will be left out of pocket. The lawyers have no need to be involved here.

      1. JimmyPage Silver badge
        Mushroom

        Are you really that thick ?

        suppose a business has been unable to access it's account, and pay a supplier, and as a result they go bankrupt ?

        Supposing, thanks to their incompetence, I am unable to pay for my cars service, and therefore can't get to a business meeting ?

        As I said, T&Cs can be used to protect you from consequential losses, in reasonable circumstances. A 4 day, unplanned, unannounced outage is not reasonable in my book, not I suspect would the high court find so either.

        I am taking your comment at face value. I apologise if you were being ironic.

    2. Anonymous Coward
      Anonymous Coward

      Re: Wait till the lawsuits start

      I worked for a building society on mortgage systems and in a decade, we once serious outage, where the system went down for nearly a whole day because of a disk crash. Our overnight update was built up of a series of jobs for different purposes, each creating daily transaction records, with one program at the end that applied them all. The benefit was that if someone buggered up one of the overnight update programs, the rest would still get through, and the program that applied the transactions almost never changed.

      1. Nigel 11

        Re: Wait till the lawsuits start

        A relative once had to pay a day's interest on the price of a house because the payments transfer system caused his completion to fail. The bank wriggled out of its liability citing T&Cs. The glitch was only a few tens of minutes, late in the banking day. Mercifully the person whose house he was buying did let him move in, even though he hadn't yet paid for it.

        But I doubt that the bank could escape liability for *several* days' failure and interest. If they try, the lawyers sure as hell *will* be involved. Wonder if it affects hundred-M completions as well as hundred-k completions?

    3. Anonymous Coward
      Anonymous Coward

      Re: Wait till the lawsuits start

      They won't start. BPO and ITO vendors write and sign contracts every week. Buyers sign such contracts once in a flood. As a result the vendors run rings round the buyers, and there's no chance that the bunglers of RBS would be able to pin something on the ITO.

      And of course, are RBS management inconvenienced? Nope, they've got big credit balances and multiple accounts. Are the 1% inconvenienced? Nope, they bank with Coutts. It's the plebs who spend most of their salary or pension each month who are in the crap. And I can assure you that RBS won't go fighting for justice for the likes of you and I.

  9. Anonymous Coward
    Anonymous Coward

    I thought I heard NatWest (the helpful bank) say somewhere

    'no one will be out of pocket because of this'

    Does anyone with a NatWest account want to lend me a tenner at a bazigilliion percent interest and a trimegazillion quid administration fee until monday and we'll split the compensation when it comes through?

    I guess I'm paying the compensation

    *always lose*

  10. Anonymous Coward
    Anonymous Coward

    Just one more reason

    To have a lump of gold and a suitcase of money under your bed.... and comprehensive contents insurance (always check the small print).

    1. Bronek Kozicki
      Thumb Down

      Re: Just one more reason

      "to have .... " that is investment based on pure speculation on the future value of whatever-it-is-you-want-to-hold. This includes gold, cash or government bonds (hints: commodity rates, inflation, yields).

    2. cliveski
      Devil

      Re: Just one more reason

      Trusting insurance companies more than banks - now that IS brave :-)

      1. Velv
        FAIL

        Re: Just one more reason

        Have you seen how much of the insurance market is underwritten by RBS owned underwriters?

    3. Nigel 11
      Mushroom

      Re: Just one more reason

      No suitcase of money. A well-stuffed wallet will suffice while people still think it's just a glitch. Only gold will be accepted after it becomes clear that TS really has hit TF. And be very careful who you let know that you've got gold, because of the knives and the guns.

      Pray it never happens.

  11. Tom Wood

    Pro tip

    Have two accounts, at different and unrelated banks, so you can always get some money somehow.

    1. AmandaJayne

      Re: Pro tip

      That's all very well, if you're not waiting for your wages to go into one of those accounts...

      1. Nigel 11

        Re: Pro tip

        It's fairly OK even if you are. Just make sure you're clear to overdraw the one that your salary *doesn't* go into (or, better, that it's a savings account, if you have any savings).

      2. Gavin 2

        Re: Pro tip

        All doesn't work when your work place banks with Natwest and cant pay you regardless of your bank

        1. the spectacularly refined chap

          Re: Pro tip

          Are you really saying that you don't have enough savings anywhere to tide you over for day to day expenses for a couple of days? We're not even talking about the big non-cash outgoings - your mortgage, car repayments, insurance etc - those can be sorted out later if need be. It isn't as if this particular snafu has not been extensively publicised. Hell, when I've worked anywhere that involves payment processing all user memos get sent round for issues far less significant than this.

    2. Primus Secundus Tertius

      Re: Pro tip

      I've done that for a long time, since "bank A" fouled up big-time on a direct debit. Now I keep my direct debits with "bank A" and my money "safe" with "bank B".

  12. Gordan

    Proof banks are not systemically important

    Maybe this is a good thing in disguise. It shows that banks aren't important enough to prevent them from going bust. We can make do without them. Maybe we should see this as a preparation run for pulling life support plug on the lot of them.

    1. yeahyeahno
      FAIL

      Re: Proof banks are not systemically important

      Your level of fail at understanding systematic failure and the importance of banks is breath taking.

      But you've clearly either not bothered to read about it and learn, or have chosen to ignore what you've read so there is no point in rehearsing the arguments here for you now.

      1. yeahyeahno

        Re: Proof banks are not systemically important

        For the fools that don't grasp the problem with bank failures, even "technical" ones like this should read this story http://www.bbc.co.uk/news/business-18547149 and explain to me how workers on strike, unpaid wages and customers not having their orders fulfilled is not systematic failure.

  13. Anonymous Coward
    Anonymous Coward

    GNEP

    In RBS parlance (I used to call it RBSBS), a GNEP is a Group Notifiable Event Process

    "A notifiable event is when something happens that could cause damage to the Group by adversely affecting our customers, our finances or our reputation."

    I'd hate to be the poor soul that had to raise this GNEP...

    1. Nigel 11

      "You know all those staff you insisted we let go last year ...."

      <letters>.

  14. Anonymous Coward
    Anonymous Coward

    Outsourcing

    The experience of outsourcing is frequently terrible, for the simple reason that custom software isn't like business cards or the car fleet. For one thing, car production is a repeated action, so you get economies of scale. Custom software is like custom anything, it doesn't scale.

    On top of that, there's often a confusion between the value that internal people bring, which is long-term experience of the software rather than experience of the technology. Someone who's been looking after a DB for 10 years doesn't need to look things up. You ask them how to find out the customer's name and address, and they'll tell you the service to use or the database fields to use.

    What consultancies do is to always look to reduce costs. They'll often start you off with really good, experienced people, then once the consultancy has its feet under the table, they start putting graduate trainees on. You won't know this, or see any discount in your fees. You'll only find out when they screw things up. But this also means that they're moving staff off and on quite frequently.

    The best run IT functions I've seen have been in-house with mostly homegrown systems and a few packages where the package was clearly a good fit. The words "we'll just get SAP and adapt it for our needs" are a sure sign that you have morons running your company.

    1. Anonymous Coward
      Anonymous Coward

      Re: Outsourcing

      "The words "we'll just get SAP and adapt it for our needs" are a sure sign that you have morons running your company."

      Been there, done that, have the scars to prove it. Thank God, I am out of here. So sick of the absolute bullshit being spouted by the consultants that know f*** all about either the business processes used or the software they're supposed to be experts in.

      Really pissed off with managers that refuse to use the correct SAP processes, then complain that things don't work, or expect me to instantly fix their cockups when it suddenly stops working because of something that they have done, but refuse to discuss what it was they did.

      Really, really, sick and tired of the marketing bilge pumped out that portrays SAP as anything other than a massive scheme to screw every last penny out of c-level execs that think they are IT experts because they can send an email from their iPhones.

      Utterly gob smacked that anyone can believe it makes sense to outsource the servers when the orignal costs quoted by the hosting company are higher than the internal IT budget.

      So long, and thanks for all the stress

      1. Anonymous Coward
        Anonymous Coward

        Re: Outsourcing

        "The words "we'll just get SAP and adapt it for our needs" are a sure sign that you have morons running your company."

        Nah, the root cause here will be Access '97 - and it'll be a password protected database built by an ops manager too! :-D

    2. irish donkey

      Re: Outsourcing

      hmmmmm Premier Brand perhaps

  15. Scott Mckenzie

    Apparently

    We have been told at work (as it's affecting us there) that the issue was down to an update to the Mobile Banking App, it failed and had a domino effect that was 'unforeseen' and has pretty much broken everything!

    Which is nice.

    1. Tim Almond
      WTF?

      Eh?

      How does the mobile banking app have a "domino effect"? How is it anything but a thin UI that talks to web services to get or put data? How does it stop transactions from an overnight run from BACS getting through?

      1. Nick 6
        Windows

        Re: Eh?

        Well they launched some new payments facility on mobile, and those payments go might go share some infrastructure as the batch or other payments, they could have a knock-on effect.

        For example perhaps there is a much higher volume than predicted. Or those mobile payments result in a much higher CPU cost or transaction time through the shared payments processing engine. Or connections to external interfaces e.g. for faster payments.

        Don't have an inside information though, these are just guesses.

        1. mrfill
          FAIL

          Re: Eh?

          How about this for a guess...

          They made a change and didn't test it properly.

          Another money saving master plan...

          1. I ain't Spartacus Gold badge
            Happy

            Re: Eh?

            Hmmm testing. I'm sure I've heard of this from somewhere...

            Maybe we should get some of that?

            An RBS executive

          2. Anonymous Coward
            Anonymous Coward

            Re: Eh?

            I'm sure the QC process (since the QC teamed was "right sized and shifted to a low cost region" is now

            it compiled with no major errors -ship it

        2. QuiteEvilGraham
          FAIL

          Re: Eh?

          Ah - if I recall RBS correctly, then the sequence of processing will be something like this (admittedly speculative, things may have changed).

          RBS have a system called Accounting Interface. It applies various accounting "rules" which reconcile the path of monetary transactions from, say, a cash withdrawal from an ATM (say, a Barclays one) back to the original customer account from which the money is debited. These transactions are then fed into the main batch account update program, and everything should reconcile at the end of exercise.

          So a mobile payment would result in (possibly):

          passage through some gateway to be added to a list of mobile transactions, which would then end up in a transaction file fed into their batch systems (plenty of scope for bog-up here).

          In batch, these transactions are typically expanded by other generated transactions such as:

          a) a debit from a holding account for the new mobile app

          b) a credit back to that holding account from the customer account

          c) a debit from the customer account

          d) a credit to another holding account for transactions to the target of the payment

          e) a debit from that holding account when the payment is transferred to the bank of the payee

          and so on...

          If the accounting rules governing each of these transactions are bogged-up in some way, the main batch account system (which updates EVERY account) will not reconcile properly, and panic will ensue. To fix this, the transactions via mobile would have to be corrected and everything re-run from that point. And then re-tested quite a few times to make sure they are correct this time.

          Anyway, fun to speculate exactly what went wrong (I don't bank with RBS BTW).

      2. Oliver Mayes

        Re: Eh?

        Or perhaps the mobile app had some debug code left in it that wasn't supposed to run on their live servers?

        One rogue DROP statement can cause havoc if they don't have proper backups and DR policies.

        1. Anonymous Coward
          Anonymous Coward

          Re: Eh?

          If it were a simple dropped table or the like a disk snapshot recovery would be the first port of call, followed by tape recovery. RBS have a truly massive storage and backup & recovery infrastructure.

      3. Anonymous Coward
        Anonymous Coward

        Re: Eh?

        "how does it have a domino effect?"

        Because the mobile app is not simply accessing a web front of existing services, it is accessing the payment transactions back end which will need to have been modified to handle the different types of transactions, and this potentially requires changes to the entire CAUSTIC transaction processing service.

  16. Ye Gads
    Alert

    On the plus-side, all RBS Group employees are forced to have an RBS/Natwest account into which their salary will be paid. This happens around the 24th of each month.

    Yep, you guessed it: RBS and Natwest probably won't be able to pay their staff until they fix the problem so they will be feeling their customers pain.

    1. This post has been deleted by its author

      1. Snowy Silver badge

        So got paid just before it all went tits up...

    2. Crisp

      You can bet that Philip Hampton and Stephen Hester wont be going hungry next week. The only RBS Group employees that will be feeling any pain are the low paid staff subsisting from paycheck to paycheck.

    3. sugerbear

      That doesn't apply to the guys in Mumbai. You know, the ones fixing the problem ;o)

    4. Anonymous Coward
      Anonymous Coward

      Actually...

      Salaries are paid into accounts on the 18th of the month, unless it happens to be a weekend or a bank holiday. Staff were paid on Monday.

  17. DrXym

    "Vast complexity"

    I would not be surprised at the end of the day that the failure was caused by something extremely trivial. An expired cert, a botched software upgrade, a misconfigured proxy, a power failure inside a centre, some doofus typing rm -rf in the wrong folder.

    1. Sir Runcible Spoon

      Re: "Vast complexity"

      "An expired cert"

      I read that as an expired cat!

      Still, if they've run out of cheese, the mice will be running everywhere and if they've only got one cat (the rest are all in India) then I'm not surprised it's probably dead from extertion.

      1. Nigel 11
        Joke

        I read that as an expired cat!

        Have you ever seen the damage that mice will do to the wiring under the floor of the server room?

        1. Pete 2 Silver badge

          Re: I read that as an expired cat!

          > Have you ever seen the damage that mice will do to the wiring under the floor of the server room?

          Yeah, the USB ones are the worst.

    2. Caff

      Re: "Vast complexity"

      or maybe typing quiesce.....

  18. amanfromMars 1 Silver badge

    Three days and still scratching around for a solution and fix?

    Things appear to much more serious than is being admitted, if you can believe what you read in the newspapers ...... http://www.independent.ie/national-news/ulster-bank-thousands-without-cash-as-bank-fails-to-fix-it-crash-3146374.html .... and on fora you get a feel for the inconvenience caused, but fortunately will all be compensated should they suffer unwarranted charges caused by the glitch? ........ http://www.boards.ie/vbulletin/showthread.php?t=2056676772

    It never rains but it pours, and it is lashing down here today.

  19. Aristotles slow and dimwitted horse
    Stop

    @ Outsourcing...

    "" The words "we'll just get SAP and adapt it for our needs" are a sure sign that you have morons running your company. ""

    Hmmm... I've been on the frontline of implementing and managing SAP projects for end clients 12 years now and I've never heard those words mentioned once. In my not inconsiderable experience the morons on SAP implementations are normally the leaders/management of the 3rd party SIs such as Wipro, Accenture, Capita etc. rather than the end clients themselves; who generally go through quite detailed business cases and product selection processes before they choose ERP systems or related products.

    I'm not sure what your experience is with SAP/ERP, but I would like to point out that I do agree with the first 3 paragraphs of your comment though.

  20. Anonymous Coward
    Anonymous Coward

    Reposted from previous article

    Unless it has changed drastically, IIRC RBS (and Natwest, they migrated Natwest customers to the RBS system after the takeover) update main customer accounts in batch (on an IBM mainframe, natch.) overnight via a number of "feeder systems" (BACS, Accounting Interface ,etc.) through a number of "streams" through their main account update system (can't remember the exact name it had, Sceptre? - something like that) which cover a range of branches. These originally reflected the distance of the branch from Edinburgh, so stream 'A' was branches in the far distant north and run first, allowing the van with the printouts to leave earliest as it had the furthest to go. Seem to recall that Natwest started at stream 'L'.

    The actual definitive customer account updates were carried out by a number of programs written in assembly language dating back to about 1969-70, and updated since then. These were also choc-full of obscure business rules ("magic" cheque numbers triggered specific processing) and I do not believe anyone there really knew how it all worked anymore, even back in 2001. I remember sitting in a meeting discussing how the charge for using an ATM which charged for withdrawals could be added as a separate item to a customer statement and waving a couple of pages of printouts of the source of one of them. The universal reaction was "wow, can you actually read that?". I can't see them having mucked about with that one too much, since good assembler people are like hen's teeth and it was decidedly non-trivial to make any changes to it, but the accounting rules did change frequently in the feeder systems. My bet is that some change has resulted in discrepancies in the eventual output from these systems, and a combination of retirement and redundancies has left them with very few people who know how it is all supposed to work, and therefore identify the cause of the error and fix it. Or they might have just blown the size limit on one of the output files from the various feeder systems (VSAM IIRC, limit was 4GB, unless they moved to the extended types), or possibly the FICON cable just dropped out of the back of whatever device some of the disk volumes live on.

    Anyway, very complex stuff which all has to just work together. Of course, the moral is, complex mainframe systems require staff with the skills, and in this case, the specific system knowledge to keep things smooth. The fewer of these you have, the more difficult it is to recover from problems like this.

    My $0.02. Perhaps someone with more recent knowledge will care to rebut?

    1. The Axe
      FAIL

      Same experience at HBOS

      My partner worked at Halifax/HBOS and their experience sounds exactly the same as yours. Complex spagetti code with stupid naming rules that made the complex software even harder to understand and no concept of testing or proper debugging. Debugging at Halifax is pretty much the use of printf style statements scattered throughout the code rather than using proper tools. Halilfax even threw out Leeds' system which was Unisys and a high level languge for IBM and assembler becasue it wasn't invented here.

    2. Anonymous Coward
      Anonymous Coward

      Re: Reposted from previous article

      I understand that your description of the RBS Mainframe based batch update process is fairly accurate. The source of the problem was a software update to Batch scheduling suite CA7. The upgrade when so well that now there is no schedule to run all of those thousands of batch jobs to receive and make BACS payments, update balance, schedule printouts, etc.

      I am sure the problem with the CA7 upgrade and the unfortunate misplacing of the Batch schedule has absolutely nothing to do the with the last UK based technicians leaving recently. The guys in India of course are perfectly able to cope and fix their mistake. I'm sure they understand how the thousands of jobs in the schedule need to ordered to make sure there is data corruption or loss. After all the problem happened on Tuesday and it's only Friday.

      I wonder how many ex-RBS staff have received very lucrative short term contracts in the last few days......

      1. This post has been deleted by its author

      2. Anonymous Coward
        Anonymous Coward

        In support/respons to AC 13:39

        I'm AC for very obvious reasons having been one of the recent 1000+ to find their roles now being done from Chennai, however I have been speeking to a few ex-collegues who are still there and can confirm that they say the same as the above poster as in a CA7 upgrade was done, went horribly wrong, and was then backed out (which will have been done in typical RBS style - 12 hours of conference calls before letting the techie do what they suggested at the very start).

        My understanding is that most if not all of the batch team were let go and replaced with people from India and I do remember them complaining that they were having to pass 10-20+ years worth of mainframe knowledge on to people who'd never heard of a mainframe outside of a museum. The Indians were keen and willing to try and learn, but with out the years of previous experience will now be deep in the smelly stuff.

        The onyl good thing is being the batch and over night processing that failed, all the data will still be in the sysetm awaiting processing so no one should find they money going missing as a result of tis incident.

        1. Primus Secundus Tertius

          Re: In support/respons to AC 13:39

          I hope they will indeed be able to execute all transactions and make things balance.

          But a few years ago I heard privately of a FTSE 250 company whose accounts got corrupted. When they turned to the backups, either they were already corrupt or they were screwed up during the "restore". So they lost everything, and had to write piteous letters to suppliers and customers.

          I scanned the financial press for months afterwards, looking for a public report, but saw nothing. Full marks to the cover-up team, then!

      3. QuiteEvilGraham
        FAIL

        Re: Reposted from previous article

        If it was a software update to CA-7 and they corrupted (or otherwise lost) the various VSAM datasets which hold the schedule database, then I think that backing out and restoring should have been a fairly simple exercise, and the complete failure of an entire overnight batch run is something they would have noticed pretty quickly. Assuming that they are even slightly competent.

        1. Caff

          Re: Reposted from previous article

          This I can't understand.

          Upgrade ca7 on a test LPAR.. all works fine!

          Upgrade ca7 on production.. panic! Then follow backout proceedure for the upgrade and continue

          maybe a 3 hour delay to batch...

          how can you completely mess that up??

          1. QuiteEvilGraham
            FAIL

            Re: Reposted from previous article

            Well, if as the previous poster says, it takes about 12 hours of conference calls to get anything done, then I guess that they held over until the subsequent night's run to try to re-run everything. Of course, unless things are staged very carefully, they then have to process twice the transaction volume, and there may just be some hard limits on the feeder system dataset sizes which are now too small, or the batch runs now take too long, so the on-line daytime stuff cannot start. And undoing the problems which cascade from there is where you really, really want your experienced system people.

            Which it seems as if they no longer have.

      4. Anonymous Coward
        Anonymous Coward

        Re: Reposted from previous article

        Based on my experience at a large bank that outsourced support to India, I'm sure the guys in there will work diligently to meet their contracted SLA.

        Probably by sending an email requesting further information 15 seconds before going off shift and then not returning to the office for three days.

      5. Anonymous Coward
        Anonymous Coward

        Re: Reposted from previous article

        If it were the mainframe batch scheduler that was failing, there would be a damn site more than just some (by no means all) incoming payments failing.

        1. QuiteEvilGraham
          FAIL

          Re: Reposted from previous article

          It is pretty much disastrous. In RBS world, there are many interconnected systems, some of which can maintain a view of an account for some time, but eventually all transactions need to be reconciled via the main overnight mainframe batch. If this is not done, the account info maintained by these satellite systems (ATM, card purchases, etc) will become stale, and increasingly risky from the bank's point of view. So the CA-7 failure seems entirely plausible. It leaving them in the shit for 4 days, however, is not a situation one would expect a competent mainframe site to find itself in. If this is a consequence of "off-shoring" support, then someone has made a very bad judgement on an essential component of the bank's ability to stay in business and heads need to roll over this.

      6. Anonymous Coward
        Anonymous Coward

        I have experience of CA Job Schedulers

        "The source of the problem was a software update to Batch scheduling suite CA7. The upgrade when so well that now there is no schedule to run all of those thousands of batch jobs to receive and make BACS payments, update balance, schedule printouts, etc."

        Yep. Had the same problem with CA batch scheduling software in the late 1990s.

        It's all a bit hazy now, but there were ways for someone who had seen it before to recover from that situation.

        Ah, in the time it took me to type the above paragraph, it has come back to me that one of my first tasks in a new job in 1998 was to write a bit of code to export then reimport CA scheduler jobs, just in case it all went titsup.

    3. Caff

      Re: Reposted from previous article

      the 4gig vsam limit should have been resolved with upgrades since but they could well have encountered limits elsewhere.

      1. Anonymous Coward
        Anonymous Coward

        Re: Reposted from previous article

        This is RBS, remember. ISTR them cutting their batch schedule from ~7 hours to 1 1/2 hours by the simple expedient of remembering that they no longer had IBM 2314's and changing sequential datasets from LRECL=80 BLKSIZE=800 to half-track blocking on 3390's. Don't get me started on METACOBOL.

  21. derbs
    Pint

    BACS?

    I transferred a couple of grand from HSBC to my natwest account last night. It uses the faster payments system. It worked fine, money was there in minutes. Now I think all the people having issues are waiting for salaries/benefits etc, which I believe use the old BACS system. So my guess is RBS's BACS gateway has gone tits up. Unacceptable though

    1. Anonymous Coward
      Anonymous Coward

      Re: BACS?

      You appear to have enough money lying about to test these things on a whim. I like that.

      I suggest you expand this testing. Transfer a couple of grand to my account, and I'll let you know if it works and how quickly.

      1. derbs
        Pint

        Re: BACS?

        Haha! Sorry if that came off as boasting. That was actually my salary this month. I was just pointing out I think it looks like a BACS issue. I think it's funny how cash withdrawals/faster payments don't seem to be affected, whilst BACS payments do...

        1. QuiteEvilGraham
          FAIL

          Re: BACS?

          You're missing something. Your transfer may show up in whatever webby system which shows incoming transactions. Your real account balance is updated overnight in batch on a mainframe.

          In other words, RBS/NatWest do not actually know what balance any of their customers actually have in their account, have not done so since this failure occurred, and will not know again until the problem is fixed. So good luck getting your money back out, since they do not know how much you actually have.

          A Maximum Fail icon is required for this.

          1. derbs
            Pint

            Re: BACS?

            Christ, that was a bit harsh. I'm just using basic logic to hypothesise what the issue could possibly be. My natwest balance is completely accurate, and cash withdrawals/card payments are working fine. People who have been paid via BACS don't have an accurate balance on the other hand. Think the maximum fail might be with you son

            1. Anonymous Coward
              Anonymous Coward

              Re: BACS?

              The systems which tell you your account balance via ATMs and handle that side of cash withdrawal run on a "snapshot" balance which is eventually updated to maintain your "real" balance via batch systems on a mainframe. So they can maintain that view, for some time at least. This "snapshot" will not be updated with your salary, say, until that is processed into your "real" balance, and therefore your salary will not be available to you via ATM or card payments until this problem is fixed.

              "Think the maximum fail might be with you son"

              Unless you have worked on the systems in question (and I have), try to avoid comments like this.

              Makes you appear to be a bit of a tool.

              1. QuiteEvilGraham
                Holmes

                Re: BACS?

                4 thumbs down? Since this is an entirely accurate description of how RBS Group do their processing, one must assume that there are shills out there. Damage limitation?

  22. Tom 38

    The "new cash"

    I've found that sometimes, you just need cash. Relying on banks (and me tbh) to not fuck up or me not losing my card or whatever other reason is going to catch you out eventually, so I keep a couple of hundred quid hidden around my flat. I appreciate that not everyone can afford to do this though..

    It is kind of ironic, given that Natwest have wall-to-wall TV adverts declaring "a better way" and promoting features to allow you to get access to your money even when you lose your card.

  23. HuwLewis
    Pint

    Shadenfreude!

    I like many others work with core banking systems. I'm just so glad that this is not one of my cutomers (nor for that matter my bank). So who is actually spending their Friday calling all the agencies?

    Who's system is RBS using?

    1. Nanki Poo
      Pint

      Re: Shadenfreude!

      You sir win word prize of the day! That word has been repeating in my head as I read the background and think of the number of IT mates struggling for work.

      Maybe this will be the tipping point that happened with off-shored call-centres, a change of attitude towards it. One can hope now that it's actually hurt a corporate so obviously.

      nK

  24. TheBodger

    There is another way, you know...

    "No customers will be permanently out of pocket as a result" - Natwest.

    Personally, I wouldn't trust them as far as I could kick them. I highly doubt that they will compensate people for the time they wasted as a consequence of this mess, for starters. This is the same bank that promised that it wouldn't close the last bank in town, then closed Farsley Branch (which was the last bank in town - see http://www.bbc.co.uk/news/uk-england-leeds-17041251).

    Right now, I only have one thing to say: "There is another way, you know".

    1. A J Stiles

      Trendy Wine Bar

      NatWest used to have adverts making fun of the fact that some bank branches had been closed and turned into trendy wine bars.

      Well ..... the Standing Order pub on Iron Gate, Derby is a Wetherspoons house -- not exactly a trendy wine bar, but ..... guess which bank it used to be a branch of?

      1. Steven Davison
        Joke

        Re: Trendy Wine Bar

        Same happened in Aldershot.... (Where for a number of years they had two branches opposite each other)

        The older building was closed, and sold.... remained empty for a while, until it reopened, as a said trendy wine bar... :P

      2. Anonymous Cowerd
        Unhappy

        Re: Trendy Wine Bar

        Also, the Mannamead in Plymouth.

    2. Alan(UK)
      Coat

      Re: There is another way, you know...

      What do you mean? Farsley is a place in Leeds. There is still a NatWest branch in Leeds. What are you complaining about?

  25. John Young
    FAIL

    And now online banking is down too...

    SSL connection error

    Unable to make a secure connection to the server. This may be a problem with the server or it may be requiring a client authentication certificate that you don't have.

    Error 107 (net::ERR_SSL_PROTOCOL_ERROR): SSL protocol error.

    1. Anonymous Coward
      Anonymous Coward

      Re: And now online banking is down too...

      Not for me.

  26. El Presidente
    Windows

    Switch it all of and backonagain !

    How hard can it be ?

    1. Caff

      Re: Switch it all of and backonagain !

      obiviously never IPL'd a mainframe...

      1. Anonymous Coward
        Joke

        Re: Switch it all of and backonagain !

        Or the RBS BRS (Big Red Switch for those not uptodate with PC history).

  27. Colin Millar

    superior queeny software?

    That'll be the quill pen and the leather bound ledger

    1. Anonymous Coward
      Anonymous Coward

      Re: superior queeny software?

      You beat me to it! I'm not sure if they use in house/out sourced software. But it's much easier for them to actually start their "contingency plan" and get everything keyed in manually or on paper ledgers. No idea how many accounts they have, but there cannot be many millionaires left right now. :(

  28. Mark Barton
    Holmes

    Natwest Online now displays Mortgage Balance

    Hell of a coincidence but logging into Natwest online I can now see my Natwest Mortgage balance - that wasn't there Tuesday.

    1. Dapprman

      Re: Natwest Online now displays Mortgage Balance

      Must be a gradual roll-out as I've had that function for over a month now. Alas it's a recent payments only function, at first i thought I might be able to pay an extra chunk off through the on-payment options.

  29. James 100
    Pint

    "The problem is that few people seem to realise the importance of good documentation until they don't have any."

    Or in this case, with the major cull and outsourcing, the outgoing guys may have known exactly how important documentation would be to their cheaper replacements ... if there was any.

    "Yes, yes, of course I'll be delighted to spend the remaining 37 hours of my contracted time documenting how Sanjeet can do my job perfectly while I'm in the dole queue. What's that? The spam box needs emptying? ... oh dear, seem to be out of time. Documentation: 'try turning the mainframe off and on again at the wall, a couple of times, quickly. if that doesn't work, pee on the socket.' Bye, enjoy the 'savings' from downsizing me..."

  30. James 47

    must... find... that... missing... semi-colon

  31. James 47
    Thumb Up

    Of course, if the panic levels are high enough right now some ex-employee who hasn't burned any bridges should call up a former manager offering his/her services for a few hundred quid an hour.

    1. Steven Raith
      Thumb Up

      A few hundred quid an hour?

      If someone out there knows what's gone wrong, I'd be putting another zero on there as a day rate, and expect a massive completion bonus of at least five figures on there too, seeing as you'd be saving RBSs global repuation.

      You'd get paid what your worth, and after three or four days now, that rate has increased significantly...!

      Steven R

      1. Munchausen's proxy
        Pint

        Re: A few hundred quid an hour?

        "... and expect a massive completion bonus of at least five figures on there too..."

        What the hell, go whole hog, executive-style, and demand a retention bonus for every day you show up.

    2. Anonymous Coward
      Anonymous Coward

      "a few hundred quid an hour"

      Yes, but how are they going to withdraw the money to pay you? ;o)

    3. Anonymous Coward
      Anonymous Coward

      Only...

      a hundred quid an hour!?!

  32. ukgnome

    at 13:24 they sent me a text apologising

  33. Melanie Winiger

    As Bill Gates said

    "Show me the code"

    I would like to see it!!!

  34. Anonymous Coward
    Anonymous Coward

    It's nice to see so many former RBS group employees airing RBS's dirty laundry in the forums.

    There's quite a few independent contractors in Edinburgh now who are available at very favourable rates to fix the problems :)

    You can reach me on 09090 69 69 69

  35. Anonymous Coward
    Anonymous Coward

    So it seems that little Bobby Tables has opened his first bank account.

    1. Sir Runcible Spoon

      First bank account

      You owe me a new keyboard :D

      http://xkcd.com/327/

      1. Anonymous Coward
        Trollface

        Re: First bank account

        After a quick name change to

        Robert');DROP TABLE Accounts;

  36. Anonymous Coward
    Anonymous Coward

    Bank run?

    How many more days before this becomes a self inflicted bank run?

    I wouldn't leave savings with Natwest. And wouldn't trust my salary to them.

    Practically, is there much to differentiate between an inability to obtain cash from a bank, and bank insolvency?

    1. Anonymous Coward
      Anonymous Coward

      Re: Bank run?

      In case you've not been paying attention: Cash can be taken out, it's cash not going in which is the issue.

      Also, it's by no means all the cash going in which is the issue, my accounts seem to be working normally as far as I can tell.

  37. Anonymous Coward
    Anonymous Coward

    Their backend systems are a joke

    My girlfriend has just left Natwest as a business manager mainly due to the horror of the backend IT systems - they are numerous, broken and useless - which means you can't reliably (real examples) : send out checkbooks (businesses still use them), close accounts, take signing people off accounts (led to massive fraud), change addresses(!), or do many other normal things.

    Her advice to everyone ,whether business or normal customer, is to leave Natwest/RBS. She has just seen far too many people screwed over by the outsourced control centre's actions or inactions. It even took her 6 months and 4 tries to get her own name changed back from her old married name.

    My favourite was that she was locked out of her bank computer account - and couldn't get anyone to unlock her account since the "two secretaries who still know how and have permissions to unlock accounts (across all of England) were on holiday". So she couldn't do anything for 4 days until one of them came back.

    The IT systems have been slowly breaking, system by system for a couple of years now.

    Wow.

    1. Anonymous Coward
      Anonymous Coward

      @AC 13:51

      Sorry calling BS on this one. As an ex-RBS employee who was caught up in the off-shoring to India I have a right to bitch about them, but for a lcoked out account a simple call to the help desk and they would have just got her to get a full time employee who could log on to fill out an electronic form and then give a second person's name for confirmation of who we was. These two would get the two halves of her new temp password.

      Longest myself or any of my colleagues were lcoked out was only a couple of hours. Normally it would take just 10-20 minutes

      1. Anonymous Coward
        Anonymous Coward

        Re: @AC 13:51

        For a windows (say, for a domain named after a one of Jupiters moons) account you're right, it's fully automated, back office is another matter, I have known people locked out for weeks because the only remaining admin for that group had left or is on holiday.

      2. Anonymous Coward
        Anonymous Coward

        Re: @AC 13:51

        I second AC 1407 - Original post is complete rubbish, I worked there for seven years...

    2. Anonymous Coward
      Anonymous Coward

      Re: Their backend systems are a joke

      Wow. I'm not sure if it's "2 secretaries for the whole UK". But it may be a local problem. IE, if the manager is on holiday, you cannot get their sign off on a password reset/IT assistance. :D

      I know someone in an insurance company who did similar to get himself 2 weeks off work.

      1. Anonymous Coward
        Anonymous Coward

        Re: Their backend systems are a joke

        Without going into detail, for obvious reasons, there is a process to deal with authorisers being off, ill, holiday, etc. Just because someone on a department doesn't know it, doesn't mean it's not there. There is also the IT helpdesk, if all else fails.

  38. digdilem
    FAIL

    Am I alone in being very suspicious that this is not a security compromise? I don't doubt the it teams are working hard to fix and won't allow credits to go ahead until they are sure the system is safe.

    I had to tell our workforce today, wage day, that if they bank with Natwest (as I do) they won't be getting paid today as usual. Not fun.

    1. Caff

      I would be very suprised if it was a security issue, there are many more ways to mess up a batch and corrupt a database. However most of these are quickly recoverable within a few hours and happen more often than banks would admit. As long as mainframe ops and support do their job noone knows they do it at all.

  39. JimmyPage Silver badge
    FAIL

    So serious they are opening on Sundays

    latest news.

    The problem is (as I have stated before) is they will have to field legions of claims where their snafu has cost people money. There was a story on the BBC of a guy who hadn't recieved his wages, and couldn't afford to pay for a headstone for his stillborn daughter. I wish them the best of luck trying to argue that down in the "court of public opinion"(c)

    (c)Harriert Harman 2009.

    1. Anonymous Coward
      Flame

      The 51st State Or What ?

      Your headstone story very much sounds like a merkin story. Will the girl not move into paradise without the tombstone or what ?? Will the "window to paradise" close in a few days ??

  40. Grivas Bo Diddly Harm
    Coat

    Phew!

    I opened an account with Williams and Glyns in 1981, and they were excellent; after a while it transformed into The Royal Bank of Scotland before truncating itself into RBS, with the emphasis on the last two letters. Even after [Sir Fred's] Mr Goodwin's departure the branch network was still solid and, from a user's point of view, the online services were good.

    Then I heard that RBS England was going to be passed over to Santander and alarm bells rang good 'n' proper - they'd done me over years before when they took over/rebranded Abbey [National].

    I moved (hence the coat icon) over to Co-op - on the basis that they seemed the least worst and owned, at least notionally, by its customers - and completed the transfer of D/Ds, S/Os, funds and notified anyone paying in last month. Just in time, it seems. Now maintaining the accounts with zero balances just to piss off RBS/Santander, though must keep an eye on sneaky account charges being introduced.

    In all my dealings with the phone banking staff over the years I unfailingly received good, polite, friendly service and it saddens me to leave. Good luck guys.

    And I work for an organisation that itself is still worshipping the false god of outsourcing, so my sympathies go out to all those former RBS IT staff who have been dumped.

    PS, Listening to an RBS Spokeswoman, Susan Allen, on the PM programme. Apparently this failure wasn't expected. That''s OK then...

    1. JamieL
      Alert

      Re: Phew!

      So you've moved to Co-Op.. In which case you've got a year or two there then until their banking platform replacement goes live - start looking for another bank soon!

  41. JimmyPage Silver badge
    Flame

    Just a thought ...

    it's all very well gloating over how much RBS/NatWest will have to pay out over this. But we all know where the money is coming from eventually ....

  42. avatar111

    Still at it

    Strangely enough only last week Lloyds Bank signed a 'no regrets' contract with Wipro to replace skilled Open Systems contractors with off-shored staff.

    I guess those contractors might be able to amble over to NatWest and amble over to Lloyds when it goes pear-shaped there.

    I wonder if NatWest has got around to ringing-up the newly-redundant skilled staff and offering them lots of dosh to come into the office this weekend?

  43. Banker
    Happy

    They've outsourced much more than that!

    I worked for RBS during and after the merger with Natwest, I left their Global Financial Markets Department in 2004 after a 5 year stint. They had already moved some IT functions to India at that point and have continued to do so year on year since. The numbers some people are quoting 1600/800 are possibly the more recent figures, the total is way way beyond this.

    The comments on documentation are comical, as if a document is the thing you turn to at a time of crisis.

    The fact is, when you work closely with systems and the business users, you understand not only the quirks of the systems, but the risks and consequences of failure. You work with those users on the work around solutions that will get the banking day complete.

    They haven't just outsourced the IT staff, but the very experienced and valuable back office / operations staff that would work with IT staff to solve the serious issues. I beleive these guys are mostly posted out in Singapore, who probably have never met the IT staff in India. The unseen cost of outsourcing is a compounding loss of shared experience and commitment, which becomes accutely apparent when the sh!t hits the ... cash machines

    The chaps I trained out in India were nice enough, but they simply lacked the knowledge and experience of Finacial Markets trading, trade and settlement processing, Swift messaging blah blah and the risks involved.

    I'll be drinking with a bunch of ex RBS/Natwesties soon enough, where we'll all be saying.....

    "WE TOLD YOU SO!!!!!!!"

    I doubt any senior manager in RBS/NW is going to say the outsourcing was to blame and change anything significant.

    Over the next 10 years, I would expect this sort of failure to happen to vertually all the big UK banks. Hopefully it will become such an inconvenience to the UK public, the government will have to step in and regulate service expectations, a kind of SLA for the public......... then maybe, the management will be forced to value to kind of 'Support Culture' I enjoy to provide. Then maybe.... the jobs will start driffting back to the UK once more.

    1. Keith 17

      Re: They've outsourced much more than that!

      The comments on documentation are comical, as if a document is the thing you turn to at a time of crisis.

      Well said sir, you can tell on this thread that there are some experienced devs.

      Documentation is _the_ last thing I would go near at a time like that. In fact, I'd go as far as to keep _all_ my devs well away from documentation at such a time. Why? Because I have no guarantee of the accuracy of any given document. You can't unit test a document, it wasn't compiled, it likely wasn't reviewed, was it even finished? Was it updated when changes came along? I don't know.

      Innacurate documentation leads to bum steers, it tells you to look in the wrong place, it describes how things work when it's not the way they work at all. Code, on the other hand, is harder to read but tells no lies. Commented code is more valuable in times of troubleshooting than documentation because it's concise and granular.

      I've been a dev for over 25 years and I firmly believe that bad documentation is far worse than no documentation at all.

      If the devs in this case were frantically grasping for the documentation, I'd be very worried indeed.

    2. Anonymous Coward
      Anonymous Coward

      Re: They've outsourced much more than that!

      well this is how we think it used to work.................

      1. Destroy All Monsters Silver badge
        Devil

        In marketese...

        "The maintenance ecosystem has been destroyed"

  44. Banker
    Mushroom

    One more thing......

    ......there's a certain UK bank with Horse. Trust me, their systems are older and more reliant on a bunch of old fogey staff in the UK than RSB/NW ever was. They've recently started to outsource their IT/Back office too.........

  45. Anonymous Coward
    Anonymous Coward

    I hope this doesn't affect PPI payouts.

  46. All names Taken

    Tst! Silly! It's not software

    It is not software related at all at all?

    RBS and its underlings have merely ran out of dosh?

    Coffers are empty so suspend all transactions until some positive numbers (well, if not totally positive at least a deceleration of negative decline numbers) appear?

    1. Destroy All Monsters Silver badge
      Holmes

      Re: Tst! Silly! It's not software

      Can't run out of dosh if there is an infinite supply at the central bank.

      That's why inflation exists.

  47. mr33

    I used to run NatWest IT 20 years ago and I suspect many of the back end systems are similar to then, but with a multitude of front end systems added for online and mobile banking etc. For all the comments on documentation and outsourcing I'd have to say:

    This feels like a major batch scheduling cock up and/or Db corruption to me. Given the length of time it is taking it would seem like they've also screwed up what might have been a straightforward rollback or recovery process. However good your documentation or ITIL processes or scheduling automation it's still possible for a production control issue to happen, but you do need people who understand how it fits together not to make it worse... In my experience that's not the people that are outsourced, at least, I hope they kept a few people who know which batch job needs to go after which...

    So I think this is less about app programming and more about production control and operations...

    In fact outsourcing the ordinary application programming makes sense and it's naive to blame this on outsourcing. It's often the case that the so called 'experts' that have been around for ever are the worst at documentation or training others because, hey, they wouldn't be the 'guru' anymore. I usually get rid of the guru's as they are often the blockers in getting the applications updated and written properly.

    1. Keith 17

      "ordinary application programming"

      that bit right there is where, for me, your credibility went straight down the pan.

      the best devs are those that _understand_ what they're trying to do, not those that are just blindly following a spec. business knowledge is a massive part of it.

      1. mr33

        Well I meant 'ordinary' as opposed to 'a bit special'. There is a difference - for a company with the diversity and scale of NatWest it makes sense to differentiate what it decides to outsource. I might decide to have different level of expertise and experience coding my fraud and risk algorithms than I do doing some simple customer service screens. Outsourcing simple stuff is obvious, outsourcing core critical stuff is riskier. Anyone that says outsource nothing is being as naive as someone that outsources everything.

        1. Grivas Bo Diddly Harm

          @mr33

          ..." 'ordinary' as opposed to 'a bit special' ".

          And there you have a neat encapsulation of the outsourcing problem. It's the 'ordinary', less experienced, programmers and developers of today who, with experience and training and identified talent, become the 'special' programmers, maintainers and developers of tomorrow. By cutting that link you might make immediate savings but you've bolloxed yourself for the future.

          But what do I know, after all I only work for an organisation that has done exactly that and then finds itself over a barrel when it does need people to jump through hoops in double quick time.

    2. Anonymous Coward
      Anonymous Coward

      @mr33

      "It makes sense to outsource", eh?

      Mmmm. Tell that to the FD or TUI Travel plc, who was shown the door when they cocked up their IT and Finance after they outsourced and offshored it all. I very much doubt they saved the £117m they had to write off after over-stating their results.

      Offshore activity at minimum cost is absolutely fine when the "value at risk" is a T-shirt, or a cheap plastic toy. When it is the integrity of a company's core financial systems and data, maybe the relatively modest costs savings need to be placed against the reputational damage, and the costs of rectification and compensation. Look at the Cable & Wireless/IBM spat back in, when was it 2001? C&W thought they'd save money, and in fact they found that they were getting a worse service and paying (IIRC) about £120m a year more than they were before - all went to the high court, and on the day of the case IBM came up with an out of court settlement. My own employers have outsourced IT to HP, and the like for like operating costs have gone up; we've cut the projects budget to keep costs within budget, and as a result we get less for more. Meanwhile the directors think that the outsource has been a huge success because total costs have gone down - they're too far from reality to understand that we pay more for the basics, and now have less being invested in new systems and enhancements than before.

      In terms of "ordinary application development", that's what British Gas did with Accenture for a utility billing system, resulting in a huge project failure, regulatory intervention, and a £180m dispute that went all the way to the high court, and resulted in BG cleaning the floor with Accenture. Was that the sort of "ordinary application development" that you think should be outsourced?

      I'm sure you could come back with many examples that have worked, but the risks undoubtedly are far greater than the clowns signing the deals will recognise, and the savings more modest.

  48. Anonymous Coward
    Anonymous Coward

    Worrying

    I'm a security consultant who works for a rival but banks with Natwest. We all know that IT systems fail from time to time and sometimes these make the news but the duration of this is extremely worrying to me.

    I've been impressed with the Natwest mobile app and thought that they must have some top notch security if they were offering 3rd party payments and cardless cash withdrawals. So I decided to take a look and it took me 20 minutes to reverse engineer it and patch so that it captured my passcode as I entered it ! I was shocked that it was so sloppy and had no discernable security measures at all, this can't have been pen tested by anybody remotely competent. This is bordering on the criminally negligent, so I've sent my findings to the FSA and needless to say I'll be moving all of my accounts on Monday.

    Obviously they have nobody skilled in risk and security left either.

    1. Anonymous Coward
      Anonymous Coward

      Re: Worrying

      I can take any Android app, and patch it so that it captures what I entered. It doesn't mean I can take those changes, magically sign them with the original private key and magically distribute that to users.

      How is that a security hole?

      1. Anonymous Coward
        Anonymous Coward

        Re: Worrying

        OK, let say this patch captured the GUID, Passcode, TWK, IssuerName (terms meaningless to you or I but the people that wrote the app would recognise them as everything that is required for logon). Now let's say it then transmitted these details to a criminal's phone and now the criminal has complete takeover of your account including the facility to take cash out without a debit card. To easily install my patch I need to be on a rooted (Android) or jailbroken (iOS) phone so the first line of defence would be to ensure that the app doesn't install or run on either of those. No such detection, so I can install my 'malware' by hiding it inside a cydia or android app or even with a well-crafted drive-by download via a well-know web exploit.

        There are then extra levels of defence that I would expect any competent banking app to deploy but this has none - sheer negligence.

        1. Anonymous Coward
          Anonymous Coward

          Re: Worrying

          From the state of many Android devices and apps, I wouldn't be surprised to see half this info leaked in the log file, even on an unrooted device.

          Out of interest, which competent banking (or other) apps do you know which employ such defences?

  49. Anonymous Coward
    Anonymous Coward

    I wonder who ended up taking this job:

    http://hyderabad.quikr.com/Batch-Admin-with-CA7-tool-W0QQAdIdZ81783921

    1. Anonymous Coward
      Anonymous Coward

      Requires some knowledge or less.

      Should've seen it coming: even their job advert comes with an off-by-one error...

      "Experience: 5 - 8 Yrs

      Looking for candidates having 4-7 years of experience"

  50. trafalgar
    Thumb Up

    Ha - Ha (Nelson style)

    Natwest charged my dad £1000s in overdraft fees.

    This was from being £15 overdrawn.

    DIE Natwest DIE.

  51. Dr Wheetos
    FAIL

    Disaster recovery anyone?

    I wonder if they were doing a DR test which went a bit awry? I wouldn't be surprised if this carried on into next week. After all this time it doesn't seem like a network issue.

  52. Jon182
    FAIL

    So pleased I left them. Now with First Direct i.e. HSBC. Would not touch any of the nationalised banks with a barge pole. This on a Blackberry scale in terms of screw ups. Used to support payment processing systems at a mobile operator but now look after mobile data. No glacial batch processes to worry about.

  53. Anonymous Coward
    Anonymous Coward

    Due Process or Fraud

    This is either down to a process similar to what is detailed here:

    http://thedailywtf.com/Articles/Slaves-to-The-Process.aspx

    Causing a massive tailspin when it collided with the next process and the dominoes ended up in a heap on the floor.

    Or

    Mahoooooosive fraud attempt in the BACS or CHAPS process, which is why fast payments are working but all else isn't, meaning that all BACS CHAPS Batches are having to be manually checked with real eyeballs in triplicate before being allowed to pass into reality.

    FirstDirect is the home of my money thank (insert deity of your choice), but my employer uses GnatWest so I fear for my salary arriving there this month.

  54. AlexS
    Holmes

    Have they tried switching it off and on?

  55. zoodle

    Unhappy?

    Unhappy RBS/NatWest customer? Vote. Feet. Now! Plenty of competition. Many banks now iron-clad by national governments.

    Unhappy creditor. Meet and commiserate with fellow creditors, then serve class action suit.

    Unhappy debtor? Wheee! RBS/NatWest will cover all your accidental failed transactions and penalty charges. Go spend.

    Unhappy RBS/NatWest staff? Whinge about rejected DR plans. Find new job. Now.

  56. Anonymous Coward
    Anonymous Coward

    Classic!

    I bank with NatWest and went to look at the update on their website (http://www.natwest.com/personal.ashx). My eye was caught by this comment from another customer:

    "You bunch of s**theads , I want my wages now , I'm fed up waiting . Need my money. And no I didn't choose to be paid by natwest sue from south of England

    by Pi**edrightoff from Kent on June 22, 2012 at 9:59 pm "

    I have the screenshot :-)

    1. Spoonsinger
      Happy

      Re: Classic!

      Nice, but following looked more proactive....

      "I'm going to take a poo on your doorstep you inbred mugs. If I don't get my money in my account within 24 hours I'm going to chain myself to your qc naked and play with my todger. I think your employes should work naked for compensation #SayNoMore

      by I hate natwest from Hmp Bristol on June 22, 2012 at 11:12 pm 0 comments "

      (also have screenshot :-)

  57. brazillian

    rbs IT problems. cyber attack

    Anybody think this might be a cyber attack? Iran is bound to retaliate.

  58. Anonymous Coward
    Anonymous Coward

    I thought HSBC systems were bad,,,

    You know I honestly thought they were bad until I read about this. I will never rip their systems again.

    Surely RBS/Natwest have the snapshot balance to fall back on, from there they can process the BACS payments and incoming payments using a secondary IO system or mainframe? I should point out I do not know RBS/Natwest systems but only HSBC. I assume they would roughly be the same but with different processes and hardware? Please tell me they have a auxilliary standby processing system.

    My assumption could be wrong though. It seems to be taking an awfully long time to correct, if they have outsourced and offshored then I would be a little worried. The HSBC bods ususally run DR preps and "what if" scenarios, primarily "what if everything went down the swanny". Guess RBS/Natwest don't and/or maybe have not done any dry/wet runs of their DR procedure.

  59. Anonymous Coward
    Anonymous Coward

    This didn't start last Monday. It started at the end of last week IMO

    Been having some very terse dealings with RBS since 14 June regarding something inexplicable that happened to one of my accounts. I haven't been able to fathom it, and RBS cannot explain it.

    I have a view of my account online that the customer service advisers cannot see. I can see transactions that they can't but they see the same balance as me. And because they cannot see all the transactions they cannot answer any queries about the erroneous transactions that I think are on there.

    When this blew up this week I realised the problems I started seeing late last week are most likely part of it. In years nothing like this has happened before. I ended up closing an account last night because a customer service person became irate when I pushed them on the causes of the problems. They don't know naff all to be fair to them. Looks like a semi-meltdown. Textbook IT disaster case study.

  60. Anonymous Coward
    Anonymous Coward

    Coutts ?

    My Business Banks with Coutts.

    Coutts did a massive software upgrade at the end of March. It was a shambles. We (all) lost access for several days, huge delays in seeing incoming payments, inability to make outgoing payments, unreliable balances (and if you logged on several times, you would see a different balance each time), numerous other "bugs" and glitches. In fact many of the problems I have seen reported in the current RBS / Natwest debacle sound awfully familiar. As far as I can tell the Coutts staff had little or no practical ability to do anything useful either.

    It did improve but was a shambles all over again at the end of April.

    The end of May was more normal.

    Coutts does basically seem to be "up", but outgoing payments seem to be going through considerably more slowly right now.

    We're now bricking-it for the end of June.

    1. Anonymous Coward
      Anonymous Coward

      Re: Coutts ?

      If that's true, I'm sadly not surprised. I have no idea who they were, or how many of them there were, but they do not learn from mistakes. :(

  61. John Munyard

    Get the facts guys

    Sorry to sound like a party pooper but before you link the poor fix time for this problem to the redundancies in 2010, have you established whether any of the job losses were in the division which runs the retail banking systems?

    1. TomTom

      Re: Get the facts guys

      Yes. It has been established. It has been posted here and elsewhere that the problem was with the CA-7 scheduling software. This is one of the roles that was transferred to India. Look at this link (as posted above) for a job advert for a CA-7 technician in Hyderabad.

      http://hyderabad.quikr.com/Batch-Admin-with-CA7-tool-W0QQAdIdZ81783921

    2. Anonymous Coward
      Anonymous Coward

      Re: Get the facts guys

      Yes. A very large number of people were made redundant from retail banking.

  62. Charles Smith

    How to comment bank program code

    Hey Boss how can I help it?

    Drop */

    is almost the same as:

    Drop *

  63. Shoddy Bob

    I've got a good idea, why don't we get a crack team in from India to fix it ? Every Indian SI company I speak to claims to have a team that can build or fix anything - even things they've never seen before or have any skills in.

    Why are we worried ?

  64. bod43

    Hardware failure?

    I worked in banking IT (networking) a while back and one thing that surprises me is that this problem surfaced midweek. In my day no changes were made except at weekends to allow plenty time for testing/ backout.

    This suggests to me something like a hardware failure rather than an update induced problem. Inexperienced and far away staff then perhaps screwed up the recovery causing further damage.

    1. Nick 6
      Devil

      Re: Hardware failure?

      I've worked at one retail bank in this country who does maintenance during a weekday afternoon-evening slot.

  65. System Administrator
    FAIL

    NatWest bank adverts ....

    Cue dainty enfeebled citizen, shivering outside bank, lamenting:-

    "My bank's turned into a rather expensive random number generator"

    So glad I left them - and glad to find out I'm not the only one who's only keeping an account open with them to 'annoy' them - every little helps LOL

  66. Anonymous Coward
    Anonymous Coward

    Ulster Bank

    This is also affecting Ulster Bank here in Ireland - they are part of the RBS Group.

    UB are saying it will be fixed by monday, I am suspicious as the UK parent have not, to my knowledge, said anything of the sort.

  67. Anonymous Coward
    Anonymous Coward

    Ok, so I have decided to vote with my feet and move personal and business accounts from Natwest ....leaving a fiver in each one just to annoy them ;) ...but to where? Has anyone worked anywhere they actually recommend?!

    1. JeffinLondon

      HSBC UK has modern core systems. Totally rewritten about 10 years ago by a very solid UK / US design team they really did a very good job. (first hand knowledge)

      1. Anonymous Coward
        Anonymous Coward

        HSBC

        HSBC also has a general and well refined disdain for retail customers. From first hand knowledge, you will be treated like the rounding error you are.

  68. Ian Johnston Silver badge
    WTF?

    According to the BBC News website

    "The problem began on Thursday.. It is understood to have arisen after staff tried to install a software update on RBS's payment processing system, but ended up corrupting it."

    Erm ... system? Payment processing system ... singular? Shouldn't anything like that exist in duplicate or triplicate, with off-site disaster recovery systems on standby?

    1. Nick 6
      Windows

      Single logical system

      Typically there is redundant hardware, sometime OS and other system software too. But above that there is a single logical version of the application and data. You can have as much tin as you, but as many copies of the data as you liked are all the same. You screw up one, you screw up them all. Payment comes in from BACS or FP into the payments engine and is replicated at disk or app level.

      Redundant systems developed and deployed in isolation to a common set of requirements is an extremely expensive option only available to mission critical systems - which excludes our retail banks apparently.

  69. Melanie Winiger

    Got the solution

    CTL-Z

    Works for me everytime.

    What? That's not how a mainframe works?

    .-)

    I am an old mainframer and I'm caught between thinking "there for the grace of God go I" and "you got what you deserve you bunch of non-technical beancounters for outsourcing something you had no clue about".

  70. Nash_Equilibrium
    IT Angle

    I have not used CA-7 or anything like it, god I would hate to be left alone on a mainframe.

    But after a bot of googling, I assume you don't want people like this either http://ibmmainframeforum.com/viewforum.php?f=38

    "cann any one please explain the commands like PF(page forward),PB (page backward) and SR(save Replace)

    Also TSO CA7cLIST"

  71. Anonymous Coward
    Anonymous Coward

    CA 7 Batch Scheduler job in Hyderabad

    I" wonder who ended up taking this job:

    http://hyderabad.quikr.com/Batch-Admin-with-CA7-tool-W0QQAdIdZ81783921"

    Followed the link for the job in India (just out of interest you understand) - it's based in Amberpet.

    What has Wikipedia (yes I know) got to say about Amberpet ?

    " It is the biggest slum of hyderabad and lack development in many areas. Amberpet was never the part of hyderabad and comes under Ranga Reddy District. Amberpet was largely famous for the spread-over open markets sprouted all over the area catering to the household needs of people from all the nearby places at very affordable prices. Now, these have been replaced by many supermarkets and other allied dairy & poultry establishments."

    Nice!

    Bet they get the best candidates there!

    Risk Management - we've heard if it - RBS senior management, they really are worth their bonuses eh!

    AC

    1. Anonymous Coward
      Anonymous Coward

      Re: CA 7 Batch Scheduler job in Hyderabad

      "Annual Compensation 8L-10L"... that's max 1 million rupees or £11200pa.

      As they say about peanuts and monkeys...

      1. Anonymous Coward
        Anonymous Coward

        Re: Re: CA 7 Batch Scheduler job in Hyderabad

        Perhaps you are not taking into account the local cost of living?

  72. Anonymous Coward
    Anonymous Coward

    in my experience...

    ...all UK banking institutions are similarly exposed to this kind of problem emerging. I have not worked for any organisation that does not - at any given point in time - have at least one piece of critical infrastructure that is unstable or have an identified single point of failure with no hard mitigation plan.

    Sadly the impact of legacy and resource constraints mean that we operate in an imperfect environment with incomplete information. The skills we have to mitigate against and handle this kind of event are the reason we can invoice for hundreds of pounds per day without blushing - that's our value.

    The core payments and ledger systems at RBS are designed to be resilient and recoverable to the N-1th degree. I have no doubt at all that, to the level observable by the customer, normality will be restored - the speed at which it is restored dependent on the nature of the problem. At the levels deeper than that observable by the customer there could be a number of infrastructure, development and reconciliation projects required to fix the original problem, prevent it happening again and tidy up behind any quick-and-dirty bodges that proved necessary to get services restored.

    I'm looking forward to hearing the final accounting of this incident, coming at a time when RBS is performing poorly in the market and introducing significant operational risk through it's Group Technology strategy (in the opinion of many who occupy the cubicles), a salutary lesson could be on offer

  73. JeffinLondon

    Short sighted management

    The facts are RBS's computer systems are ancient in computing terms. Many dating back to the 1960's and using obsolete technologies.

    RBS has also been directed to cut costs sharply, and in banking that means staff cuts and particularly expensive, older IT staff. So many / most of RBS's experienced staff have left recently leaving inexperienced on-shore youngsters and off-shore outsourcing companies to fill the breach.

    So combining the two causes, ancient software and very few experienced staff left to look after it was a ticking time bomb that finally went off this past week.

    A pity so many customers were impacted by the result of decades of poor management by the RBS Group.

    1. Anonymous Coward
      Anonymous Coward

      Re: Short sighted management

      What do you mean by 'obsolete technologies' ? Are you referring to mainframe related technologies ?

  74. Darren Barratt
    Happy

    Apparently we're all redundant

    Well if the NatWest hire me (scheduling and operations expert, due to be redundant at the end of this week), this'll never happen again.

    Call me!

  75. Frustrated sofftware dev/test

    Not just developers ...

    ... but also a testing issue.

    Development can screw up, but surely the "update" had been tested prior to putting on the live system.

    Regardless of how old the system is testing should have taken place (and arguably more testing for an eldery system) within both development and in a system test environment.

    Oh must not forget the project/programme/release managers who say "There is no need to test that its not changed."

    Those involved in testing the update should not rely only on what the delevopers say the have changed, but to look at what is actually being changed and identifying potential areas where problems may arise.

  76. Anonymous Coward
    Anonymous Coward

    Having worked with RBS group in the past, my experience was that the change management process was extremely bureaucratic and it was very difficult to alter anything. I was told that Fred's right hand man (who managed the Natwest/RBS IT merger) was partly responsible for this, because he had a very "robust" management approach, demanded 100% server availability (!) and left everyone terrified of making changes.

    I was responsible for implementing a minor change to production, and it all had to be tested, documented and supported by backout plans. On the night, I had support chasing me by phone to ensure I was sticking to the implementation schedule, and imposing a fixed deadline for successful implementation or full roll-back.

    Because the same process was applied to all changes (regardless of the scale and impact of the change) there was a suggestion that it was crippling the ability of the business to make timely changes (e.g. even changing some copy on the website could involve a full change board review and take weeks).

    Looks like things have loosened up a bit now though...

  77. Matt Finish
    Facepalm

    Wot? no rollback?

    We have to spell out the rollback procedure for a proposed system change in order for the change to be authorised. Even our plumber left our old ball valve in his van, so when the new one he'd fitted started whistling like a steam train he could put the old one back until he'd found a better replacement.

    Basic stuff.

  78. Ant Evans
    Paris Hilton

    Cloud

    Maybe RBS should move their back office to the Cloud..?

  79. A Leach

    interesting background - thanks

  80. Duffaboy

    Saving Money.

    To save a bit of cash they switched the Eletric off.

  81. Andrew James

    I personally have a huge amount of respect for RBS. They've sorted themselves out just in time for the completion of my house purchase today, so all the hassle that goes with delaying the removals firm, deliveries to the new address arranged tomorrow, wednesday and thursday etc is eliminated just by them sorting the problem over the weekend.

    Hooray. Just need to call at the shop on the way home and get beers for the fridge.

  82. Anonymous Coward
    Anonymous Coward

    You didn't need a Crystal Ball to predict this....

    This was an accident waiting to happen. The IT outsourcing strategy in RBS is corporate suicide - An incident of this nature has been predicted by those working in IT for a very long time - RBS have dodged several bullets recently, their luck has now run out.

    As always the management team are running around trying to sweep the outsourcing issue under the carpet - Mr Hester a word of advice challenge & challenge again your IT Managers for the real incident review not a fabricated one that covers up their fatally flawed outsourcing approach.

    Also speak directly to the administrators to get a real understanding of the outsourcing issues & the risk this puts on your business. Sadly this will likely be the first of many incidents on this scale - RBS get your head out of the sand or watch this space for further business impacting carnage caused by inadequate outsourced staff.

  83. Anonymous Coward
    Anonymous Coward

    Processing backlog catch-up time

    At a bank I once worked for we did the maths for a major system outage. Suppose your systems are near to fully committed (more likely with global 24 hour banks than, say, a UK-only bank with bulk load maybe 8:00 to early evening for real-time processes then up to midnight for less time-critical stuff like statement production). You might have something like 20 hours a day of heavy system usage with 4 hours "slack", so if you lose a days processing the catch up time to process the backlog is 5 days. In our case we calculated that more than 2 days total systems loss would mean we'd not be a bank any more.

    Outsource and you lose the guys with a 20 year career on the same system, knew it inside out and could home in on a problem in minutes - at the same time the bean counters are looking at the spare mainframe capacity and regarding it as "a waste" so trim down to insufficient spare capacity to withstand a disaster and too little expertise to know how to fix it quick. The outsourcing provider has its own bean counters and regard testing as a bit of a luxury doing as little as they can so the "tested" box gets ticked - their worst case scenario is they might lose a customer, the bank's worst case scenario is total melt-down. The triple whammy is when the major problem falls at month end when transactions peak with monthly salaries and consequent bill payments (and you'd not wish a major failure on your worst enemy when it's year end too).

    The IT guys sound all the warning signals but the bean counters win and when everything goes tits up it's the IT guys fault.

This topic is closed for new posts.

Other stories you might like