back to article RBS must realise it's just an IT biz with a banking licence

Banks need to start thinking of themselves as IT companies, said David Chan of City University London. "A senior banking technologist has said to me: 'A retail bank is nothing but an IT company with a banking licence'," Chan told The Reg. "While this may seem extreme, when one looks at the economics of any retail bank, it is …


This topic is closed for new posts.
  1. Volvic

    "IT is the business..."

    "...and the business is IT", as so many ITIL trainers are fond of saying

    1. BillG

      Re: "IT is the business..."

      A retail bank is nothing but an IT company with a banking licence

      That truly is a brilliant and insightful statement.

      The RBS problem is ultimately symptomatic of a culture within many large organisation where the senior team believe that they can rule by "dictat" without having to understand the consequences of their decisions.

      ...out-sourcing can exacerbate bad management...

      Or, as in the motto of Dilbert's pointy-haired boss, bad management, feels that anything they don't understand must be easy.

  2. This post has been deleted by its author

  3. Anonymous Coward
    Anonymous Coward

    > trend towards pigeonholing folk

    > And it's not just a culture issue: there's a trend towards pigeonholing folk into particular roles

    > within a process, ensuring that they do one or two things well and need not worry about the

    > rest of the machine.

    Ain't that the truth... I was recently told by a line manager "we don't need people like you who understand everything any more". Well, next time the proverbial hits the fan I almost certainly won't be there (hence icon - would you given a hint like that) and I wonder how well they'll get on...

    1. Anonymous Coward
      Anonymous Coward

      Re: > trend towards pigeonholing folk

      Indeed, many of our organisations are ruled by ignorant fascists.

      1. Anonymous Coward
        Anonymous Coward

        Re: > trend towards pigeonholing folk

        > Indeed, many of our organisations are ruled by ignorant fascists.

        There's at least one ignorant fascist reading your msg ...

    2. Yet Another Anonymous coward Silver badge

      Re: > trend towards pigeonholing folk

      It's not just pigeonholing people but companies

      Even when the job isn't outsourced there are so many different divisions/subsidiaries/competing business units/etc all of who are at each others throat then if some other dept asks you to do something that will destroy the company it's better for you to do it - and meet your SLA - than say no that's wrong.

      It may even be a legal requirement to "your" shareholders to destroy the bank rather than fail to perform your contract tasks !

    3. Anonymous Coward
      Anonymous Coward

      Re: > trend towards pigeonholing folk

      OP: I had exactly the same thing occur to me. The system is very stable and we're not sure we have anything for you to do any more - i.e. they wanted to flick me off and use one of the grads that is smart but can't even code in the languages used (he's not a developer) to cover the system. The handover took less than 1hr - they weren't interested as management felt it must be easy because I did it so easily although the grad didn't appear too at ease with this.

      Fast forward and they now realise they can't make any changes to the system because they don't understand how it works - it was RAD and they chose feature implementation over documentation or even TDD (takes too long) - and nobody in their team is a proper developer (a few can VBA). Fortunately it still works and hasn't needed fixing. Never ceases to amaze me how management see developers as just commoditised meat-sacks until the fan gets spattered. They now have zero chance of passing an audit on the grounds that nobody understands, can enhance, can support, or can fix the system. It needn't have been that way.

      We need a muppet icon or, given the copyright, a puppet with a hand up its arse.

    4. GitMeMyShootinIrons
      Thumb Up

      Re: > trend towards pigeonholing folk

      "we don't need people like you who understand everything any more"

      And yet with systems becoming more and more integrated, with more dependancies etc, these middle management drones are heading for a fall, as happened at RBS. The blinkered corporate mentality, complete with little empire builders and the 'cover my @rse' attitude never ceases to amaze.

  4. Syed

    Question "So. How can they stop that meltdown from happening again?"

    Answer: Pay the CEO more.

    1. amanfromMars 1 Silver badge

      Re: Question "So. How can they stop that meltdown from happening again?"

      "Question "So. How can they stop that meltdown from happening again?" ..... Answer: Pay the CEO more. .... Syed Posted Wednesday 1st August 2012 10:43 GMT

      That is a very bad joke, Syed, whenever nothing can stop it happening again if IT wants such meltdowns happening again. In fact, you surely will have admit that it is IT facilitating such meltdowns in the banking system, which is led to server profit [money for nothing and in bankings case, also money from nothing] to the very top, with instruction from the very top, or are we to be led to believe that sat the very top of the banking industry, are puppets following orders rather than issuing them. I don't think so, bubba, because that would extraordinarily render them totally unnecessary.

      But then such meltdowns are quite logically how it should be, whenever a system is so perversely corrupted to server itself invented riches which it then loads onto others as if it were their debts, with additional punitive and arbitrarily decided interest charges and sundry miscellaneous administration costs to ensure no chance of repayment and freedom from its deceptively easy and conveniently efficient slavery model.

      It is bound to quite naturally fail catastrophically quickly whenever those pennies drop and whole populations start to realise that they are being ruthlessly conned and treated as ignorant fools to be treated as useful tools.

      The most dangerous jobs in the world today, are being a banker and a politician, in full support of the existing status quo position which seeks to maintain the slavery to invented mounting debt models, rather than get rid of it altogether.

  5. pc

    Ulster Bank issues are still ongoing

    1 week? 3 weeks? 6 weeks so far, and still awaiting reconciliation of some transactions.

    Thanks RBS..

  6. Pete 2 Silver badge

    I blame "consolidation"

    It's the easiest way to cut costs - i.e. staff numbers.

    Instead of having 20,000 servers, each one doing it's own specialised task in it's own customised environment, why not axe a ton of the hardware and run all the mess on one single box o' tricks - maybe even virtualise it all, for added points in the buzzword bingo stakes. Even better, spin the consolidation as being "green" by persuading the gullible and terminally trendy types that we're really saving all those gigawatt*hours because we care about the planet - not for all the money it saves.

    However, it does mean that you can end up with 20,000 servers all being dependent on a single DNS box, or that one honkin' great hub is responsible for *all* your enterprise's core traffic. Even if you've built a resilient or redundant system who's ever had the balls to press the big red button to see if it does actually fail-over?

    So instead of an insignificant bugette or hardware failure just knocking a small part of your biz offline until the on-call engineer puts down his/her sandwich and shuffles over to press "reset" a whole long line of tits go up and suddenly all the lights go dark - and a funny smell seeps through the IT centre.

    So what sounded like a good idea to the accountants who run the corporation, turns into a tangle of interdependencies and unknown unknowns that makes the Butterfly Effect look like a piece of string connected directly to a lever (with a sign saying Do not pull on it). It's not surprising that these systems fail. It is surprising that anyone ever manages to get them running again - though maybe the next major crash won't leave us surprised, at all.

    1. John 62

      Re: I blame "consolidation"

      Since Galvanize by the Chemical Brothers was used in the London Olympics opening ceremony, I have had the song in my head quite a bit. Just now I had Q-Tip/Abstract in my head saying 'Virtualize! Come one, come on! [Don't hold back]"

  7. sugerbear

    Good article

    Mr Chan, has in my opinion, hit the proverbial nail on the head.

    The middle layer of management never seems to get mentioned, it isn't Mr Hester that is choosing to take risks by de-skilling the workforce but it will be his targets that are causing the problem. Required reduction in IT costs = Management finding an easy fix = ship the development off to a cheaper country whilst retaining their own jobs/salary/bonus.

    It is the middle management layer that has failed. They chose not to listen, they ploughed ahead with changes and it has cost the company dearly. Given the culture at RBS, no one questions their boss above project manager level. If you did then the end of year rating system was used as a big stick.

    I have said it before but the outcome of any review will be "more control". It wont point the finger of blame at any management strategy, it will be just an extra layer of change control that slows the development process down even further.

    1. Anonymous Coward
      Anonymous Coward

      Re: Good article

      The tech staff in Edinburgh are competent and know what needs doing (at least, they were a few years back) but the middle managers are handcuffed by terror of the VPs. Any question or criticism of those above you is a serious CLM.

      Many of the technical staff are contractors who have little vested interest in the job beyond this weeks pay cheque. (Admittedly, a very generous one.) I asked a colleague why so many critical systems ran on failure-prone, support-intensive MS Windows systems. He replied with a shrug "We get paid by the hour."

    2. Fatman

      Re: Good article.. Mr Chan, has in my opinion, hit the proverbial nail on the head.


      But, not only did he `hit the nail on its head`; he used a 10kg sledgehammer to accomplish it.

      I suggest that punishment of the C level ranks at RBS start with the CEO, and he should have his balls whacked with said sledgehammer. Then follow down the line as required.

  8. mark 63 Silver badge

    what about the money

    I'd say primarily a bank's business is making my money grow, and not losing any of it.

    The I.T part , which is secondary , is

    Making some of it accessible instantly in as many ways as possible e.g

    theirs and other's cash machines

    web site

    direct debit

    maybe a smartphone app

    cant think of anything alse i require from a bank, even though they keep trying to sell me breakdown and travel insurance for £7 a month.

    1. Daren Nestor

      Re: what about the money

      While what you said kind of makes sense, you're wrong in your judgement of a bank's business. "Making your money grow" is a secondary objective (as is building trust, making massive profits, and having good IT systems!)

      A (retail) bank's business is Risk Management. This should be beaten into everyone working at the bank. Everything they do is about risk management. Too many banks have lost sight of this, and no longer assess risk correctly. This either inflates or deflates their view of the risk of any product or proceudre, leading to either laissez-faire or being overly restrictive.

      1. Anonymous Coward
        Anonymous Coward

        Re: "A (retail) bank's business is Risk Management."

        Rubbish (assuming you mean something like Investment Risk Management, which is the usual interpretation in this context).

        1) A retail bank might want to understand its risk exposure in the "what if our IT fails" sense.

        2) A retail bank's business should be processing its customers transactions reliably, efficiently and cost effectively.

        3) A retail bank has no reason to put any of its customers money at risk.

        RBS management failed on all three.

        1. Rob 103

          Re: "A (retail) bank's business is Risk Management."

          It sounds like you are deliberately missing his point. The core function of a retail bank is to take customers deposits and invest them to make more money (traditionally by lending this money long term to individuals/businesses).

          If all these investments went up the swanny, then the bank's customers would lose their deposits.

          Hence, the risk of the investments is of utmost, critical importance.

          Hence "a banks business is risk management".

          1. Dave Bell

            Re: "A (retail) bank's business is Risk Management."

            Keeping track of the money is a core function. It doesn't matter what else you do if you can't do that.

            These days, we're dealing with computers, it's not clerks with ledgers and hand-written cheques. Yes, when you lend that money out, the bank is taking a risk. Doing the banking right depends on managing those risks. But without reliable working IT, the business is dead in the water. All that's left is the petty cash drawer.

        2. Alex Rose

          Re: "A (retail) bank's business is Risk Management."

          "3) A retail bank has no reason to put any of its customers money at risk."

          Rubbish. A retail bank takes in customer deposits at one rate and lends them out at a higher rate, the difference being their profit. Even if we ignore the fact that they "create" money by lending out more than they have on deposit there is always going to be a risk involved in lending out.

          1. Anonymous Coward
            Anonymous Coward

            Re: "A (retail) bank's business is Risk Management."

            "A retail bank takes in customer deposits at one rate and lends them out at a higher rate, the difference being their profit. Even if we ignore the fact that they "create" money by lending out more than they have on deposit there is always going to be a risk involved in lending out."

            Why so? Let the investors (shareholders) carry the serious risks. Maybe even the senior staff. After all, they are the ones who expect the serious rewards. Just ask any Barclays shareholder how much their shares have grown (prior to Diamond joining anyway).

            Or, alternatively, withdraw the Government protection on (relatively) small deposits (large deposits aren't protected). Then see which banks start running scared because they know the public will found out that they are risking too much depositor money.

            1. AndrueC Silver badge
              Thumb Down

              Re: "A (retail) bank's business is Risk Management."

              Sadly the only time the public will realise the risks is when it all goes titsup and they discover the bank can no longer give them any money. Although everyone ought to know how banks work it's a bit harsh to teach Mr. Average Joe about it by depriving him of the ability to buy food or clothes for his family and forcing him out onto the street.

            2. Daren Nestor

              Re: "A (retail) bank's business is Risk Management."

              Forget about YOU when it comes to the bank because you, personally, are one depositor. A depositor is a class of customer. You are a liability on their balance sheet.

              The Bank, itself, needs to make money and it makes it by using your money as security against loans from the money markets, and then lending that out at a higher rate of interest than they paid. Simples. The charges, etc, against your account, barely scratch the surface of the kind of money they need to make the kind of money they want. Mostly they are to offset the costs of running the branch. The more you have on deposit, the more they can borrow, meaning the more they can lend which makes them more money.

              But lending is risky. So their job is risk management with a side order of risk assessment. Everything else is trimmings. So when I say a bank's job is risk management, I mean that they have to manage the risk of their customers not paying back. By which I mean YOU not paying back, meaning they cannot honour their debts, and the bank that lent to them cannot honour THEIR debts, and all the way up. See?

        3. despairing citizen
          Big Brother

          Re: "A (retail) bank's business is Risk Management."

          I would love to have a look at the Risk Logs and Register for the Downsize, outsource and offshore projects that RBS ran!,

          I would also like to see who signed off on the corporate risk exposure of halve your change team, and run the risk of creating £300m+ in liability dents in your balance sheet.

          Although I doubt the Fundementally Supine Authority has demanded these as part of a review of RBS/Natwest's suitability to hold a banking liscense.

      2. umacf24

        Re: what about the money

        I'm with Daren. The payment system that the banks co-operate in running is fantastically important to the nation, but not to each individual bank. Banks borrow short term and lend long -- that's how they make money and it's what defines a bank -- and they care most about managing that risk to ensure that they are able to meet their obligations over the counter every day. And that's not primarily about payments, whether it's phone apps or CHAPS. We think that's what banks do, but it's not what they think they do.

        Nonetheless, the payment system is vitally important to the rest of us, and RBS have exposed the risks of leaving it, unsupervised, in the hands of people for whom it's always going to be a secondary interest. I think there's scope for regulation -- a technical assessment distinct from the banking regulator -- to ensure that every participant in the payments system, merchants, acceptors and banks, works in an efficient and recoverable way.

    2. A Leach

      Re: what about the money


      I guess I take processing transactions and moving money as the primary function of the bank. Though accept there's a lot of off-the-systems work involved in deciding what to move where.

      All the websites, apps, direct debits have to funnel the transactions thru the core IT, the mainframe.

      1. This post has been deleted by its author

    3. AndrueC Silver badge

      Re: what about the money

      > making my money grow

      But it isn't your money. You gave them your money and it became their's. This is the side of banking that gets glossed over too often. The only difference between giving me your money and giving the bank money is that I don't have a license and you expect the bank to always be able to give you some of their money whenever you ask for it at a future date.

      Banks don't 'grow your money'. They just agree to credit your account with additional funds every now and again so that - you hope and expect - over time you they will give you more of their money than you have given them.

    4. sugerbear

      Re: what about the money

      [quote]I'd say primarily a bank's business is making my money grow, and not losing any of it.[/quote]

      The primary role of any business is to generate a profit. Risk, IT and so on all flow from that primary goal.

      If you are shareholder, then I agree. Growing the share price and dividend are the primary objective of a publicly listed company. Growing turnover/market share count for nothing and are just vanity unless they can be turned into solid income (and profit).

      As a customer the primary goal of any bank is to minimise the amount they spend on you, whilst making you generate income. They will be doing everything they possibly can to ensure your money grows as slowly as possible, because the margin between what they pay you as a saver and what they lend out is where their income is generated (actually probably a lot less reliant on you as a saver as they have the money market to borrow from so even less incentive to pay you anywhere near a competitive rate of interest).

      On a side note, co-op premier account charge of £13 a month is pretty good I think for mobile phone insurance, worldwide family travel insurance, European RAC breakdown cover, £300 interest free overdraft.

    5. tony2heads

      Re: what about the money

      'making my money grow'

      FAT CHANCE - with the low interest rates now available. Even given the low inflation

  9. umacf24

    Change Management

    CM is a communications tool. It works if it puts the change plans in front of people who are technically and institutionally able to challenge them. It works better if those people include the ones who will have to fix what goes wrong. What can't work is a lovely paper trail with all the manager's signatures.

    1. Anonymous Coward
      Anonymous Coward

      Re: Change Management

      The phrase you're looking for is 'Blame Management'. Not only does it allow peons to (at least attempt) to defend their actions, it means that you can follow the paper trail along to the end, fire the guy who signed off all the other signatures and fire him.

      This way, nobody else is to blame, the management can be seen to be Doing Something and no business processes need fixing. Its an amazing technique.

    2. FutureShock999

      Re: Change Management

      Nail - meet hammer head. You've hit it exactly - the best written Change Management policy document does NOTHIING unless it is interpreted by someone who (or a team of whos) that can holistically assess it's true impact and risk, in obvious and non-obvious ways. Which as the article points out, is precisely what is missing when you offshore so much of your expertise and demand a uniform process populated by specialists and drone workers. Once you have the latter condition, no amount of Change Management policy makes any difference - there are too few people left that can actually assess the change for non-obvious impact. RBS is not alone in the banking world in ending up in this positoin - in fact, they are in very close company with many other banks these days...

      1. Yet Another Anonymous coward Silver badge

        Re: Change Management

        >Nail - meet hammer head

        Unfortunately it's more like "that's a thumb"

        "Yes but we are required to hit something 99.99% of the time"

        "But it's a thumb"

        Well we could escalate a "not hit a thumb" request to the new 6levels of change management but in the meantime keep hitting the thumb - we don't want to trigger a failed to hit something incident.

        1. Anonymous Coward
          Anonymous Coward

          Re: Change Management

          Change management is great. In RBS its an arse covering exercise. So you end up with folks processing meaningless changes and not spending time doing the diligence on the changes that are liable to fuck you over.

        2. Anonymous Coward
          Anonymous Coward

          Re: Change Management

          Oh my, I've never seen someone actually get the phrase so close to how the management said it... I guess you hit the nail on the head of that one.

          The amount of times I tried to say "if we carry on as normal, we will carry on breaking the thing that just broke". But getting through management to get a fix or change to the broken thing was near impossible.

  10. AceRimmer

    Every company

    There was an article in Forbes last year covering this very issue:

  11. Anonymous Coward
    Anonymous Coward

    "Banks need to start thinking of themselves as IT companies, said Professor David Chan of City University London"

    As opposed to PayPal that seems to be an IT company that thinks it's a bank.

  12. Magister

    Strongly disagree

    'A retail bank is nothing but an IT company with a banking licence'

    So wrong in so many ways. A retail bank is a bank; IT is just part of the infrastructure, a means to manage the business processes. If the processes are bad, no amount of IT will change that.

    I would agree that IT is now fundamental to how a lot of businesses work (including banks); but if a bad manual process is turned into an electronic process, that doesn't suddenly make it a good process. The businesses have to get their processes correct (including making sure that they manage risks) and if a business doesn't understand what its core function is, then it is doomed to fail.

    Let the flames begin!

    1. despairing citizen
      Big Brother

      Re: Strongly disagree

      Whilst I agree around the principal of there is no such thing as an IT project, only business projects with larger or smaller amounts of IT, in the case of our national banks, the Prof is right.

      RBS is an IT company with a banking liscense, because if you switch off the IT, the bank ceases to trade, and is therefore not a functioning Bank.

      A lot of managers, in both IT and Business areas, have not realised the mission criticality creep of technology, and many still think that IT is an optional extra. RBS demonstrated the fallicy of that perception.

      So maybe today the "IT Project" mantra in this tech heavy world should include a second line that says, "there's no such thing as a pure business project, only IT projects with a larger to smaller business component"

      1. Anonymous Coward
        Anonymous Coward

        Re: Strongly disagree

        They actually still have an entire "paper based contingency plan" for just about everything AFAIK. So no, the bank does not cease to trade or function. It just functions painfully slow and gets a throwback to the 18th century.

        1. Anonymous Coward
          Anonymous Coward

          Re: Strongly disagree

          There may be a "paper-based contingency" on paper which would give the bank tellers and other wetware an illusion of functionality.

          In reality, the inter-bank transfer processes cannot happen manually.

          If those fail, the bank will be gone inside 3 days.

      2. Anonymous Coward

        Re: Strongly disagree

        "RBS is an IT company with a banking liscense, because if you switch off the IT, the bank ceases to trade, and is therefore not a functioning Bank."

        Any business depends on a selection of tools, people and processes. IT is fundamentally a tool closely attached to the processes. It is the money making processes that define what the business is, not the tools. If I cut RBS electricity, water supply, or postal service off they'd soon cease to trade, but that doesn't make them a post office, or a 'leccy or water company. The utilities have large IT infrastructure, but that doesn't make them IT businesses either. The Prof needs to think again.

        1. Richard Wharram

          Re: Strongly disagree

          It's not just a tool. It's the ONLY tool which can allow a modern bank to run. Money exists in IT. Whether that makes a bank an "IT company" is just a matter of words. The fact is that RBS (and other banks in my experience) have allowed a senior management culture to develop that thinks that IT is just something that can be bought in as and when required as a commodity. Reality is not like that but who has the balls to tell them the risks they are taking?

          I've been on the sharp end of senior management sharing their views with us about the relative unimportance of IT compared to the real banking business and suggesting that if we want to work in IT we should get a job in an IT company :) Quite charming really.

      3. Fatman

        Re: ...many still think that IT is an optional extra

        And it is these Luddites that often cause so dammed many problems for IT because they fail to understand how IT impacts their business.

        Imagine a wholesale distributor who (IMHO, stupidly) migrates core billing services to a `cloud provider`, and has to deal with the aftermath of the cloud provider fucking up. Can you say insolvency??

        Yet there are those members of damagement who only look at the spreadsheet, and see the line item cost for IT, and think: "We have to cut that, or else we won't get our quarterly bonus". I have seen it happen. My employer bought up such a company, and the first thing we did was to kick the asses of the entire `C` level suite out the door. The morale there was in the pits, as the employees did not give a shit about damagement, because damagement did not give a shit about anyone else but themselves.

        One of the first questions we were posed with was: "What is going to happen to the remaining IT staff"? A lot of jangled nerves were soothed when they were told that the cloud was being ditched, and the outsourced IT functions were being brought back in house.

        "Cloud??? We don't do stupid shit like that for core billing services." We don't want to put a core business function in the hands of a third party, whose only risk is having to refund a portion of your monthly bill for failure to live up to its SLA; while for the business, the risk has no limits. Bad proposition, IMHO.

    2. FutureShock999
      IT Angle

      Totally agree...

      A bank has to be a good IT practitioner, but it also has to have good marketing, good sales, good service management, and as noted above good risk control. Oh, and also know a whole lot about, er, banking, which many in IT will probably think is a lot easier than it is, as there are just so many rules, regulations to comply with, and intricacies - BASEL III anyone?

      When I look at the banks that I have consulted for or worked with, they KNOW the importance of IT - they can see it in their budgetary lines, and by how much floorspace IT takes up. Unfortunately, they also know that most in IT sit a lot lower on the totem pole than bankers, marketers, risk management staff, etc. IT may just beat out the call centre staff...maybe. So when a bank wants to look in a mirror, they do NOT want to see themselves as an "IT organisation". Maybe if we in IT decided to start wearing suits again, take off our MP3 players, hire more women - maybe then it would be more attractive. But as long as IT are socially and paywise down the scale, banks (run by bankers, you will remember) do NOT want to think of themselves as IT companies. And can you blame them?

      But that does not excuse the central point of the article, which is about IT expertise, not just focus. You can have IT as a support function, but you can still develop, nurture, and retain stellar expertise in IT. LIke Taleb's Black Swan event, you can count the cost savings of not having true expertise for many years running - until the day it all goes titsup, leading to huge losses... Banks are supposed to be good at risk management, and it is time for them to re-run the numbers of their savings, factoring in the possibilities of huge negative events like these...

    3. No, I will not fix your computer

      Re: Strongly disagree

      You can strongly disagree all you like, but your argument just shouts agreement, you say "IT is now fundamental to how a lot of businesses work", it would be much easier job to say which businesses are not fundamentally dependent, even the corner shop has accounting and ordering systems which are crucial, and if they could get by without IT then could their suppliers? (i.e. they are still dependent).

      Mapping a business process to an IT process takes a special kind of animal, one who understands business and IT and can translate between the two, you can consider this a business function or an IT function, but in reality it's irrelevant where you put the headcount SOA exists because you can have off the shelf systems which do what most businesses need - in which case you're just an IT reseller, providing wetware for the SOA, if your product can work with SOA then you're developing bespoke IT systems.

      The reason why you're wrong (and the statement is right) isn't to do with the fact that a valid business plan is required it's because you're forgetting that "IT Company" is just "IT + Company", the paradigm has already changed.

  13. despairing citizen

    Still missing the DR/BC plan?

    Lots of talk about change management preventing the problems, which is always a good approach, but RBS is suppose to have a BC/DR plan.

    In this day and age it should be unacceptable for a major bank to lose mission critical capabilities for more than 24 hours.

    So where was the BC/DR plan that says this part of the system has been "nuked", (in this case by operator error rather than the data centre burning down), this is how we restore it?

    1. Anonymous Coward
      Anonymous Coward

      Re: Still missing the DR/BC plan?

      Even if they have one, what are the odds of it being tested regularly?

      Most businesses cost-save, so having a duplicate system to fully try stuff out on (without breaking the operational system) is seen a a luxury, until there is a centrifugal excrement incident to be managed...

      1. Anonymous Coward
        Anonymous Coward

        Re: Still missing the DR/BC plan?


        Even if they have one, what are the odds of it being tested regularly?


        Every system goes to DR at least once a year (well, that's the intention), there's an entire team for it. This issue was because the old version couldn't be restored with backup data from the new version, the old data couldn't be used because it was two days out of date.

        Imagine having an iPhone with iOS 5, you have a backup and upgrade to iOS 6, you use your phone, have emails, messages, games etc. after a few days you find out there are serious issues and need to go back to iOS 5, fine you have a backup, but you'll lose any changes, not a big deal, little annoying, but hey - now imagine the same thing with a major banking system, you have a "point of no return" where you've been running fine for two days, data is changed significantly so you can only "fix forward", but it turns out that that ceases to be an option, you're in a no man's land, what if you were told that the backup from the new system could be loaded into the old system? DR and BC plans are all well and good, and even testing them is practical, but there are limits, can you test a gun without putting a live round in it? can you test a payment system without actually making a real payment? and if you can, is one payment enough? how many days before a new system is tested?

  14. Anonymous Coward
    Anonymous Coward


    How can "money" grow when it has not existed since 1971? Its all numbers in a machine :) Which makes RBS and all banks IT companies.... Of course the fact they sack most of their IT staff at every single economic dip kind of says it all...

  15. Phil Endecott

    Does this screwup matter to the bottom line?

    The question that interests me, and I suspect many high-ups at RBS and other banks, is how much damage will this ultimately do to their bottom line? I.e. how many customers will move their accounts etc.

    For years, telecoms companies obsessed about getting the reliability of their systems up e.g. 99.9999% uptime on your landline. Then along came mobiles and VoIP, which were popular because they were more ubiquitous and cheaper respectively, despite being much less reliable. Companies like AT&T were looking the wrong way and were too big to change when they realised what was going on; they were optimising the wrong thing.

    So, do bank customers really care enough about problems like this to actually close their accounts? If most of them don't care and stay with RBS, then RBS got it right by making their IT less reliable. What do customers care about enough to change banks? I suspect that no-one really knows....

    1. Red Bren

      Money where my mouth is

      "how many customers will move their accounts?"

      Speaking from experience, one. When I needed to release some equity from my home to re-organise my finances (like a small scale version of a bank bailout), RBS didn't want to know, even though it would have reduced my total outgoings. So I'm moving from RBS to the Co-op as I'm tired of the mis-managment, rewards for failure, privatisation of profit and nationalisation of risk.

      It just happens to be a convenient time for me to change banks, not everyone is so lucky.

  16. Michael 36

    'Professional' manglement?

    "the senior team believe that they can rule by “dictat” without having to understand the consequences of their decisions"

    You can't manage what you don't understand.

    1. bpfh

      Re: 'Professional' manglement?

      O rly?

      This is the definition of a diktat, or as Jacques Chirac once said about a young upstart of a home secretary (Nick Sarkozy) : "I decide, He Executes". I dictate what I want, just make it happen.

      With the size of these "too big to fail" companies, this is what delegation is. You need underlings to sign off because there are not enough hours in the day to read 2000 emails, sign 30 contracts, manage unions, budgets, meetings etc etc etc ad nausam.... the problem is also that you are so far up the chain and removed from day to day operations, what ever those operations are (IT, HR, FM, banking operations that can be subdivided into zeus knows how many subdivisions (high street, business, corporate, stocks, investment etc...), then you end up managing your company though the balance sheet. Screw the effects, my accounts say that salaries, IT and real estate rental for the high street banks are my 3 biggest net costs, let's axe 20% across the board... but the net operational effects are totally different from front office and back office operations. This has not always been taken into account...

    2. Anonymous Coward
      Anonymous Coward

      Re: 'Professional' manglement?

      There are two sides to this particular coin.

      The business managers who say "I don't do IT", and think that is a good thing. Rather than a reason to be fired the same way that would happen instantly if they told the CFO " I don't do budgets"

      The other side of the coin is the number of IT departments and managers that I have met that are incapable of explaining the implications of the various IT options to a business manager in business terms, so that he/she can take an informed business decision. (this includes one knot where IT where saying iminent collapse of business critical system, but what the business manager was hearing was IT want to spend money on new toys that don't add business functionality. Once explained, the budget for the replacement kit was authorised PDQ)

      The edge of this coin is made up of the issue of credibility in the organisation, and IT managers tend to be pretty bad at "selling" IT to the organisation as a proffesional advisor that finds ways to make things happen, with many companies viewing their IT department as an obstacle rather than an enabler and innovator.

  17. Andy Farley
    Thumb Up

    I'd like to hear more about

    how the change management process is counter-productive.

    It's something I've, well, "felt" for a while but if it could be quantified & qualified I'd be very interested in hearing the details.

    1. Peter Gathercole Silver badge

      Re: I'd like to hear more about

      It depends on your perspective.

      If you are the person planning the change in a hurry, or the management committing the resource to do the change, then an over-rigorous, time consuming process is the last thing you want to add to your work or costs, so you do your best to short-circuit the process to make the change happen faster and cost less.

      If you are the risk manager, who is on the line if changes cause problems, then you want as much process around you to protect your butt (and to a lesser extend, the organisation they are working for), and then a bit more. You feel most secure if there is no change at all (that's counter-productive).

      If you are a diligent IT professional, then you want the *right* amount of change management to make sure that the change has been carefully considered, and has a good backout plan, but not so much as to make planning the change more difficult than it has to be.

      It is this balance that is missing. You see it swinging from cost-reduction to risk management according to the current trends in risk and management style and the most recent disaster. And always nowadays, the people who understand it least are the ones dictating the processes.

      If you are in a large organisation involved in change management, take a change and estimate how much the change costs in people and financial terms. Look at the time necessary to cross the 't's and dot the 'i's. Count the number of people involved. Look at the number of people who have to read and understand the change. Add up the people-hours spent sitting in the change board meetings.

      You can often find in places like a bank that a change to switch servers from one DNS or time server to another (often simple but with a potential high risk and impact if it goes wrong), which may actually only take minutes to do, ends up costing you dozens of man-hours (or even man-days), involving people on quite high salaries, and many days or weeks to drive the process. All of these things cost money, one way or another.

      1. Andy Farley

        Re: I'd like to hear more about

        Cheers Peter & Sir Spoon.

        Don't know why that other guy is talking to himself.

    2. Sir Runcible Spoon

      Re: I'd like to hear more about

      I can give you one example - a process so convoluted that 80% of the change raisers time is spent on navigating the multiple levels of sign-off - leaving just 20% to focus on the impact of the change.

      Most of the time, if the people are good enough this is sufficient - but sometimes it isn't.

      Great article btw Anna, hits all the right spots without being rabid :)

      1. amanfromMars 1 Silver badge

        I'd like to hear more about novel intelligence solutions, for in those is where the answers rest.

        I agree about the great article btw Anna, Sir Runcible Spoon, but it missed out on all the juicy hot spots which are highlighted here ....... "The American people have suffered for decades from the declining purchasing power of the dollar. The Federal Reserve has abused its position as the monopolist issuer of currency to enrich Wall Street and impoverish Main Street," stated Chairman Paul. "The Fed can effectively create money out of thin air with impunity, while creators of gold and silver currencies face jail time. This is a travesty. The only way to stabilize the economy is to return to monetary freedom by legalizing Constitutional money. Until the American people are free to choose the money they want to use, and not what the government forces them to use, the economy never will be truly stable and any recovery will be illusory." ..... with the Bank of England also falling into that sticky sweet honey trap of creating wealth from nothing and trying to deny it is something which is roundly abused in closed circles for fabulous riches. And we haven't even mentioned the euro trying to get in on the act and create another layer of extremely well paid fools who would think they can use money as power to try and control the masses with its rank withholding rather than stock supply ...... and another group of self-serving, wannabe emperors who would excel at fiddling to the detriment of everything else.

        Apparently tomorrow there is a pow-wow about it ......

        And as Libyans and Iraqis have found to their cost and at the cost of increasing monumental US debt which is never intended to be repaid, but rather more crazily expects to be able to expand exponentially with no ceiling, is military invasion and a war of terror and death and resultant destructive chaos the other preferred option to trump any nonsense of jail time for creators of gold and silver currencies and/or any other default method of payment outside of existing increasingly worthless paper notes/dodgy government bonds/disgraceful IOUs. And that is an unsustainable and self-defeating socio-political strategy which creates a world of smarter enemies to attack and remove all means of wealth generation and global support.

        1. Anonymous Coward
          Anonymous Coward

          Re: I'd like to hear more..

          Mr Mars, as time goes by we are seeing and hearing more about the strings that are controlling our destinies. Trouble is what we as individuals can do about it - and if someone does have a good idea about what to do how are we to trust again?

          1. amanfromMars 1 Silver badge

            Re: I'd like to hear more.. OK, sure. It is a pleasure to oblige

            Mr Mars, as time goes by we are seeing and hearing more about the strings that are controlling our destinies. Trouble is what we as individuals can do about it - and if someone does have a good idea about what to do how are we to trust again? … Anonymous Coward Posted Wednesday 1st August 2012 19:53 GMT

            That is quite the paralysing Catch 22 for controlled humans, AC, which fortunately does not affect that and/or those which lead them/feed and seed them new processed information, which would be novel advanced intelligence already comprehensively beta-tested in a radically stabilised alternative situation ……. completely changed environment which would require and provide completely different leadership arrangements.

            In such completely changed environments, are new ideas easily exercised autonomously by revisionary leaderships as they encounter no prevaricating opposition and significant encouragement from earlier catastrophically failed administrations which have themselves benefitted immeasurably from revised and revolutionary human programming and ITs Reprogramming of Powerful Command and Awesome Remote Control of Virtual Reality Systems for the new completely changed environments.

            And with regard to ….. "Trouble is what we as individuals can do about it - and if someone does have a good idea about what to do how are we to trust again?" …. you as individuals need do absolutely nothing about it, which is more than just fortunate should you not be intellectually equipped to do anything about it, for IT is doing everything for you. And trust will return, if ever it is again needed, whenever intelligence levels are raised and one starts to see the benefits of changes that radical solutions you may not have been immediately aware of, are shared/revealed/exposed/leaked. ……. and one further realises that that is no chance coincidence either, but all part of a pre-ordained and simply complex master plan, with SMARTR Leading Parties/IntelAIgently Designed Entities simply following complex instructions received from communicating nodes in …… well, let us just call them Surreal InterNetworking Systems for Virtual Space Places in Live Operational Virtual Environments and be done with it, and get on with it and get it on with IT.

            Or you could just remain much as you are, paralysed in ineffective inaction in systems which are collapsing.

            Methinks a quantum leap into radical change is a much better option though. What say you?

  18. Anonymous Coward
    Anonymous Coward

    "out-sourcing can exacerbate bad management"

    That's not hard to see, for managing at a distance is harder than being there in-person. And that distant relationship also tends toward "work to the letter, not the spirit", even without factoring in cultural differences. Not to mention that arguing back and not losing the contract come renewal time is that much harder at a distance, too.

    Now please follow my reasoning here: If you're good at managing, you know how to attract and retain good people. You can make do with a small crew of highly skilled people, and don't need to resort to off-shoring to try and "cut costs" despite all the extra back-and-forth due to, well, the myriad differences you get from hiring people a continent or two away. Thus, while you could argue that there's nothing "intrinsically" wrong with outsourcing apart from the black box-thinking that also manged to exacerbate the problems, the reasons for trying it on stem from failure to get it right locally. IE, bad management.

    Then again, we'd already established there were signs on the walls aplenty, and we have a nice reminder that rules alone do not an organisation make. So what do they do when the rules fail? Add more rules, of course. Might as well put the whole thing in the bin and start over. With qualified management, this time.

    Too bad that banking itself is regulated to the gills, so you won't see an actual lean&mean IT-shop-offering-banking anytime soon. There is no such thing as a sanely run bank for exactly that reason. And yes, I'll take a no-red-tape-attached banking licence suitable for an entirely 'net-run banking IT shop for a hundred quid, Adair.

  19. Anonymous Coward
    Anonymous Coward

    Another rulebook of procedures?

    Such 'rulebook of procedures` are designed to insulate senior management from the consequences of their decisions. As such they are highly effective ..

  20. Christian Berger

    Realizing you are an IT biz won't help

    Since more and more IT bizzes are run like banks now anyhow. That's why there has been so little innovation in the last years.

    What they need to realize is that IT is important. That spending money on it, is a strategic investment into the future. They need to understand that it's not just about keeping their current systems running, but also about inventing the future of automated banking. They need to understand that if they don't so that, they will go broke eventually, loosing the market to new upstarts like PayPal. PayPal is not successful because it's good, but because the other banks are so bad at doing their business.

    However in a world where banks are "to big to fail", that doesn't matter. It doesn't matter what you do because in the end the taxpayer is going to pay your bonus anyhow.

  21. This post has been deleted by its author

  22. celticnomad

    Good Article - 2 key mindsets that have assisted in creating this scenario.

    This article pretty much sums it up. There are however 2 other issues that were touched on but which I have drawn my own conclusions over the years.

    1 - Pretty much all organisation consider IT as only a cost centre (as opposed to a profit centre). They therefore continually press for reduction in operating costs. The lack of real intelligence is startling as there seems to be little appreciation of the fact that without IT the organisation will cease to be. In the RBS case it would be very interesting to see the figures provided for the cost savings of the IT restructure and how that stacks up against the financial impact of this one incident (I'm sure there are others that have costs the organisation but have not been so widely publicised)

    2 - The change management regeime at RBS has (for many, many years) been driven with the wrong philosphy. It is run like a police state. They use it to create fear and to enable isolation of staff at fault. The change management process SHOULD be a service for the IT function (as well as other business areas) to facilitate change. I should assist in ensuring change is implemented in the correct manner , with the correct considerations and as efficiently as possible. As intimated in the article, the process is so complex @ RBS that the staff involved in change spend so much time on the administration that (given difficulties with obtaining change windows and the volume of change that is required to keep the operation running), that administration takes away focus from the technical detail. Again as the article indicates each time there is a change related technical incident , the WHOLE of the IT business endures increased process/admin, in most cases complete lock down. Which again creates backlog of mandatory change (either technical to remain on supported versions, or regulatory to ensure tha bank is allowed to operate). The whole situation creates inevitable failure and only does more so as time goes on and a multiplier effect takes hold.

    Both of the above can be explained by the management style at the bank, where it's extremely rare for management to be challanged by their direct reports once you get past the front line technical staff.

  23. Trixr

    I hate to say it - but an engineerinng perspective is appropriate

    I dislike it in general when engineering principles are shoehorned willy-nilly into IT operations, but in this instance it's worthwhile. The analogy with aircraft at the end of the article echoes this.

    Firstly, someone has actually carried out a proper root-cause analysis here, in terms of looking at the actual decision and process chain, not just individuals and software products. That's a refreshing change in this business.

    The next step is to follow up that root-cause analysis with actual fixes and improvements.

    In the aviation industry, while certain standards are thoroughly mandated, what you do underneath to achieve the right outputs is kind of moot, as long as you have your standards covered (e.g. triple redundancy, outputs in x format, etc etc). If there is a failure, then the regulator can and will pull an operating licence if there is a standards breach. Root-cause analysis will go right back to manufacturing processes, staff management/training processes and so on and so forth to base its airworthiness directives on to enforce the appropriate fixes.

    Cumbersome processes have been stripped away with airworthiness directives - at one point, there was a dozen-step procedure for pilots to get through before they could evacuate themselves from a burning plane. I believe it is now 4 steps (shut down engines and fuel being the major components).

    Of course, in this instance, the regulator has the power to enforce anything they like, with the operating licence as their ultimate fallback. Financial sector regulators seem to only concern themselves with accounting standards - ok in the days of paper - but this is not the reality now. Perhaps they need to start concerning themselves with all the operating aspects of a financial entity, including how transactions - people's livelihoods and life savings - are processed, secured and managed, beyond the basics of balancing the books.

  24. Eduard Coli

    Funny that

    Out-sourcing to the lowest bidder brain dead out-slaver had a little to do with it too.

This topic is closed for new posts.

Biting the hand that feeds IT © 1998–2021