back to article Do you work in IT at RBS? Or at the next place to get hit ...?

Those nice people at RBS have provided me with a worked example for how you can best get through fan-hitting situations without having to take up minicab driving or going to work in local government. What I’m writing applies to the any of the very worst of screw-ups, when technology has gone so wrong that the IT director …


This topic is closed for new posts.
  1. Anonymous Coward
    Anonymous Coward

    Help your manager look good?

    Don't make me laugh.

    They are in 'self preservation mode' just as much as you. The only difference is that it is far easier for them to knife you in the back that it is the other way round.

    In situations like these, don't trust anyone not to have a large pointy thing hidden behind their backs and giving them the opportunity to 'stick it to you' will give them an immense amount of satisfaction even if you were sunning yourself on a tropical beach then the catastrophy happened. somehow it will end up being your fault.

    The advice to keep your head down and look busy is spot on though.

    Anon as I don't want to give my boss a target. He reads 'El Reg' when he does not have a lot to do which for some reason is at least 6hr per day.

    1. Captain Underpants

      Re: Help your manager look good?


      Bear in mind that one way to "help your manager look good" is to help him show that he's on top of what his hard-working team are doing to help fix the problem.

      I agree that helping your manager look good without also making yourself look good at the same time is silly and downright dangerous, but TBH given that the thinking going on here will be giving relatively little weight to issues like "actual technical competence" means that dealing with corporate insanity is the order of the day. So in that regard, same as it ever was...

    2. Psyx

      Re: Help your manager look good?

      "Don't make me laugh.

      They are in 'self preservation mode' just as much as you. The only difference is that it is far easier for them to knife you in the back that it is the other way round."

      And who are they going to knife first? The person supporting them and being positive and not shafting them to other managers, or the bitter, miserable cynic who loudly slags them off as well as senior management.

      Helping your manager doesn't mean trusting your manager. It means making yourself less of a target than the people being awkward.

      1. Sir Runcible Spoon

        Re: Help your manager look good?

        "And who are they going to knife first?"

        The old adage eh? You don't have to run faster than the wolves, just your colleagues :)

    3. Anonymous Coward
      Anonymous Coward

      Re: Help your manager look good?

      "Don't make me laugh."

      Oh dear.

      1. Aaron Em

        Re: Help your manager look good?

        That said, if he'll helpfully don a bullseye like that, he probably isn't bad to have on your team when it gets around to sauve qui peut o'clock.

    4. hplasm

      Re: Help your manager look good?

      If you haven't worked in Banking, you won't understand...

    5. Anonymous Coward
      Anonymous Coward

      Re: Help your manager look good?

      Ya sure it's your boss and not an unemployed websurfer? OK you probably don't have those INSIDE your building;)

  2. Annihilator
    Thumb Up

    Great article

    Should be committed to memory for anyone working in any discipline. SNAFUs like this happen to varying degrees to everyone at some point and how you react to it is by far the biggest action that is judged.

  3. IT Hack

    Having been in the driving seat of teams that deal with extreme outages/fuckups the worst thing you can do as a team manager is demotivate the troops. For me its not about looking good but doing a professional job both technically and managerially. What goes on in the stratosphere is of no concern to me as a support manager. I have absolutely no chance of influencing anything there. What I can do is make my team look really good by leading and ensuring that the right messages are being sent. Dominic is absolutely correct that saying a problem is fixed why the symptoms are gone is madness without a time for the change to bed in. This is easily covered in a fact the ECB should give the time when a decision is made that a change is successful. I say ECAB coz this kind of thing is an emergency and needs precise management.

    Anyway back to the point about making your team look good...when it comes around for bonus time I have a stronger case to get a larger bonus for my team than if I was just a manager in name only and kisses the ass of management. For me the profit lies in having not only a dedicated team but a happy team.

    Of course the killer to all this is when you a sociopathic VP running around like a headless chicken embarrassing not only himself but the company. Seen that too many times.

    The Pint speaks truth.

  4. sugerbear

    Isn't it easier just to pass the blame around

    Batch services can blame the implementation crew

    The Implementation crew can blame the developers

    The developers can blame batch services

    .. and repeat.

    With so much blame being passed around senior managers wont actually know who to blame.

    Actually, I know the best thing to do is to keep your mouth shut. And then counter by claiming victimisation, bullying, stress, overwork, you can always throw in some blackmail as well.

    Another point (and I think a very good one) whoever was responsible will know that if they do get the push they will be in line for a nice 6 figure sum from one of the weekly rags. Max Clifford is very likely to be your best friend. Just make sure you come out looking the victim. (Nick Leeson didn't do too bad).

  5. Chris Young
    Thumb Up


    Cracking article, well done. I'd love to see the face of a senior RBS manager while reading this. Horrified, annoyed, furious ... all thw while knowing there is more than an element of truth to it.

    This is possibly the one and only time that I'll say I'm glad I bank with HSBC ...

    1. Anonymous Coward
      Anonymous Coward

      Re: HSBC

      I have dealt with senior management at HSBC. Never, ever again. I rather keep my money in a sock with holes in.

      1. Colonel Mad

        Re: HSBC

        First Direct are running some tasks tonight..................................

    2. Anonymous Coward
      Anonymous Coward

      Re: Excellent

      You wouldn't say that if you'd had to deal with their IT dev teams.

      Clue - they're not based in this country.

  6. Jason Hindle

    Another excellent article

    Deliciously cynical as always.

  7. Anonymous Coward
    Anonymous Coward

    Sitting on information....

    we just had a cock-up on our multinational application where one section tried to fix the problem then after 4 days of fluffing around, instructed my section to perform a major reversion. Pity that in 5 minutes of querying them, I had identified the root cause but after 4 days of failures, management did not want to risk another attempt to fix.

    Keep you ear to the ground for problems that you might not yet know about, and don't be afraid to butt in where you can reasonably do so (I always try to keep good social relations with the senior technicians in the various sections, so I can approach them casually about business issues that managers may not want to admit - call it "back-channels").

    1. Anonymous Coward
      Anonymous Coward

      Re: Sitting on information....

      I've always believed that if something even moderately nasty is on the cards, it is absolutely necessary to inform the boss. Of course, if that boss is the kind that likes to march around the office telling people he only wants to hear about results and solutions...

      1. Anonymous Coward
        Anonymous Coward

        Re: Sitting on information....

        If you see something nasty on the horizon you absolutely have to inform the boss, preferably in writing and holding onto a copy for yourself.

        Don't try and throw your manager under the bus, but it's handy to have in reserve incase your manager tries to do it to you. Where I work we are circumspect within our own team, but tend to do it sideways to other teams, we highlight issues and tell them about it, and if that issue happens because they have ignored it, and it has a knock on effect on us (and it will, or we wouldn't have seen it) we happily throw them to the wolves during the problem review.

  8. Anonymous Coward
    Anonymous Coward

    Is this guy capable of writing an article where he doesn't present himself as a smug tosser?

    1. Anonymous Coward
      Anonymous Coward

      "Is this guy capable of writing an article where he doesn't present himself as a smug tosser?"

      If he does present himself thusly, it's because he can.

      I like people who have nothing to prove.

      1. Jacqui

        banks and tossers

        Are there any bank staff who are not complete and utter W^HBankers?

    2. Anonymous Coward
      Anonymous Coward


      "Is this guy capable of writing an article where he doesn't present himself as a smug tosser?"

      Would you want the tone any different? Maybe the dry tone of an HR policy on "Professional Conduct in Critical incident Situations" would suit you?

      Personally I've been suffering withdrawal symptoms since last November*, and I'm delighted to read another beautifully crafted piece from DC. If anything, it isn't smug and cynical enough.

      * Yeah, DC's written some other pieces, but they've not been as provocative.

  9. Anonymous IV

    For "RBS managers" read "any managers", passim...

    1. Shagbag

      Best Advice for those in RBS IT?

      Get out.

      You're not valued by their management.

  10. proto-robbie

    That's a nice cheery punchline...

    ...just after I've banked a cheque.

    1. Matt Bryant Silver badge

      Re: That's a nice cheery punchline...

      Having worked for a few financial institutions, I can assure you that disasters from the "oops, that's gonna cost us" right up to the "man the boats" variety are not exclusive to RBS. Banks and their systems are designed, implemented and run by human beings, and no matter how good they think their quality control and project management procedures are, their systems are never perfect (not that I've seen anyway).

    2. I ain't Spartacus Gold badge

      Re: That's a nice cheery punchline...


      I've just arranged a mortgage with them...

      But that's OK, buying my house isn't too important a transaction. Oops!

      Oh well, what could possibly go wrong? If £100 turns up in my account then it just won't go through. If it's £100,000 job's a goodun', and it's £100 million, then "goodbye losers, I've just bought Tahiti."

      1. proto-robbie

        Re: That's a nice cheery punchline...

        Well, if it's been my £100M, I trust you'll stand me one of the attached down the Papeete seafront.

        1. I ain't Spartacus Gold badge

          Re: That's a nice cheery punchline...

          That's an impressive cheque you're paying in there!

          If it comes off, I'll be happy to buy the beers. I'll own the bar after all. I won't be paying your air fare though. Plus I may be shooting down incoming aircraft full of bank lawyers and anti-fraud policemen, so you'll need to be careful who you fly with...

  11. proto-robbie


    ...putting the "R" into "outage".

  12. Destroy All Monsters Silver badge

    Objectivism married to Machiavellism?

    Office Attilas and Witch Doctors shall get no traction!

    1. Aaron Em

      Re: Objectivism married to Machiavellism?

      ...and people say I put my politics where they don't belong...

  13. Anonymous Coward
    Anonymous Coward

    "That’s not enough for me to say that you shouldn't use RBS for any banking, but if it’s a transaction you care about, best do it somewhere else"

    Interesting thought. Consider your comment "understand the obscenely complex ways of ancient multi-layered systems". I've worked for a number of Banks, Pensions Providers and other non-FS businesses, and I can honestly say they are all getting MORE obscenely complex. Banking may have been complex before its time, but IT is getting more complex across the board.

    So when you choose to "do it somewhere else", think VERY carefully about where that somewhere else is. Everywhere is susceptible to unplanned errors and outages, and when that happens, would you rather a good experienced Banking IT operation was attempting to fix it, or would you be just as happy if it was the IT operation of, for example, a supermarket or a record company.

    1. John Smith 19 Gold badge
      Thumb Up

      "Interesting thought. Consider your comment "understand the obscenely complex ways of ancient multi-layered systems". I've worked for a number of Banks, Pensions Providers and other non-FS businesses, and I can honestly say they are all getting MORE obscenely complex. Banking may have been complex before its time, but IT is getting more complex across the board."

      Well duh.

      IT is a support service and bank support service managers love to do 2 things.

      Save money

      Deliver the same level of f**kups (at this cheaper price) that the old system does. They'd like less but they know the predators up the food chain won't eat them if it's at least as good.

      So how do you do this?

      Well don't *redevelop* stuff that already works. That's a complete waste of money (I know better but I've seen systems whose core code ran on emulation about 4 layers deep. Yes it was a bank and no it had never failed since the early 60s)

      Pull in other data from some other systems (on different hardware platforms & OSes) synch it all up, add in a (small) bit of new code and you have a whole new financial instrument for the front office to to play with.

      Banks buy *lots* of middleware.

      In *theory* the smarter move is to rationalize the data, clean out the 6 decades of cruft then upload to some super duper package so the support costs are spread across *all* package customers.

      Making that work in one of the UK high street banks will probably take *huge* bespoking of any package up to the job (I guess SAP has an app module for that 'cause they seem to have one for everything else) and the selective assassination of certain senior staff (not advocating it you understand, just saying things might go a bit smoother if some of the obstacles to "progress" were greased out of the way).

      Otherwise a first rate piece on how to survive the eternal battle between doing a piece of work you can be proud of while avoiding becoming the next prey animal for the backstabbing PoS who supervises you without resorting to lethal force.

      1. Christian Berger

        That depends on the decade of code

        Lots of middleware may be the lesser evil compared to really bad code from the 1990s. Just imagine a bank running on an Access server or some form of MS-SQL. You'd never be able to interface that with anything else reasonably.

      2. Anonymous Coward
        Anonymous Coward

        The rip it out and replace it with SAP route is exactly what Nationwide are currently trying to do. It's fair to say they are struggling somewhat to make it all work. Someone closer to the project will probably be able to enlighten you on the current state of the mess.

        1. John Smith 19 Gold badge

          I am dimly aware of this. Personally I think SAP's got too many fingers in too many pies and a better solution would have something industry specific which no one on the Board had ever heard of. but SAP gets the "Oh yes, I've heard of them. They're big and solid. Let's go with them"

          I guess Nationwide users will be getting a personal taste of how good a full replacement really is quite soon.

  14. Anonymous Coward
    Anonymous Coward

    "Do not get involved in arguments, wait until a consensus emerges and reinforce it."

    'nuff said!

  15. KroSha

    This is why I don't use a bank; I use the Building Society. At least their shareholders aren't driven by short term profits...

    1. Anonymous Coward
      Anonymous Coward

      Re: the Building Society

      As a public service, perhaps you'd like to list the ones that are left: the ones that did not turn themselves into, or sell out to, banks.

      1. This post has been deleted by its author

        1. Mage Silver badge

          Re: the Building Society: Nationwide

          But not the one called Irish Nationwide

        2. Anonymous Coward
          Anonymous Coward

          Re: Nationwide

          Interesting. I thought they'd sold out ages ago, but I was wrong, and I'm glad to know it.

          Thanks very much.

          1. Anonymous Coward
            Anonymous Coward

            Re: Nationwide @Thad

            "I thought they'd sold out ages ago, but I was wrong, and I'm glad to know it."

            Well, their board are the usual city fat cats, and I haven't even had to look that up on their website because I've just had my voting papers through. They are overpaid, and to judge by comments made earlier, they've swallowed the consultant's medicine and are now trying to install SAP, which if it is anything like their enterprise software will IMHO be a pile of complex, user unfriendly shit, wholly unsuited to a retail banking business.

            As a mutual you might expect them not to be so slavish in following the fashions of outsourcing and offshoring, but they outsourced IT infrastructure to BT back in 2008, and other "usual suspects" named in the SAP adventure include Capgemini and KPMG. That'll be doing wonders for me as a member, then.

            Regretfully, I have the impression the Nationwide ain't that different from the other retail banks (admittedly minimal invvestment & casino operations), and the board are rather comfortable knowing that nobody ever reads their annual report, they cannot be taken over, and many of the members will surrender their votes to the board proxy.

            1. Aaron Em

              SAP is well named

              If you think it's a good idea, it's evidence you are one.

      2. Velv

        Re: the Building Society

    2. Anonymous Coward
      Anonymous Coward

      Go to the COOP?

      Not a Building Society (and not technically a co-op, just owned by one), but ethical banking (or as close to ethical as you can get), they also have had their own share of IT problems, they turn away unethical business (make less money because of it), they close accounts of people who they think don't fit with their dignity and diversity policies.

      They might screw up, they might not pay the best interest, they may not have the best customer service, but they're pretty good and they are ethical (ish, it's money - can money ever be ethical?).

  16. h4rm0ny

    Fundamental misconception.

    The author of this article does not seem to understand that people who are good engineers are usually so because they enjoy doing things right and take a viewpoint that is based around what is optimum for the system, not just their little nugget of it. We have become engineers because we find greater satisfaction in actually being good people who move things forward, than we do playing internal politics and bickering over our share of an increasingly small pie.

    Ultimately, companies that foster this sort of attitude lose all the really good people.

    1. Anonymous Coward
      Anonymous Coward

      Re: Fundamental misconception.

      You're unlikely to be working in this kind of a company.

    2. BoldMan

      Re: Fundamental misconception.

      Sadly what you fail to appreciate is that in a business you can't JUST be an engineer, you have to be a politician to survive to use your engineering skills in teh future, otherwise you might end up exercising your burger frying skills...

    3. Anonymous Coward
      Anonymous Coward

      Re: Fundamental misconception.

      It could be argued that being a good engineer is not going to do you or the system much good if you get shitcanned because you didn't have the social / political skills to stay in the game.

      Think of it as a coupled system with politics being an integral part of it.

      1. Anonymous Coward
        Anonymous Coward

        Re: Fundamental misconception.

        Sorry, I've just repeated what BoldMan had just said.

        1. I. Aproveofitspendingonspecificprojects

          Re: Fundamental misconception.

          Sorry, I've just repeated what BoldMan had just said:

          It could be argued that being a good engineer is not going to do you or the system much good if you get shitcanned because you didn't have the social / political skills to emigrate to India.

          There IFTFY.

          No icon until we get a fixed inane grin Thatcher icon.

    4. I ain't Spartacus Gold badge

      Re: Fundamental misconception.

      The author of this article is probably fully aware of that. He seems to have taken his mission on El Reg as interpreting management to the IT labouring classes.

      The whole point of the article is that this is not how senior management at RBS operate. If it did, they wouldn't be a nearly bankrupt, partially nationalised, badly-lead, deadbeat organisation with a tenuous grasp on morality and competence.

      As you say, companies have forgotten that most people would like to do a decent job, and leave the politics to someone else. Morale is more damaged by not allowing people to do their job properly than it is by pay cuts. If only management could align the company's and staff's interests more, they'd have fewer management problems and make more profits. The problem is that this is very hard to do. See as an example the idea that giving CEOs vast amounts of shares will align their interests with shareholders - and how badly this has failed. And how much harder than getting one person's pay and conditions right, is it to set the whole culture of the company?

      1. TheOtherHobbes

        Re: Fundamental misconception.

        There were - occasionally - companies with saner values. But currently most corporate culture, especially in finance, is 75% plain old feudalism and 25% lies and spin.

        The smart thing for competent engineers to do is simply to boycott it. Not only will you avoid Hello-magazine levels of pointless shitty drama, but you also won't be spending your days developing ulcers trying to please people who are - in a very literal sense - mentally ill.

        1. Aaron Em

          Re: Fundamental misconception.

          Boycott away -- I never could afford college, so I'll never see a better chance to learn how to do your job!

    5. Aaron Em

      You may be a better engineer than I am

      but I'm more likely to survive the giant balls-up with employment intact, because I understand there's a game and, distasteful as I find it, I know it's worth playing. It's that or sit on the sidelines, and being an unlettered hick with a big mouth means non-profit organizations find me unsuitable for employment, so I get to choose between "sit on the sidelines" and "miss my mortgage payment" -- and I guarantee you the bank is not going to inquire into the specifics of my high-minded sense of ethics as part of the foreclosure process.

    6. JDX Gold badge

      Re: Fundamental misconception.

      The author of this article does not seem to understand that people who are good engineers are usually so because they enjoy doing things right and take a viewpoint that is based around what is optimum for the system, not just their little nugget of it. We have become engineers because we find greater satisfaction in actually being good people who move things forward, than we do playing internal politics and bickering over our share of an increasingly small pie.

      It's your choice - you can listen to the guy with 25 years' experience. Or you can practice reciting that speech for the job interviews you're going to be hoping for soon.

    7. John Smith 19 Gold badge

      Re: Fundamental misconception.

      "The author of this article does not seem to understand that people who are good engineers are usually so because they enjoy doing things right and take a viewpoint that is based around what is optimum for the system, not just their little nugget of it. "

      I think you'll find he gets it very well.

      If you happen to work for someone who does value that viewpoint congratulation. You've very lucky.

      This is for the rest of the leadership to avoid being caught holding the ticking box with the blank redundancy notice when the music stops. His comment about people becoming nearly clinically depressed is a big of a clue on this.

  17. BadmashJavawallah
    Thumb Up

    Splendid Stuff

    And some of it so eerily familiar.

    I think we need a regular Agony Uncle column from Dominic to solve our political dilemmas.

  18. chris lively

    Having been through similar drama several times myself, this article is absolutely correct.

    Summary: Band together, listen to all solutions, and execute while taking care of yourself and others.

  19. Stephen Channell

    Make lots of suggestions

    Always looks good, and sometimes actually is.

    A “responsible” manager under stress will be looking for a scapegoat and fixate on the first goat facsimile; making suggestions increases the probability that the selected goat is actually the right one.

    If it’s is your fault, make cunning suggestions like “when did the DB2 DBA last check the CA-7 backup”.. they may not know that the plan is actually in a VSAM file, plus lots and lots of options gives you a chance to fix it while problem managers are looking elsewhere.

    .. but on a serious note: “Or at the next place to get hit” might be an observation of stochastic probability, but these kind of disasters are now less likey to happen because the FSA is going to put a price on the disaster. Full weekend regression testing and seasoned lags starts to look like very good value for money.

    1. Richard 12 Silver badge

      Nope, they're not less likely to happen

      Corporate fines are at best a lagging driver. Nobody takes much notice until somebody else has been hit with a big one, and then it takes a long time to change course anyway.

      Aside from this, the risk severity was already as high as it could get - this current disaster would probably have triggered a state takeover of RBS had it happened when they weren't already zombified.

      Even as it is, they are going to lose a lot of customers.

      At the end of the day, the probability of this kind of failure is directly related to the complexity and importance of the system, and inversely to the experience of the people looking after it.

      As they get amalgamated, more financial astronauts come up with more or more complex 'instruments', and experienced staff retire, "retire" or are made redundant, the probability increases.

      So this is very likely to happen at least once more, and the only difference is that it's more likely to take a CxO with it next time.

  20. Anonymous Coward
    Thumb Up

    So, Dominic,

    Not looking to RBS as a client, then, I guess?

    First-class realworld sadworld badworld writing.

  21. EvilGav 1

    Sounds frighteningly close to reality. The number of times a major SNAFU occurs and the seniors all turn up asking "who did this?" rather than "how do we fix this?".

  22. Anonymous Coward
    Anonymous Coward

    Excellent Article

    My congratulations to Mr. Connor for a thorough and useful article, which is also entertaining to read. This stands in such stark contrast to the shallow, populist pieces written on this subject by Ms Leach.

  23. Peter Holroyd
    Thumb Up

    Fantastic article

    I've 33 years IT experience and this article is spot on.

    I just need to find a suitable place to store the bookmark

  24. Bruno Girin
    Thumb Up

    Brilliant article! My fav bit of it being: "Their offers of protection have less credibility than the reports of a Greek finance minister."

  25. Jeff 11
    Thumb Up

    Nice article

    Perhaps you should offer a Hindi translation?

  26. Anonymous Coward
    Anonymous Coward

    "blaming the indians"

    Even if it were someone in India that pushed the wrong button and irrevocably destroyed your customer base, that doesn't mean it was "the indians' fault". That's absolutely offensive. It is the fault of how outsourcing is set up in the beginning. Put untrained, inexperienced people in positions where they can blow their hands off, and you can't be surprised when gore splatters the walls. See the section "if it was someone you manage" and take it to heart.

    I have worked with phenomenally sharp and experienced IT personnel from India. I've also personally worked with "Peggy" -- an "admin" who quite literally says "yes" to everything you say, while you hear him in the background furiously paging through written procedures trying to find something that matches some phrase you uttered.

    What you get is (pay attention now, this is important) what you pay for. If you outsource to save money, it is inevitable that you will lose expertise. And you will *never* gain it back, because every time you train up new personnel, it is guaranteed that they will migrate to better paying jobs. You will never be able to keep qualified personnel, because you can't pay them what they're worth, even in India, without breaking the outsourcing paradigm.

    And this is because the very premise of outsourcing -- that you can get sufficiently trained personnel substantially cheaply while simultaneously paying the overhead of the outsourcing process (and worse, the profit margin of the outsourcing company) -- is false. What you get is an infrastructure that may chug along on sheer inertia for a little while, and then collapse.

  27. Anonymous Coward
    Anonymous Coward


    Articles like these is what makes me want to stab puppies. Let's take it from the top

    1)Respect your colleagues. Even if you think they are dicks. They will eventually come to appreciate you for it, and it will do wonders for the "office politics" (ugh)

    2) Help your colleagues. Pass on your knowledge. Give them praise for work well done. You are not a fucking drill sergeant.

    3) Never ever backstab. Never. it's just not cool, your karma will be shit, and you will be remembered for it.

    4) Own up to your mistakes, clearly. If you get fired for it, it will be the best lesson possible, you will *never* make the same mistake twice (unless you are a sociopath with masochistic tendencies)

    5) For the love of $DEITY, don't posture in front of your colleagues, we all know bullshit when we see it.


    1. Anonymous Coward
      Anonymous Coward

      Re: WTF.. (in order)...

      1. There's no point in respecting dicks. They'll never respect you - that's why they're dicks. If you can't sideline or sack them, make their lives hell.

      2. Completely agree (apart from the dicks).

      3. Only backstab dicks. Don't worry - they'll be doing it to you. That's why they're dicks (remember?)

      4. Only ever own up once for a mistake, no matter how many times you make it. The second time? Blame a dick.

      5. Completely agree. Don't ever be a dick. Unless you can be one anonymously.

  28. John Smith 19 Gold badge

    I smell some payback

    <- Who they always call on to fix things when it turns pear shaped.

    From the article he's met a few of these management types (not sure where the nearest golf course to BH is but not far by Merc S class) and not been too impressed when they're under pressure.

    Head hunters live and die by their records

  29. Anonymous Coward
    Anonymous Coward

    Not so funny

    I had a good laugh with RBS,

    Until thursday.. we had a very similar problem.. big company not giving service, cheap L2 service, no queue processing.. we just finished solving the problem.

    But we DO have scapegoats!!

  30. Anonymous Coward
    Anonymous Coward

    Problem was unique says head RBS honcho

    Well the screw up was unique. I've been in similar situations but then I could sit in the same room as the Ops guys and look over their shoulders. Don't envy having to give detailed instructions to an operator on the other side of the world ('I said post the dependencies and delete the job from the queue NOT delete the queue!').

    You also need the strength of personality to DO NOTHING until you fully understand the scope of the problem and fix required. Had to deal with management jumping up and down about missing BACS windows and online day being late and had to bat them back because we know that is nothing to the wasteland that we could create by destroying the data. Sorry if this is another dig at the Indians but I find that they are too eager to please and under pressure they want to do something. Disasterous in the situation that engulfed RBS. I've told many an Indian chap that they need to learn to say 'Fuck off' but they think it is just a joke.

    You can tell Hester what you think of his explanation. I'm sure he'll read it.

    1. mark daly

      Re: Problem was unique says head RBS honcho

      Excellent points.

      It takes years of hard experience in IT to know that in a crisis the best move is often to step away from the keyboard while you think through the issues The next step is to investigate the scope of the problem and perhaps most importantly to preserve the details of the software,the data and the logs at the time the original error occurred. Panicked attempts to apply quick fixes not only often fail but also frequently obscure the cause of the original problem. Management often ignore this uncomfortable truth in their rush for quick solutions. Of course, the problem for RBS and its associates is that they essentially threw away years of hard earned experience which wouild have been invaluable in this crisis when they made 1800 staff redundant. The charge for that folly has now arrived.

      1. Dominic Connor, Quant Headhunter

        Re: Problem was unique says head RBS honcho

        The RBS honcho was correct, I've done 25 years in banking IT and every single significant problem I ever had was unique, but three were patterns. If you have people out of their depth they will always do something wrong in different ways.

  31. Homer 1
    IT Angle

    I read as far as...

    ..."example for how" and "applies to the any of", then gave up.

    The article should be renamed "Blaim Avoidingance of for the an Iliterate".

    Here's my advice, and I don't need two pages to articulate it: If your situation is so bad that you live in constant fear of failure, reprisal and dismissal, clearly you're in the wrong place.

  32. Christian Berger

    The UK must be in a horrible job situation...

    ... when its worth holding onto even a job at such a company.

    BTW, if you want shitty jobs, go to Germany. We have plenty of those in the car industry. You can even get the BBC via satellite.

  33. Anonymous Coward
    Anonymous Coward

    With my more limited 6 years of experience in the banking industry I concur with pretty much every word in the article. Where I worked we had scenarios very similar to RBS but because we were IB and not retail it didn't get quite the same level of publicity. Nice article, shame it will change nothing.

  34. Anonymous Coward
    Anonymous Coward

    "I do get paid a few hundred quid an hour to do it"

    And there's the problem, right there. If banks stopped paying small fortunes to people like this, and instead used the money to hire/retain decent people who can do their jobs properly, then shit like this wouldn't fucking happen in the first place.

    1. Homer 1

      Well said.

      And not just in the banking industry, it's an endemic problem throughout the UK. There's far too many pointless dilberts with la-la qualifications in made-up positions, spouting meaningless gibberish for "a few hundred quid an hour", and bleeding the country dry as a result.

      They might as well hire a Klingon interpreter while they're at it, for all the good these buzzword-bingo monkeys do.

  35. Dominic Connor, Quant Headhunter

    Dominic replies to various comments…

    Dominic replies to various comments…

    @BadmashJavawallah That one comment justifies the effort I put into those pieces, if you’re ever in the City please contact me directly and I’ll buy you lunch somewhere like the Capital Club.

    Several people have said that that they prefer building societies, I point them to the example of “Thrifts” in the USA which are much the same thing, or should I say *were* since most exploded.

    @Meh is right about suggestions, I should have included that, but with the rider that you must make sure they are not seen as frivolous.

    @Thad RBS is not and will never be a client of my firm. They have outsourced recruitment to a firm whose male staff help you to understand that “bimbo” is not a term that only applies to women.

    @I ain't Spartacus Yes, explaining this stuff is a “mission” of mine, indeed one my better sidelines is lecturing on “soft skills” for bankers for a private training firm. You are also 100% right that nearly everyone wants to do their job well and get really depressed, sometimes to clinical levels if they are prevented.

    @jacqui asks if all bankers are wankers. I have to say that the vast majority of the ones I know (my work means I know more than most) would not be classed by any reasonable person as such. Certainly the City is the least sexist or racist industry I’ve encountered (note I say least, not zero). I’ve done stints in manufacturing, IT firms like IBM and Nortel and of course the media.

    Go to a City bar and you will see Managing Directors who are coloured and female occasionally both, do we see union leaders who aren't fat white males ? Is there any senior female UK politician who didn't get there on a quota ? Is there anyone with a "creative" job at the BBC or Sky whose dad who earned his living through manual labour ?

    Hint: Being finance director at a construction firm is not manual labour.

    City types of which I suppose I must include myself are not popular partly because so many are noveau riche and/or first generation immigrants like myself. and we're not freak show celebrities.

    I don't believe Bob Diamond, the head of Barclays should resign, I believe he should be taken in a carriage to Mansion House, given a nice lunch then publicly horsewhipped in front of the Bank of England. Either he knew or should have known.

    So please don't think I see the City through rose tinted glasses.

    As it happens I know a bit about the Libor process and can prove mathematically that BarCap did not act alone, the short version is that the system *cannot* be subverted by the rogue data points of one firm. This is an area in which I have direct experience with both the maths and personally writing code, I just haven't yet worked out how to turn it into a Reg article.

    1. John Smith 19 Gold badge

      Re: Dominic replies to various comments…

      "As it happens I know a bit about the Libor process and can prove mathematically that BarCap did not act alone, the short version is that the system *cannot* be subverted by the rogue data points of one firm. This is an area in which I have direct experience with both the maths and personally writing code, I just haven't yet worked out how to turn it into a Reg article."

      Well, that's one option.

  36. naive

    RBS is probably not the Red Army

    The picture painted in the article resembles the culture in the Red Army until 1943. Up to 1940 all experienced officers were killed or sent to Siberia by a paranoid Stalin and in case of a lost battle some surviving scapegoats, who were at the wrong place and time were put to the wall. It did not bring much, although they fought with a majority 3:1, they kept losing most major battles until culture changed somewhere in 1943 (and Germans had to sent soldiers to the defense of the French coast).

    The assumption that RBS senior management would be at a level were this culture is present is not proven. Probably someone told them they have a life threatening situation on their hands, the Bank IS the computer, not being able to process financial transactions for more then 48 hours is the death of any Bank. I worked at a bank, and have witnessed several times that in very serious situations, things are suddenly handled well. They will by now have formed a taskforce to sort this out, led by the most competent project leaders trusted by senior management. They also will understand that the recommendations of the taskforce should be the input of a blame-game held in a later stage. It is also widely known that firing some low ranking operatives within a week of a disaster does not improve their image in the media.

    1. Fatman

      Re: formed a taskforce to sort this out


      Look out for the 21st Century version of the Spanish Inquisition.

      You will be accused of heresy if you fail to support `the true Church` (aka manglement's version of events)

  37. ITSecurityGUY

    Failure to comply with Regulations, Policies, Procedures and Standards !

    I wonder why this change at RBS wasn't tested? and why it has taken so long to resolve? clearly the change was not properly vetted or signed off before being implemented. There can have been no "backout" incase of failure. Whoever is responsible for thie implementation without any contingency plan must be held accountable, and this includes the full managment chain....right to the very top. They failed in their regulatory requirements to perform due-diligence on the change/implementation, which at director level is incompetence and failing in your duties as a company director is already covered in UK Criminal Law... I would like to see this used in this instance to prosecute the Board at is time that the Law govening these institutions was fully exercised, if it is not, we will have no economy and no jobs...this event is worse than any cyber attack or physicall disaster...

  38. ITSecurityGUY

    Help!! RBS ???

    And I am AVAILABLE to help dig them out of the hole they find themselves in .....

  39. Anonymous Coward
    Anonymous Coward

    typical for anglo / usa banks

    nw-europe is quite different in a positive way

  40. Anonymous Coward
    Anonymous Coward

    Fuck that for a game of marbles!

    I'd rather end up in a council flat eating dog food at 70 that sucking up to some prick higher up the so called "food chain"

    They say pride comes before a fall, well, I'm prepared to fall - I *HATE* that internal politics bullshit, that rewards people who talk most, suck up most and tell the boss what he wants to hear.

    It's why this world is so fucked up. It's why our banking systems are so massively corrupt.

    They call it "playing the game" - I call it dirty tricks.

    It's the small company / freelance / contract route for me - there's no ways, especially at the age I'm at now, I'm ever going to want to play the corporate game.

    If your in this game and hate it, just do yourself a favour and get the hell out while you still can - surely anything is better than kissing the arse of upper management.

    1. JDX Gold badge

      Re: Fuck that for a game of marbles!

      I'd rather end up in a council flat eating dog food at 70 that sucking up to some prick higher up the so called "food chain"

      Then you probably will get your wish. There is the real world, and there is the nice little world in your head where people are promoted purely on merit, everyone confesses their own mistakes and never take credit for someone else.

  41. N2

    "Do not try blaming senior management out loud..."

    OK, so if 'senior management' isnt to blame how did the 1000 or so IT staff get culled; plague? deported? or just reprocessed?

    Perhaps if they were they still employed the fault may not have happened at all or if it did the consequences not as severe and systems restored sooner.

    In my opinion, senior management must accept responsibility for putting the banks systems at risk, in reducing key IT staff excessively & failing to supervise & oversee outsourcing correctly. But hey, dont let it get in the way of your next bonus....

    1. The Baron

      Re: "Do not try blaming senior management out loud..."

      Dear Sir/Madam,

      The article's assertion is not that senior management are not to blame, but rather that if you have the misfortune to work for said management and do not wish to be part of the forthcoming inevitable redundancies, then you would be well advised to refrain from pointing out that they are to blame.

      Any right-thinking person would agree that senior management must be accountable for their decisions, and anyone with much industry experience would probably agree that RBS senior management made a number of spectacularly rash decisions with regard to their IT. The author does not seek to dispute this; he merely offers advice to people who wish to keep their job when all about them are losing theirs.

  42. James 47

    The news is now reporting that the issue was caused in Edinburgh. Can any RBS internals shed some light?

  43. Fatman

    Senior RBS Management

    Needs to accept responsibility for this clustefuck, no matter how it happened.

    May I suggest that they express this `acceptance of responsibility` by committing Seppuku (details here: http: // )

    From the cited Wikipedia article:

    'Part of the samurai bushido honor code, seppuku was either used voluntarily by samurai to die with honor rather than fall into the hands of their enemies (and likely suffer torture), or as a form of capital punishment for samurai who had committed serious offenses, or performed for other reasons that had brought shame to them.`

    They might achieve `redemption` by appearing as a group in a public location, and simultaneously committing seppuku. Then, and only then, may I feel "sorry" for them.

  44. jaycee331

    !Thanks Dominic

    For another article breeding all that is wrong in the IT industry. These kind of truths should be published from a perspective of highlighting and stamping out such negative behaviours, not promoting them. Truely shameful. Every person who buys into this unethical tripe is just another a-hole I have to work around in my daily grind. Thanks for the dis-service and damage to my profession.

    1. Jeff 11

      Re: !Thanks Dominic

      Did you even read the article? This has got nothing to do with unethically avoiding responsibility - because a fuckup this severe fundamentally cannot be the fault of even a small group of IT staff - and everything to do with avoiding the BLAME, which lies, in my mind, *completely* with the top levels of management in the bank for making this scenario possible - but this buck will of course will be passed down to IT.

  45. Russ Williams

    Another classic article from Dominic

    Absolutely love his irreverent style... but the real gem is always all the butthurt comments showing such staggering levels of naïveté that you have to wonder how some of these people tie their own shoelaces.

    Yes, it's all senior management's fault. So what? You think, as a tech monkey (2nd class) pointing that out you're going to get Stephen Hester's head on a platter?

    Probably the best general defence with this sort of situation is a solid reputation of competence. Do the things Dominic suggests. All of the time. Make sure people have heard of you. If your company has technical mailing lists/forums, post to them and help answer questions. Give people credit (and CC their boss) when they do things that help you. Nobody will remember why they've heard of you, but they _will_ recognise your name, and that's handy when promotions and comp are being discussed. Make your boss look good by _doing your job well_ and giving him plenty of warning if something's going wrong. This isn't rocket science.

    When things do go wrong, it may be fun to sit in the corner, making snarky comments and apportioning blame, but you weren't hired for your comedy skills. You were hired to make the computers work, and they don't, so you're already in deep shit.

    1. mark daly

      Re: Another classic article from Dominic

      Sadly for current corporate managements the assumption that one can avoid the consequences of ones actions for ever and ever was also prevalent amongst French aristocrats in 1789, and we know what happened to them. I get the feeling the mob are starting to tire of constantly being presented with the execution of minor functionaries every time ta major corporate ship runs aground. They now want to see some toffs heads stuck on pikes.

  46. Ted Treen
    Thumb Up

    There are comments describing DC's style as "Smug", "Cynical", etc. etc.

    Possibly with a degree of justification, but if he's smug it's because he's been there, done that, got the T-shirt - AND survived.

    If he's cynical, it's because he's realistic and pragmatic, rather than being wrapped up in some touchy-feely "let's all be buddies" philosophy - which would (commecially speaking) be somewhat akin to painting a target on your back.

    Experience is worth more than theory: Cynicism IS realism...

  47. Anonymous Coward
    Anonymous Coward

    Very good...

    Unpalatable as the truth is, this article is pretty damn close.

  48. Anonymous Coward
    Anonymous Coward

    Sir Ian Blair?

    What? Fired into the House of Lords?

    That's funny. I thought he was fired for allowing the police to raid Tory Damian Green's room, and to have him arrested because a decaying Labour government was trying to have opposition MPs jailed, as part of a Stalinist attempt to stay in power, more appropriate to Uganda, than Britain.

    And I then thought he jumped straight into the House of Lords because he'd publically admitted taping Tony Blair and the then Attorney General's private conversations with himself, and has/has proof that Tony had lied to Parliament.

    Of course, that's only what I think. I'm not saying I am right, but it is what I think.

  49. Keith 12


    Probably one of the best articles I've read at The Reg.

    It's simple stuff really - but we've all seen so many people get it wrong on their way down the pan.....

  50. Anonymous Coward
    Anonymous Coward

    "Blaming the Indians - They’re cheap and foreign, so Unite and other unions are quite happy for them to carry the can."

    Interesting remark, as it was being reported last night that the fault seems to have originated in that well-known Indian province of Edinburgh.

    "Aftershocks - during this week something else bad will happen. I put that at about 80 per cent during this week, with about a 40 to 50 per cent chance that the consequences will leak out of RBS"

    Another interesting statement. Today RBS felt the need to advertise loudly that, just like Barclays' ongoing corrupt trader scandals, they have had to sack a load of their traders due to them conspiring to alter inter-bank interest rates. However, these people were sacked last year! Why are they feeling the need to make a loud, public apology now? Could it be to leech from Barclays' exposure to divert attention from people looking for "aftershocks" from the current RBS computer glitch?

    If so, they'd be admitting old news that's already done and buried rather than something that hasn't yet been proven true (i.e., more problems due to aftershocks)

  51. Big_Boomer

    The Blame Game

    Oh what fun!

    Politics, human interaction, soft skills,...

    All the things that most of us got into IT because we were crap at and/or despised.

    Personally I found that the best thing to do is work for an SME.

    These days your job longevity is longer than at larger companies, pay is comparable and you rarely have to deal with incompetents.

    Given a choice my days of working for large companies are over.

  52. ElNumbre

    Written by Scott Adams?

    This article reads like a Dilbert book: "leadership is nature's way of removing morons from the productive flow" and also the Peter Principle: "employees tend to rise to their level of incompetence".

    I just hope that RBS don't decide to sack the people who got the problems resolved as they will have certainly learned a lot over the past couple of weeks - it would be a shame to send those lessons to the job centre.

  53. Anonymous Coward
    Anonymous Coward


    Spot on. RBS today admitted that some customers are seeing direct debits being deducted twice from their accounts.

  54. Anonymous Coward
    Anonymous Coward

    Turkeys Singing "Deck the Halls ?"

    Whilst there is undoubted resonance in the article for me , having worked at Big Blue Chips and indeed RBS for some time, I'm uncomfortable with the lack of a challange to managers and management techniques. Surely this is the the loathed "fix" of rebooting and getting on with it. The fact that there are vast swathes of management that profer form over function, PR over production etc , is at the heart of MANY of ALL organisations problems. Unless somebody shouts that these "emporers" have no cloths nothing can change, it has not always been thus. Management used to be knowledgable and respected because they knew the business and their areas of responsibility. They progressed because of what they actually achieved and could offer, rather than what they've fabricated and shouted about. That no longer happens. In many organisation Managers very few managers have either Respect or Knowledge, those orgs are a House of Cards. They WILL fall, it's jus ta matter of when. But hey , play by the "rules" provided and it'll be a LITTLE longer before you are directly impacted by it all ... Enjoy

This topic is closed for new posts.

Other stories you might like