back to article So, that vast IT disaster you may have caused? Come in, sit down

I have a little sideline in interrogating IT professionals who are suspected of doing bad things. Sometimes it is quite hard to objectively tell the difference between incompetence and malice. In fact it is rare that either are the root cause of the worst screw-ups. The most dangerous techie in your firm is not the disaffected …


This topic is closed for new posts.


        1. Anonymous Coward
          Anonymous Coward

          Re: wow, who the hell . . . . .

          "....the ability to recognize those days when you shouldn't type "rm -rf *" for any reason whatsoever"

          It is exactly to avoid that sort of inattentive moments that I'm always logged in as root all the time. Keeps you on your toes.

          I mean, I learnt this from a BOFH article--what could possibly go wrong?

          1. proto-robbie

            Re: wow, who the hell . . . . .

            rm -Rf /

          2. Pirate Dave Silver badge

            Re: BOFH article

            Of all the funny BOFH articles I've read, one of Simon's quotes that's always stuck in my head - "root IS my account."

            Pretty much sums it up...

      1. Bernard M. Orwell

        Re: wow, who the hell . . . . .

        Yeah, the other techies and I make mistakes here from time to time; it happens...

        Problem is that our customer doesn't think we should make mistakes, and will not accept anything less than 100% performance 100% of the time. We always used to deal with that unreasonable expectation...

        ....then our new parent company moved in. Seems the management/directors that we now work for also think that so much as ONE failure in a year is a reason to go on the warpath...

        They're going to be left with no one who can even change a toner cartridge at this rate.

        Feet are for voting sometimes.

        1. Dominic Connor, Quant Headhunter

          Re: wow, who the hell . . . . .

          @Bernard, that's another reason for my soft line in these cases. If you say "failure==death" then people will only attempt things that can't fail. That means only tinkering with systems in modest ways and talking no risks which leads to over specifying hardware and dressing up minor changes as big initiatives.

          also of course harsh "security" lockdowns, since everything has a risk, you block everything that hasn't been signed off by by a security manager and a line manager is blocked, deleted and banned.

          Also as you say good people leave, but since the culture is basically "all IT people are crap" they won't know which are the good and which are not so useful.

    1. ReggiePerrin

      Re: wow, who the hell . . . . .

      in the real world stuff like this does happen... you still stuck in the matrix?

    2. Anonymous Coward
      Anonymous Coward

      Re: wow, who the hell . . . . .

      There are two types of people, Neon. There are all those chumps that make mistakes, and there's me, who also makes mistakes.

      (Though on a good day I spot them before anyone else, and fix them without telling anyone.)

      To all those people who make mistakes. How crap are you? I'd give up work were I that bad.


    3. Anonymous Coward
      Anonymous Coward

      Re: wow, who the hell . . . . .

      "The good boss hires only good people. Here we are hiring family and friends, wannabees or cheap slaves. No respect for the true value of the human resource.

      RBS's cockup is the result of the lack of basic knowledge. In the real IT world things like this do not happen."

      RBS's cock-up is not the scope of the article. Go comment on an RBS article of you want to drag that into it.

      And in real IT world, sh!t like this *does* happen. Sh!t happens because people are tired, rushed, miss a line on the screen, suffer cognitive bias or blindness, and a hundred other reasons.

      A tech who has never taken down a production system and packs the attitude that he never will because he's 'too good' is a dangerous liability, to my eyes.

    4. relpy

      Re: wow, who the hell . . . . .

      Don't be too hard on the poor little mite, we all thought we knew everything once.

    5. Anonymous Coward
      Anonymous Coward

      Re: wow, who the hell . . . . .

      >> "the once who know what they are doing"

      You once knew what you were doing? Congratulations!

      What do you know now?

    6. Danwold

      Re: wow, who the hell . . . . .

      I hardly ever make mistakes, in fact I've only ever made one mistake in my entire life. That was one time when I though I'd made a mistake, but I hadn't.

    7. Severen

      Re: wow, who the hell . . . . .

      "(the once who know what they are doing)"

      Oh, reeeeally? :-D

  1. 1Rafayal

    scaringly interesting

    The author almost sounds like a Mafia fixit man, but in an IT context.

    Would be very interested to find out how someone gets into this line of work

    1. Aaron Em

      Re: scaringly interesting

      He pretty much is, it seems like. I don't know how you get into that line of work, but I'm damn glad someone in it is taking the time to spill secrets to the hoi polloi in the trenches, so we'll at least be able to make it less likely that the bullet we catch comes from the side of the line we weren't even looking at...

      1. AdamWill

        Re: scaringly interesting

        The existence of people analogous to Mr. Connor in previous centuries is one of the main reasons our less ideologically brainwashed forebears formed unions...imagine if the IT staff in the financial sector got organized, you think they'd have to put up with having the thumbscrews applied by over-compensated outside consultants then? Fat chance.

    2. Lockwood

      Re: scaringly interesting

      Don't piss this guy off. If you do, you'll wake up with a floppy drive motor on your pillow, and if you aren't careful, you'll have an accident where you were walking home over a canal bridge and accidentally slipped, in the process of that, accidentally getting your feet caught up in old PC chassis, and accidentally binding your wrists with some CAT5e that you happened to be carrying.

      1. 1Rafayal

        Re: scaringly interesting

        I am deadly serious when I say I would really be interested in working in this area of IT.

        It would be kind of like a 3D puzzle mashed in with IT and elements of Midsomer Murders.

        I am constantly trying to pry the truth out of devs at work when the build mysteriously breaks...

    3. Anonymous Coward
      Anonymous Coward

      Re: scaringly interesting

      Ah but to negate his questions to you...

      Smile at him and say

      Nice car you arrived in.

      I didn't know you lived there, I know the place well.

      Do you have family?

      A friend of mine came across you recently, I'll have to tell him you're here.

      Now what was that question you wanted to ask me.

      He sounds a little to sure of himself that could be to your advantage so,

      Deny nothing

      Admit nothing

      Never elaborate on an answer to a question.

      Remain deadpan

      Raise eyebrow slight smile shake head look exasperated at the question if accused but don't speak

      Answer question with a question, in good politician style.

      If under work pressure when incident occurred point to line manager and say 'Mr .... said that it needed... Isn't the right Mr .....' deflecting the causation.

      Keep cool, never argue.

      Mention employment tribunal as a passing comment.

      1. Lockwood

        Re: scaringly interesting

        "Who is responsible for the design of the Frobulator?"

        "I am."

        "So you're saying that it IS your responsibility if it fails?"

        "Yes. And no."

        "Then who's fault is it?"

        "The attribution of fault cannot possibly fall unto just one man, any actions or omissions must have the blame or credit apportioned equally amonsgt the team"

        "So part of the blame then must fall to you?"

        "Whilst at first read it may seem that way, the blame is shared amongst the members of the team, not the leader of the team."

        "And the credit?"

        "That's all mine."

        "And you think that's a good way to operate?"

        "Yes, Investigator."

  2. James 132

    Prentice Hall's lovely 'Unix and Linux System Administration Handbook' remains the only tech helper I've read that mentions politics - it is an important issue for techs to understand. It is as real and important as any work they will do, and horrible as it is engineers by nature do not tend to understand these things and so end up getting crapped on.

  3. Anonymous Coward

    Management foibles

    After a major mistake why should management get together to brain-storm an approach to a solution when they can have a blame-storm instead!

    It comes with the added advantage that they don't require any competence in the technology being discussed.

    1. Danny 14

      Re: Management foibles

      shareholders like heads to roll. Those in power prefer to blame other people outside of their sphere so that they can keep their jobs. The game of houses.

  4. Anonymous Coward
    Anonymous Coward

    The author

    clearly thinks far too much of himself, his reputation and his abilities. What a terribly boring read this was.

    1. Anonymous Coward
      Anonymous Coward

      Re: The author

      Sometimes he gets it right. I quite like his expose's of the IT recruitment industry.

    2. Aaron Em

      Coming from the faceless

      that doesn't mean a hell of a lot, does it? I doubt his clients agree, for one thing.

      You know, I'm really at a loss to know what the Guy Fawkes icon is even for, save to proclaim: "I don't know what the hell I'm talking about, and you may safely ignore me." About one time in four hundred, someone uses it to say something useful that they'd get in a lot of trouble over if it could be traced back to them; the rest of it is laughable noise, on the strength of which I assume most Guy Fawkes users are juuust this side of shameless -- not enough that they button their lips rather than exposing their ignorance, but just barely enough so that they choose not to embarrass themselves in a way for which they're responsible. Penny Arcade figured this one out years ago; I continue to wonder why the eldritch dogs in charge of the Reg comments haven't yet spotted the same...

      ...meanwhile, I eagerly await the five hundred anonymous idiots whom I'm certain will shortly come along and prove me right.

      1. Anonymous Coward
        Anonymous Coward

        Re: Coming from the faceless

        In my case it's because it's someone else's login. See, I'm honest. If I state my name here, it's about as useless to anyone as yours, but what I very occasionally say might be relevant. I do mean *very* occasionally. Most of the time save 5 or so, someone else has already said it beautifully.

      2. Anonymous Coward
        Anonymous Coward

        Re: Coming from the faceless

        I prefer faceless to the wearing of many faces.

        Each identity you assume on any given site is effectively a lie.

        The only time this doesn't apply is if you have a rare name which you use consistently. Google returns "about 128,000,000" hits for "Aaron Em".

        1. TeeCee Gold badge

          Re: Coming from the faceless

          Yes, but you're not posting on Google, you're posting here where there is only one "Aaron Em", one "TeeCee", etc, etc.

          As I see it, posting here anonymously sends one of two messages. Either "I don't want anyone to know who said this, 'cos it's dynamite and could land me deep in the shit" or "This is utter bollocks and I don't want my account name to become forever associated with the word 'plonker' in the minds of other commentards". Which were you going for?

          1. Anonymous Coward
            Anonymous Coward

            Re: Coming from the faceless

            There's also the factor of 'I want to say something that a large number of people will find informative/insightful/witty, but there will probably be one or two ignorant morons who would persist in persecution of my good name because they don't agree with my opinion or are too entrenched in their own ignorance to see that they might not be correct or there might be other views'.

    3. Anonymous Coward
      Anonymous Coward

      Re: The author

      I agree completely, it could have been an interesting (yet content light/free) article but instead my lasting impression is of one of those washed up 80's actors who keeps saying "you must know me I was in movies" when they're trying to get a table in a restaurant.

    4. mike panero

      Re: The author

      Yeah it did read like one finger up his arse lipstick on his wrist

  5. Dominic Connor, Quant Headhunter
    Thumb Up

    Dominic Responds

    @Christoph, I agree absolutely the if you push people to hard, the will do things of apocalyptic stupidity.

    @umacf24, I'm not a great fan of recording since I'm after truth not prosecution, they are quite different disciplines. Tell someone they are being recorded and they may well clam up, don't tell them and there's legal crap to deal with, neither make the job easier.

    @Annhilator, As it happens I usually do this at banks and some things audit is the right function to deal with it, but I deal with situations where you need to understand the technology and the technologist, helps that I know banking as well. However the idea of the FSA doing this is just laughable and in any case management at the firm (whether financial or not) really want to know the truth before regulators or law enforcement get involved.

    @1Rafayal Oddly I have no idea how you'd get into this line of work, it's not even a job, just something I do when people ask me. I got into it because some people who know my work and who trust me in spite of being a City headhunter asked for help.

    1. Oninoshiko

      Re: Dominic Responds

      If you interviewed me, *I* would demand recording.

      If there is not a recording it's your memory. I'd rather not be subject to that.

      The recording doesn't protect YOU it protects ME. I'm not too suprised you're not a fan (no offence, I wouldn't be if I where on the other side of this either)

    2. amanfromMars 1 Silver badge

      Re: Dominic Responds

      You do realise I am supposing, without me having to ask, Dominic Connor, Quant Headhunter, that whenever it is folk just like yourself and who would be a few steps ahead of the Great Game, that dictate with impunity and glorious immunity, global direction? Make a wrong bet and take a wrong step though and the world crashes in around you, unless you have dirty little secrets to tell involving others, and then you can walk away with a substantial package which no one will try to ensure is not paid in full and quickly.

      And thanks for that excellent article which so many here appear to have understood exceedingly well, and that doesn't bode well for the dodgy casino and money-laundering banking sector.

  6. Wensleydale Cheese

    To err is human...

    From the article:

    "One thing that non-IT types find hard to grasp is that the scale of the consequences and the cause don’t correlate at all - although RBS management probably gets that idea now."

    As I was taught in my first IT job:

    "To err is human. For a really big cockup you need a computer."

  7. localzuk

    Everyone makes mistakes...

    Anyone who thinks they've had a career or will have a career in IT without making any mistakes is very much mistaken...

    Everyone errs sometimes, the issue that managers need to sort out is whether these mistakes are 'oops, why did I do that? Must've had a bad head on' types or 'I have no idea what I did there' ones.

    The prior are forgivable, the latter indicate that the person should be nowhere near that system.

    I work in schools, and there is a weird mix of amazingly talented IT people and people who simply shouldn't be allowed near a PC in that environment. If schools used people like Dominic, they'd be well employed!!

    1. PsychicMonkey
      Thumb Up

      Re: Everyone makes mistakes...

      and the talented ones move on very quickly. I've always thought that if an IT tech has been working at a school for a reasonable amount of time they can't be that good....

      Doesn't cover everyone of course, but as a rule of thumb....

    2. Vanir

      Re: Everyone makes mistakes...

      My mistake was exchanging a career as an electrician / technician for one as a programmer / software engineer.

  8. Pete 2 Silver badge

    For the most monumental screwups ...

    don't focus on the sysadmins (competent, incompetent, overworked, lazy or malicious). Instead look at the system designers. Ultimately they are the ones who make the biggest, most expensive, longest lasting cockups imaginable - and some that extend a long, long way beyond what anyone thought was the limit of human stupidity.

    The problem with trying to point the finger at the designers is that by the time the scale of their errors is known, it's all far too late. The systems go live, despite everyone knowing that they're utterly doomed. The processes needed to use and maintain them are complicated, error prone, people-intensive and unreliable. However the blame is never passed to those who created the shambles, it's always attributed to the person who pressed the badly designed button.

    1. Anonymous Coward
      Anonymous Coward

      Re: For the most monumental screwups ...

      Meh. I've seen designers put out some absolutely amazing work, then after every sr. mgmt has taken a shit on it, the result that gets implemented is totally unrecognizable as being anything related to the original design.

  9. LordHighFixer

    The truth, the whole truth, and something more than the truth.....

    I have found that if you come out and tell the whole and complete truth, to the person who will ultimately be making the decisions about your fate, and put a slight spin on it on how this was ultimately the best thing that could have happened (best to catch these flaws now, timely manner, could have been worse if, etc, etc). And do this over a couple of beers at the local pub, behind the scenes, you will never talk to the "investigator".....

  10. Anonymous Coward
    Anonymous Coward

    Any company that brings someone in to do "forensic interviews"...

    ...deserves everything it gets from IT disasters. You don't need a consultant who writes vaguely smug articles for the Register to identify the root cause of an incident, you need a proper root cause analysis function who know what questions to ask, and how. What a ridiculous way you make a living.

    Also, what a splendid way to instill a culture of blame & blame avoidance!

    1. Anonymous Coward
      Anonymous Coward

      Re: Any company that brings someone in to do "forensic interviews"...

      That and try not to give all the keys to one person? Passwording out the management from their own computers etc.

  11. Fred M

    " In my experience good techies are really uncomfortable with actually lying"

    In my experience of RBS staff in India, they have absolutely no problem at all with lying. Lying and not getting caught seemed to be the complete moral equivalent of telling the truth,

    1. Lockwood

      I can be upgradings your accountings to a grade that is belong your current gradings

    2. Anonymous Coward
      Anonymous Coward

      The reason for that, in my experience, is that if somebody over there is found to have made a mistake while honestly trying to do what they thought was right, and in the interest of the customer, they fire them. No doubt after a "forensic interview" by some headhunter consultant.

      As opposed to the frankly MENTAL approach of training them so that they understand why what they did caused an incident, making them provide more value to the business because they're better at their job.

      Much better to get rid of them and replace with someone who has a degree but literally no idea about your services and business processes.

    3. Guus Leeuw

      Re: Fred M

      Dear Fred,

      lying to somebody who is sitting next to you and lying to somebody on the phone who is actually half a world away, are two completely different skillsets.

      I am very happy praising people in mails, telephone calls, and the likes. Never, though, when people can see where I'm looking, or when my face or hands are visible to the listener.

      I loved your post, and was glad to have had the chance to respond to it.


      Guus L

    4. Anonymous Coward
      Anonymous Coward

      Oh come on... They are Indians. Have you lived in India for a while?

  12. Dominic Connor, Quant Headhunter

    More replies from Dominic

    @Anonymous Coward: I am not "vaguely smug", I am arrogant, not the same thing at all. Yes, you do need a proper root cause analysis, sadly the set of firms who have this is very much smaller than the set who need it.

    @LordHighFixer: I agree that for the vast majority of screwups a quite word over a pint is by far the best way. However there comes a point where the people who want answers are so far away or high up in the management orgchart that you may only find out they exist when the fan has already been hit.

    @pete2. Agree the designers never get the kicking they deserve. From my experience the absolute best recipe for a disaster is to get a firm of consultants in. Management will get some pretty diagrams in whatever methodology is fashionable a nice trip to a golf course and an unworkable system.

    @Fredm: I did say *good* techies don't like lying.. The evidence seems to be that the RBS Indian staff were being paid low wages even by Indian IT standards. As for India in general, my current favourite is an Indian Dell tech supervisor who claimed to be Dell's Head of Compliance. Yes really. I had told them I wrote for the Register and they still lied to me, pretended that all managers everywhere in Dell were in a meeting, hung up on me, etc.

    I put this down to an efficient quality control system at Dell and RBS. Their managers want things to happen a certain way and select staff who seem to genuinely believe things that no normal person would entertain.

    1. Kevin Turvey

      Re: More replies from Dominic

      a quiet word over a pint


This topic is closed for new posts.