So, that vast IT disaster you may have caused? Come in, sit down I have a little sideline in interrogating IT professionals who are suspected of doing bad things. Sometimes it is quite hard to objectively tell the difference between incompetence and malice. In fact it is rare that either are the root cause of the worst screw-ups. The most dangerous techie in your firm is not the disaffected …
"The problems are being caused by him or her trying too hard"
With the boss standing right behind yelling "WHY ISN'T IT FIXED YET? HOW LONG IS IT GOING TO TAKE? THE CUSTOMERS CAN'T ACCESS THE SYSTEM! HURRY UP. AND GET IT FIXED! WHY ISN'T IT FIXED YET?
Odd that sometimes they can' concentrate properly.
Cool under pressure is the name of the game. When I used to work hard, I used to play hard. Repairing financial systems with branch managers wanting their systems back up and running. Some were better than others.
A few years ago, I exchanged the pressure for less money. Net result; probably less liver damage in the long haul.
Money isn't cheap.
Standard answers to this sort of interefering, no matter who it is from
>WHY ISN'T IT FIXED YET? - Some idiot keeps interrupting me - once you've said that to your bosses boss a few times and you're still around you know you're valued by someone even higher up, however you still won't get a pay rise.
>HOW LONG IS IT GOING TO TAKE? - This is my favourite question and my stock answer is, a lot longer if people keep asking how long is it going to take. It won't stop them aking
>THE CUSTOMERS CAN'T ACCESS THE SYSTEM - And? As if I didn't know and even more presumptious, as if I cared.
>HURRY UP. AND GET IT FIXED! - I take a breather, the quickest surest way to get me to slow down is to tell me to get a move on and the quickest way to get me to stop is to look over my shoulder. No spectators, if someone is going to look over my shoulder they can do the work while I do something more entertaining.
And yes, I do use these responses, I also unplug the phone, what's the worst that can happen? They fire me, as if I'd be upset. Panicking and dropping your trousers never solved anything.
One day I was fixing a problem (don't remember what, and doesn't matter). And this guy was "Go, go, go! Fix it already! I can't wait much longer! Hurry, hurry, hurry!"
Then I looked to him, and said: "Slowly is faster."
I could see him changing colors... :D And 5 minutes later the problem was gone.
This post has been deleted by its author
I never had to work in the financial sector, but I did work at the University hospital. Whenever one particular senior staff member (non-computer science) tried to meddle in my work and ask what was going on (with an implication that things weren't going fast enough), I tended to explain in detail what I was doing, with as much mathematical jargon as I could muster (which is a lot when you are doing image analysis), in a friendly, fellow-scientist-to-fellow-scientist tone (i.e., I do not have to explain every concept in detail, do I).
The first mention of "fast Fourier transform" or "optimal convolution kernel" was generally enough to make them run for cover.
A couple of times I've been roped into fixing computers for friends, who then proceed to hover nearby and do an impression of Stan Laurel just after he's done something untoward to Oliver Hardy while you're trying to think. Obviously you can't just tell them to go away, but if you ask them to put the kettle on it gives you a few minutes to get started. When they see you with your head down getting on with things they tend to leave you alone.
"A couple of times I've been roped into fixing computers for friends, who then proceed to hover nearby and do an impression of Stan Laurel just after he's done something untoward to Oliver Hardy while you're trying to think. Obviously you can't just tell them to go away, but if you ask them to put the kettle on it gives you a few minutes to get started. When they see you with your head down getting on with things they tend to leave you alone."
If only this would work on the IT Director!
Although I suppose you might get some respite, while he's recovering from the shock...
I suppose there's always the BOFH option? Cattle-prod, and lock them in the tape safe.
In some countries it's perfectly legal to record a conversation you're part of, even without other parties knowing about it. In Finland, for example, where it was actually confirmed by the (Finnish) Supreme Court some years back. And yes, obviously such a recording would be usable as evidence.
My manager won't allow me in certain meetings anymore, especially those involving those above him because I refuse to lie.
My Nan, the most passive and gentle soul on the planet went into one, once and only once because I lied to her. Watching her losing it frightened the living shit out of me, I find it incredibly hard to lie outright.
You lie and people find out later on they're more pissed off than if you just said in the first place, "Sorry I fucked it up and I'm fixing it, OK!". Bending the truth within the context of what I have to work with, yes I can do that but lying I can't do, I just can't pull off the required poker face. I will forever be a techie on the factory floor, no giddy heights of management for me, that suits me just fine.
I have had at least two managers who have suggested that any communication should go via their good selves. The usual phrase is 'The Team should provide a consistent message.....'. In my experience this is the time to get a new job. Not quite sure if this the same sort of situation you are describing above Mr AC.
I did an internship many moons ago where I was tasked with reviewing use and integration of a minor piece of in-house, front-end software at an international bank..
After asking me to re-write and expand the report multiple times to include ALL the problems AND solutions, the bumbling idiot shifted it straight up the command chain to head of IT without proofing it.
I get called up, a personal letter of recommendation from the head, and offered a job when I graduate. I have a meeting with the bumbling idiot where I am told that "In these reports, one shouldn't be too blunt or it could cause problems."
He was fired within the quarter.
Some people need to protect from themselves.
Reminds me of a time when I was asked to do some work for a 3rd party company. It was checked and re-checked for bugs and was then sent to management to be sent on. Not long after a red-faced manager comes steaming in screaming about how he'd sent it to the customer and it didn't work and how it was an embarrassment.
There was a config issue relating to the clients computer, I'd fixed it in minutes.
It wasn't until I read the email trail that I found out why he took it so personally.
Oh yes, he said he'd written it personally. When the (techie) client asked him to fix it, he got monumentally rumbled.
Heh, had that a few times. Funniest was when sent out on deployment for another department. Had my programming boss, the professional services boss in the UK, the foreign department boss, boss of the company, and various leeches throughout the company all demanding that I send THEM the daily diary and that THEY would be the one to use it to bash the others with. I spoke to the mail admin and asked him to set up a dailydiary@comp... address, and that I'll send the reports to that and that only. Then the managers can all argue amongst themselves who should get access to that list.
Wasn't long before I had the emails telling me who wasn't considered important in the company as they were asking for the diary before I sent it to the agreed address, just to check it was factually correct.
Bunch of donkeys. 2 mangers+ for every actual worker, with many people reporting to more than one manager. Wasn't too obvious how badly it all ended up.
Not necessarily. I've done a fair bit of customer-facing work in the past, and one thing I've learnt is that significant communications (i.e. relating to deliverable content, timescale or cost) should be written, read again and fixed, and then read by someone else and fixed. A three-paragraph email may take an hour to write and send. This might seem ridiculously wasteful, but believe me, it's nothing compared to the cost of dealing with the fallout when you find that your badly-written email has led to a high-level flamewar.
As lots of people have mentioned, engineers aren't always switched on to the implications of what they say. Compare and contrast emails saying "we knew about this bug but we didn't fix it" and "we knew about this bug, we discussed it with your liaison Mr Jones, and collectively we decided it was a low-risk issue compared to feature X (see meeting minutes attached)". Eventually the first email will probably get all the relevant information out and the flames will die down, but you really don't want the grief involved.
Nor are engineers often very savvy about committing their company to things. An engineer's typical response is "sure we can do that", because technically they *can*. But it'll take an extra month of work, the customer's deadline is in six weeks time, there's three weeks of acceptance tests to run, and you don't have budget for that extra work anyway. Do you fancy doing unpaid work for your company?
should you tell the truth if you think it can be skewed by those who receive the report? That 5% malicious may be enough for them to say "its possible it was intentional" and bin you and hide the report (seen it done to others, been put on report myself using a similar tactic).
Dominic, the thrust of your recent articles seems to show how technicians have a difficult time with politics. Having also read "the way of the rat" by Joep Schrivers, I think there is a market for a "how it works" book about politics for techies. Fancy bashing one out?
The thought of having BCS representing me in any kind of official capacity terrifies me far more than the prospect of losing my job.
IT does run the world, it is a shame we cannot fully unionize. Bob Crow and the RMT think that tube drivers control London, imagine if all the techies went on strike.
There's no cure for mild autism.
That's harsh, but the vast majority of good techies genuinely have seriously under-developed social skills. They often also have this blithe attitude towards politics and are completely happy to call lesser mortals (who y'know.. PAY THEM...) idiots to their faces; which is never helpful.
That's why sensible managers don't let the techs go to meetings without an escort... or at all, if can be helped. You'll invariably find that the one extrovert tech with any skill in diplomacy in the department eventually gets saddled with every Change/client/crash meeting, which is the best solution for everyone. The techs who hate everyone can stay tucked away doing what they do best, and nobody gets hurt.
I'm not sure a lot can be done to help, in many cases. Learning to keep quiet and smile is probably the best advice that can be given to a great number of people.
" Learning to keep quiet and smile is probably the best advice that can be given to a great number of people."
Funny you say that. I almost got an employee of the month for saying about two sentences in a client facing kick off meeting. However, one of my co-workers almost got terminated for dribbling on for thirty minutes in the same meeting.
"That's why sensible managers don't let the techs go to meetings without an escort... or at all, if can be helped."
This would be fine with most techies. The problems start when people decide that not having the borderline/ narcissistic personality disorder needed to play office politics well, is some kind of objective personality deficit that should be rectified by exposure
If "the truth" puts the root of this at deliberate malpractise by you, I don't have any sympathy. But if "the truth" means saying who told you to do what, and the shortcuts you were required to take on testing, and showing the list of defect reports you raised, you're a lot better off. If this also includes emails, then great.
Amongst techies, rats have a very short half-life-to-detection, but most techies (especially newbies) aren't so good at dealing with them. The only good defence against a rat is to block up ways in. If his moan of "this was 2 weeks late" CC'ing his manager is replied to with a "sorry, but you added on extra work for which I estimated 2.5 weeks (emails attached)", you're sound. You don't need to be a rat yourself, you just need to stop rats from getting to you. Call it anti-politics, if you like.
"“Interrogation” is of course exactly the wrong word and if I’m dealing with a firm that has handled this sort of situation before, one that is utterly forbidden."
Indeed, it's "Root Cause Analysis" but in this particular instance I suspect that PWC or similar will be in to do audit of how the change controls allowed this sort of c0ck up. There are FSA requirements around this sort of control, but more chilling for RBS is the fact that they're also an NYSE listed company and are subject to the rigours of Sarbanes-Oxley compliance, which makes for an even duller and intense process.
Looks like good advice. I would also suggest you make sure you are really prepared before heading into any meeting like this. Make sure you have copies of key e-mails, have througly reviewed any files before the meeting and have prepared answers to any of the questions which may be asked. Don't be fooled by the convivial atmosphere, anything you say here could form part of a later disciplinary.
As the tone of the article suggests, the aim of such an investigation is to assign blame. Usually there will be enough to go around and you just need to make sure it isn't disproportionately dumped on you.
This article appears to me as totally generic. What we (the once who know what they are doing) do, has nothing to do with luck!
The good boss hires only good people. Here we are hiring family and friends, wannabees or cheap slaves. No respect for the true value of the human resource.
RBS's cockup is the result of the lack of basic knowledge. In the real IT world things like this do not happen.
Neon
Ah, an 'I never make mistakes' man.
I'm glad you've got this far in your life without having that one day where you did something that 99.99% of the time you would never do but slipped up once because you'd had a fight with the wife/gf and couldn't concentrate, were ill but chose to work anyway, etc. When that day comes, you want to hope that your one mistake is on something trivial because people forget about the 99.99% of perfection the one time it's not.
Not suggesting that the RBS thing wasn't the result of using people without the right skills but you are generalising a lot. It's always someone else who knew much less than they should have. Until it's you.
Yep, one sign of "maturity", both as a person and as a computer nerd/professional, is the ability to recognize those days when you shouldn't type "rm -rf *" for any reason whatsoever. Days like that happen, and when they do, you're sure to fuck up in ways that will make you miserable. Better to realize it's gonna be a bad day before it becomes a Bad Day (tm), and avoid doing heavy-duty stuff. Make it LOOK like you're doing Stuff That Matters, but in actuality you're just testing all the various switches to "ls" and writing scripts to send email to yourself.
I like to think that I'm one of those who is above screwing things up - at least in production. I take the time to do it in staging... case in point - rebuilding vmdks on a system the other day. Had to be done as a rebuild in parallel and swap out of the drive via e2label etc. Got all done. DELETED THE WRONG BLOODY VMDK FILE... At least it was in staging but still took out a day of work and another day trying to coax the VM back to life
Ask the people here and they will tell you that I never fsck things up, know wall the obscure commands on *BSD,Linux,Windows, MacOS etc.. But truth be told even then you can still screw things up. Its just a matter of where you do it and who it impacts that keeps the depth of the crap down sometimes.
(Luckily not had a production version of this yet - but I know one day I wont be so lucky).
Or you could be the kind of person who thinks too hard about what might go wrong and how you could cock it up, even though 99.9% of the time you won't, and then you don't do anything, even though you can.
That's when you learn the trading floor ain't the right place and you move on, happy in the knowledge that those people who are earning more than you for being no better, do actually deserve it.
I wouldn't go back, but I'm very glad I was there for a while.
I'll get my coat, because I did.
"....the ability to recognize those days when you shouldn't type "rm -rf *" for any reason whatsoever"
It is exactly to avoid that sort of inattentive moments that I'm always logged in as root all the time. Keeps you on your toes.
I mean, I learnt this from a BOFH article--what could possibly go wrong?
Yeah, the other techies and I make mistakes here from time to time; it happens...
Problem is that our customer doesn't think we should make mistakes, and will not accept anything less than 100% performance 100% of the time. We always used to deal with that unreasonable expectation...
....then our new parent company moved in. Seems the management/directors that we now work for also think that so much as ONE failure in a year is a reason to go on the warpath...
They're going to be left with no one who can even change a toner cartridge at this rate.
Feet are for voting sometimes.
@Bernard, that's another reason for my soft line in these cases. If you say "failure==death" then people will only attempt things that can't fail. That means only tinkering with systems in modest ways and talking no risks which leads to over specifying hardware and dressing up minor changes as big initiatives.
also of course harsh "security" lockdowns, since everything has a risk, you block everything that hasn't been signed off by by a security manager and a line manager is blocked, deleted and banned.
Also as you say good people leave, but since the culture is basically "all IT people are crap" they won't know which are the good and which are not so useful.
There are two types of people, Neon. There are all those chumps that make mistakes, and there's me, who also makes mistakes.
(Though on a good day I spot them before anyone else, and fix them without telling anyone.)
To all those people who make mistakes. How crap are you? I'd give up work were I that bad.
</irony>
"The good boss hires only good people. Here we are hiring family and friends, wannabees or cheap slaves. No respect for the true value of the human resource.
RBS's cockup is the result of the lack of basic knowledge. In the real IT world things like this do not happen."
RBS's cock-up is not the scope of the article. Go comment on an RBS article of you want to drag that into it.
And in real IT world, sh!t like this *does* happen. Sh!t happens because people are tired, rushed, miss a line on the screen, suffer cognitive bias or blindness, and a hundred other reasons.
A tech who has never taken down a production system and packs the attitude that he never will because he's 'too good' is a dangerous liability, to my eyes.
He pretty much is, it seems like. I don't know how you get into that line of work, but I'm damn glad someone in it is taking the time to spill secrets to the hoi polloi in the trenches, so we'll at least be able to make it less likely that the bullet we catch comes from the side of the line we weren't even looking at...
The existence of people analogous to Mr. Connor in previous centuries is one of the main reasons our less ideologically brainwashed forebears formed unions...imagine if the IT staff in the financial sector got organized, you think they'd have to put up with having the thumbscrews applied by over-compensated outside consultants then? Fat chance.
Don't piss this guy off. If you do, you'll wake up with a floppy drive motor on your pillow, and if you aren't careful, you'll have an accident where you were walking home over a canal bridge and accidentally slipped, in the process of that, accidentally getting your feet caught up in old PC chassis, and accidentally binding your wrists with some CAT5e that you happened to be carrying.
I am deadly serious when I say I would really be interested in working in this area of IT.
It would be kind of like a 3D puzzle mashed in with IT and elements of Midsomer Murders.
I am constantly trying to pry the truth out of devs at work when the build mysteriously breaks...
Ah but to negate his questions to you...
Smile at him and say
Nice car you arrived in.
I didn't know you lived there, I know the place well.
Do you have family?
A friend of mine came across you recently, I'll have to tell him you're here.
Now what was that question you wanted to ask me.
He sounds a little to sure of himself that could be to your advantage so,
Deny nothing
Admit nothing
Never elaborate on an answer to a question.
Remain deadpan
Raise eyebrow slight smile shake head look exasperated at the question if accused but don't speak
Answer question with a question, in good politician style.
If under work pressure when incident occurred point to line manager and say 'Mr .... said that it needed... Isn't the right Mr .....' deflecting the causation.
Keep cool, never argue.
Mention employment tribunal as a passing comment.
"Who is responsible for the design of the Frobulator?"
"I am."
"So you're saying that it IS your responsibility if it fails?"
"Yes. And no."
"Then who's fault is it?"
"The attribution of fault cannot possibly fall unto just one man, any actions or omissions must have the blame or credit apportioned equally amonsgt the team"
"So part of the blame then must fall to you?"
"Whilst at first read it may seem that way, the blame is shared amongst the members of the team, not the leader of the team."
"And the credit?"
"That's all mine."
"And you think that's a good way to operate?"
"Yes, Investigator."
Prentice Hall's lovely 'Unix and Linux System Administration Handbook' remains the only tech helper I've read that mentions politics - it is an important issue for techs to understand. It is as real and important as any work they will do, and horrible as it is engineers by nature do not tend to understand these things and so end up getting crapped on.
that doesn't mean a hell of a lot, does it? I doubt his clients agree, for one thing.
You know, I'm really at a loss to know what the Guy Fawkes icon is even for, save to proclaim: "I don't know what the hell I'm talking about, and you may safely ignore me." About one time in four hundred, someone uses it to say something useful that they'd get in a lot of trouble over if it could be traced back to them; the rest of it is laughable noise, on the strength of which I assume most Guy Fawkes users are juuust this side of shameless -- not enough that they button their lips rather than exposing their ignorance, but just barely enough so that they choose not to embarrass themselves in a way for which they're responsible. Penny Arcade figured this one out years ago; I continue to wonder why the eldritch dogs in charge of the Reg comments haven't yet spotted the same...
...meanwhile, I eagerly await the five hundred anonymous idiots whom I'm certain will shortly come along and prove me right.
In my case it's because it's someone else's login. See, I'm honest. If I state my name here, it's about as useless to anyone as yours, but what I very occasionally say might be relevant. I do mean *very* occasionally. Most of the time save 5 or so, someone else has already said it beautifully.
Yes, but you're not posting on Google, you're posting here where there is only one "Aaron Em", one "TeeCee", etc, etc.
As I see it, posting here anonymously sends one of two messages. Either "I don't want anyone to know who said this, 'cos it's dynamite and could land me deep in the shit" or "This is utter bollocks and I don't want my account name to become forever associated with the word 'plonker' in the minds of other commentards". Which were you going for?
There's also the factor of 'I want to say something that a large number of people will find informative/insightful/witty, but there will probably be one or two ignorant morons who would persist in persecution of my good name because they don't agree with my opinion or are too entrenched in their own ignorance to see that they might not be correct or there might be other views'.
@Christoph, I agree absolutely the if you push people to hard, the will do things of apocalyptic stupidity.
@umacf24, I'm not a great fan of recording since I'm after truth not prosecution, they are quite different disciplines. Tell someone they are being recorded and they may well clam up, don't tell them and there's legal crap to deal with, neither make the job easier.
@Annhilator, As it happens I usually do this at banks and some things audit is the right function to deal with it, but I deal with situations where you need to understand the technology and the technologist, helps that I know banking as well. However the idea of the FSA doing this is just laughable and in any case management at the firm (whether financial or not) really want to know the truth before regulators or law enforcement get involved.
@1Rafayal Oddly I have no idea how you'd get into this line of work, it's not even a job, just something I do when people ask me. I got into it because some people who know my work and who trust me in spite of being a City headhunter asked for help.
If you interviewed me, *I* would demand recording.
If there is not a recording it's your memory. I'd rather not be subject to that.
The recording doesn't protect YOU it protects ME. I'm not too suprised you're not a fan (no offence, I wouldn't be if I where on the other side of this either)
You do realise I am supposing, without me having to ask, Dominic Connor, Quant Headhunter, that whenever it is folk just like yourself and who would be a few steps ahead of the Great Game, that dictate with impunity and glorious immunity, global direction? Make a wrong bet and take a wrong step though and the world crashes in around you, unless you have dirty little secrets to tell involving others, and then you can walk away with a substantial package which no one will try to ensure is not paid in full and quickly.
And thanks for that excellent article which so many here appear to have understood exceedingly well, and that doesn't bode well for the dodgy casino and money-laundering banking sector.
From the article:
"One thing that non-IT types find hard to grasp is that the scale of the consequences and the cause don’t correlate at all - although RBS management probably gets that idea now."
As I was taught in my first IT job:
"To err is human. For a really big cockup you need a computer."
Anyone who thinks they've had a career or will have a career in IT without making any mistakes is very much mistaken...
Everyone errs sometimes, the issue that managers need to sort out is whether these mistakes are 'oops, why did I do that? Must've had a bad head on' types or 'I have no idea what I did there' ones.
The prior are forgivable, the latter indicate that the person should be nowhere near that system.
I work in schools, and there is a weird mix of amazingly talented IT people and people who simply shouldn't be allowed near a PC in that environment. If schools used people like Dominic, they'd be well employed!!
don't focus on the sysadmins (competent, incompetent, overworked, lazy or malicious). Instead look at the system designers. Ultimately they are the ones who make the biggest, most expensive, longest lasting cockups imaginable - and some that extend a long, long way beyond what anyone thought was the limit of human stupidity.
The problem with trying to point the finger at the designers is that by the time the scale of their errors is known, it's all far too late. The systems go live, despite everyone knowing that they're utterly doomed. The processes needed to use and maintain them are complicated, error prone, people-intensive and unreliable. However the blame is never passed to those who created the shambles, it's always attributed to the person who pressed the badly designed button.
I have found that if you come out and tell the whole and complete truth, to the person who will ultimately be making the decisions about your fate, and put a slight spin on it on how this was ultimately the best thing that could have happened (best to catch these flaws now, timely manner, could have been worse if, etc, etc). And do this over a couple of beers at the local pub, behind the scenes, you will never talk to the "investigator".....
...deserves everything it gets from IT disasters. You don't need a consultant who writes vaguely smug articles for the Register to identify the root cause of an incident, you need a proper root cause analysis function who know what questions to ask, and how. What a ridiculous way you make a living.
Also, what a splendid way to instill a culture of blame & blame avoidance!
The reason for that, in my experience, is that if somebody over there is found to have made a mistake while honestly trying to do what they thought was right, and in the interest of the customer, they fire them. No doubt after a "forensic interview" by some headhunter consultant.
As opposed to the frankly MENTAL approach of training them so that they understand why what they did caused an incident, making them provide more value to the business because they're better at their job.
Much better to get rid of them and replace with someone who has a degree but literally no idea about your services and business processes.
Dear Fred,
lying to somebody who is sitting next to you and lying to somebody on the phone who is actually half a world away, are two completely different skillsets.
I am very happy praising people in mails, telephone calls, and the likes. Never, though, when people can see where I'm looking, or when my face or hands are visible to the listener.
I loved your post, and was glad to have had the chance to respond to it.
Regards,
Guus L
@Anonymous Coward: I am not "vaguely smug", I am arrogant, not the same thing at all. Yes, you do need a proper root cause analysis, sadly the set of firms who have this is very much smaller than the set who need it.
@LordHighFixer: I agree that for the vast majority of screwups a quite word over a pint is by far the best way. However there comes a point where the people who want answers are so far away or high up in the management orgchart that you may only find out they exist when the fan has already been hit.
@pete2. Agree the designers never get the kicking they deserve. From my experience the absolute best recipe for a disaster is to get a firm of consultants in. Management will get some pretty diagrams in whatever methodology is fashionable a nice trip to a golf course and an unworkable system.
@Fredm: I did say *good* techies don't like lying.. The evidence seems to be that the RBS Indian staff were being paid low wages even by Indian IT standards. As for India in general, my current favourite is an Indian Dell tech supervisor who claimed to be Dell's Head of Compliance. Yes really. I had told them I wrote for the Register and they still lied to me, pretended that all managers everywhere in Dell were in a meeting, hung up on me, etc.
I put this down to an efficient quality control system at Dell and RBS. Their managers want things to happen a certain way and select staff who seem to genuinely believe things that no normal person would entertain.
If you want to see this sort of thing done well, go read on of the Marine Accident Investigation Board's accident reports (http://www.maib.gov.uk/home/index.cfm). The primary goal of these --- which is printed in large letters at the top of every report --- is to determine what happened and how to stop it happening again. Also --- and again, this is printed in large letters --- the reports are inadmissible as evidence in any judicial proceeding that's attempting to apportion blame.
The reports themselves are a miracle of brevity and clarity, stating clearly and simply what happened, what went wrong, and what needs fixing. They're also not shy about pointing out the mistakes people make, but the point is always that people always make mistakes, and the system needs to be designed to cope.
I recommend them to anybody who wants to see how an accident report should read. (I gather it's possible to subscribe to a tiny four-page flyer containing summaries of the accident reports; I've seen them, and they're fascinating reading, but have been unable to find this on their website.)
This post has been deleted by its author
did they even paid u from TheReg? Or did u paid them? :)
next time I look for more consultancy hours, I know how to handle my marketing...
p.s. I usually get involved just a minute before the shit hits the fan... to try to save the situation
...before the company gets to the point to call in a blame/fingerpointing guy like you (thing that usually doesn't happen)
Please Can We Have (PCWH) a test & QA system? - No Too Expensive (NTE),JFDI
PCWH Version control - Why waste time with that JFDI
PCWH have some training - NTE, JFDI
PCWH some budget for resilience - NTE, JFDI
PCWH a process to manage changes - NTE, no way you want us to specify what we want before you implement it? what do we pay you for?, JFDI
etc.
Oh yes we allowed them to interfere with your systems without telling you, after all they are corporate - why is it broken what did YOU do?
Oh we changed our minds, can you guess what we want and do it for tomorrow?
Oh that's not what we wanted, I know you wrote loads of specifications but we didn't read them we want something different - that one really gets me.
Now the times when I have convinced them that we need to do it properly all goes well. today for instance our SAN had a module replaced but the users never knew. four years of five 9's and counting!
> PCWH Version control - Why waste time with that JFDI
That one, at least, is easy to fix. You do JFDI. Installing Git or similar takes no time flat.
Some time later, the need for VCS becomes apparent, and much fretting occurs. That's the time to mention in passing that you might be able to ressurect some data from some random copies you made earlier. It'll take a few hours :-)
Vic.
Yo Dominic, my man! Any generously-bonussed shit-will-never-be-my-fault-if-it-goes-wrong too-well-paid-to-fail jobs going?
Or does "headhunter" on this occasion mean you have the contract to hunt down the RBS wrongdoers and display their heads on spikes on the roof of Canary Wharf, pour encourager les autres?
I can dream ...
Barclays were looking for someone for that kind of position but the job description just changed, things will still never be your fault but if they go really really bad and you're found out you'll be asked to leave, nicely mind you, and you'll still get a generous leaving package.
Now, all's it takes is a discussion of politics to make the terminal fuckwits come out and illuminate us with their wisdom, all comfortable behind the pseudo-anonymity of their Fawkes masks.
Nothing draws them as well as something they have no way of understanding, and politics is definitely one of those things.
Nice article, Dominic.
My advice is to find an intelligent manager that understands politics, and be direct and honest with them, and stick with them wherever they go. They know how to handle the inevitable screw-ups, as long as you don't EVER leave them hanging by being dishonest. In this case, the company almost never needs Dominic's help. I know enough about tech to know I am out of my depth and need to assemble more knowledgeable resources -- and it is no different when it comes to politics.
...as opposed to the lack of pseudo-anonymity that comes from using a name like "perlcat"?
What difference does it make if people are using AC, there's valid points being made here.
If a company uses this interrogation approach then all it will result in is
a) more and more lying after incidents because everybody's terrified they'll get fired for trying to do the right thing but cocking it up;
b) a culture of constant blame shift in a desperate attempt not to have a consultant finger one of your team, and
c) Dominic's pockets getting fatter and fatter.
I'm a major incident & problem manager and in our root cause reviews I don't give a shit which individual caused the incident, I just want to know what is being done by the relevant supplier/operational team/etc to ensure that a similar incident doesn't occur again. Whether that's tightening access rights, training, documentation, or whatever else. Blaming one person and shitting up the entire IT staff while you're working out who to blame doesn't help anybody. Except Dom obviously.
"I'm a major incident & problem manager and in our root cause reviews I don't give a shit which individual caused the incident"
*You* will not give a shit. Nobody should give a shit whose "fault" is was.
Unfortunately, and in spite of all your best intentions and any big red disclaimers you may put on your reports, somebody (often someone with a nagging sense of inadequacy) will end up using your finding to blame somebody.
This post has been deleted by its author
I have made plenty of cockups in my career, luckily none of them happened in an operating environement. Doing stupid stuff like DELETE * FROM CUSTOMER_DB ; WHERE CUSTOMER_ID = 5 ; (see the misplaced semi-quote, you just deleted the whole database) are much less likely to happen when you have two people, one to check the work and one to signoff. Errors get picked up BEFORE they impact the live environement, that is why you have senior developers and change management. They are there to gate keep the idiots under them.
Problems can occur for any number of reasons, but not checking what people do (ie slashing resources ) and given them root access to your o/s or god privilages on your database seems like the perfect way to end up in sticky position. Often the root cause isn't disaffection, malicious behaviour, its just fat fingers and the right (or wrong) user privilages.
I have only ever once come across one deliberate attempt at sabotage when a developer tried to add a timebomb into some code. No idea why he did it (I think his contract was coming to an end and no renewal possibly) because it was picked up when someone QA'd his code. I have also heard of code being altered to adding malicious code to extract card numbers, again picked up in QA. If you dont have checks and controls you will eventually come unstuck.
If you get to the stage where you have to call someone to interview you have already failed as a manager and deserve to be put out to pasture to find your next "opportunity".
>Errors get picked up BEFORE they impact the live environement, that is why you have senior developers and change management.
Half bollocks. Senior developers can reduce but not eliminate the risk of errors, that's where you're half right. They may well pick up syntax errors like the example you gave but at teh time of execution that won't stop someone with fat fingers entering the command as you wrote it and not double checking before they hit return.
Change management do not understand technicalities, their job is mainly to schedule changes to make sure all those involved will have someone available or on call when a change is made and to tick off boxes. They don't have a cat in hells chance of understaning your rollback procedure, all they see is that you've filled in the required section and they tick off the box.
>> Change management do not understand technicalities, their job is mainly to schedule changes to make sure all those involved will have someone available or on call when a change is made and to tick off boxes. They don't have a cat in hells chance of understaning your rollback procedure, all they see is that you've filled in the required section and they tick off the box.
Change management are not there to tick off the boxes. They are there to ensure (to the best of the abilities) that everyone who need to know about a change is aware of it and that everyone who needs to approve a change (tick a box) has ticked the box.
Change Managers won't understand your roll back procedure, but they should understand that Business Continuity need to sign off your roll back procedure - Business Continuity sure as hell should understand your roll back procedure. Likewise, Operations, who will likely be doing the back out, should fully aware of the back out procedure and understand it before accepting the change. Testing should be satisfied with the testing before accepting the change.
If RBS had followed a proper change management procedure, then there would have needed to be multiple failures/points of blame involved. Is suspect that there was no Change Management involvement in the RBS issue. My best guess is that someone who doesn't understand or value change management (probably outsourced/offshored) thought that seeing as they could do it all within Operations, there was no need to follow the change management process. After all why waste a week or more following the change management process when you can just do it now?
* NB: worked as a Change Coordinator (many moons ago), interfacing between Change Management and off-shore developers and DBA's who never seemed to understand why they couldn't just make changes on live productions systems willy-nilly - thank God our Operations, BC and Testing staff were still in-house.
@Chris_W
Sounds like you've never done proper change management. A change like running "DELETE * FROM CUSTOMER_TABLE WHERE CUSTOMER_ID = 5;" under a proper change management procedure isn't tapped in by some developer on the DB console.
Instead, it is a change - a data migration. Someone writes a small module to effect that change, the module is tested against sample data, verified to be correct by both tests and code inspection, verified on a replica of the production system and then finally performed on the real data.
This doesn't necessarily stop all errors; the automated tests could be inadequate, this could be missed by code inspection, and a bad change is deployed. However, it does eliminate "fat finger syndrome".
Obviously, all of this is a lot more work than just tapping away at a console. Depending upon your project and requirements, you may want changes to happen more easily than that (your bosses would call this being "agile", I know, the irony), but then you have to accept that more syntax and semantic errors will reach your production code base.
In a previous role I used to have to conduct "Post Mortem's" after significant outages and major incidents. Normally I'd spend a hour or so asking everybody what they did when in order to build up a basic timeline, then I'd line up the teas and go trawling through logs to find out what really went on.
The two would never match up.
Even if an engineer's actions were good and correct, they're recollection would be imperfect and there's always a tenancy to want to put yourself in the best light, and when filling in the gaps of memory the tale becomes skewed.
Most problems were caused by honest mistakes made with the best intentions, it was rare for problems to be caused by recklessness or full-on malice. Disgruntled employees who could do such things, can rarely be bothered to put the effort in to either try do things wrong or right.
+1 for the value of the logs. I would do +10 if I could.
There is nothing like a log. Nothing in the world. Of course you have to really understand them -- it's easy to make accusations on the basis of a superficial understanding of the event meanings. I did that once, and the culprit apparently confessed to what he was suspected of -- even though, as I later came to understand, the evidence was worthless. I told my boss when I realised and she laughed.
You can learn a lot in an operational environment, just by observing. Especially in an informal environment, you can get people talking, and gently steering the conversation you get the information you need. Programmers have favored methodologies, and bad ones will use that for everything. You have the managers who work with their staff, and those who consider it a good day where they don't make a decision, or make a decision which is outside their realm (mgmt - strategy, staff - tactics).
You find gaps between people. You find out the users are doing unexpected things. You find a platform stretched enough to where race conditions happen. Woe become you if you tick off a test procedure, and someone else can demonstrate it.
You find out enough about the what happened, and why, blame matters less. A nice valid chain of events leading to failure is very tough to refute, and generally spreads the blame out. As an engineer you want to know those so you don't repeat it. A good saying is there's a name for people too smart to learn anything.... dumb.
I have lost count of the amount of times I have written emails along the lines of...
X was my fault, this is what the mistake was, this is what was affected by the problem and this is what I have done to correct the problem. ongoing issues from this may be Y
I have never lost a client due to an error and believe me in 18 years in IT there have been some huge ones (errors and clients)
@Oninoshiko, if you ask for a recording then that is fine, I am against covert or enforced recording.
I have no problem with a union rep, indeed although I don't like UK unions I regard access to one as a basic human right. But that's irrelevant, if you ask for a union rep and you don't get one then the employer's legal position is undermined, and so you should get one.
Whether it is in your interests in a tricky question that depends on what sort of rep and the situaiton.
The moment you ask for a rep or a lawyer then in some managers eyes you have admitted that you did bad. That's not fair, sadly almost none of my work is in the land of "fair" though I hear the weather is better there.
If you're going for a defence of bullying or discrimination then it might help, but that is a serious escalation.
Getting a lawyer would be really very hard for the average ITPro. Firstly they ain't cheap and unless you regularly get involved in screwups you probably won't know a good one, or even be able to accurately judge their competence. Recently I helped someone get a lawyer for a big issue in their life that I knew to be good for the specific task, quote came in at £ 1-3K
As for the anonymous coward who compared me to a faded film actor, that hurt, however when I dined in the Savoy last week I had no problem getting a table and if you will take my recommendation the pork was excellent. I guess that promotes me from faded 80s actor to faded 90s ?
Dominic,
> The moment you ask for a rep or a lawyer then in some managers eyes you have admitted that you
> did bad. That's not fair, sadly almost none of my work is in the land of "fair" though I hear the weather
> is better there.
... which for me would be exactly the reason to "lawyer up". If there is no "fair", it can not be a major escalation from IT's side, especially since the firm itself already uses outside help. So I am potentially going to lose a job with a lawyer or more without one.
And for getting a "good lawyer" for the IT pro: Assessing the lawyer may go along the same lines as your job. If I tell him what happened, he understands what is going on and finds a strategy that is not purely based on lies and denial, then he is certainly better than me at avoiding mistakes (also because he is less stressed out than I am certainly going to be). And the company carries additional risk, namely that the lawyer is good and finds fault in the process.
So sorry, no compelling arguments to not get someone to help myself.
"So sorry, no compelling arguments to not get someone to help myself."
Would I be correct in surmising that you have not found yourself at the pointy end of a corporate sword yet?
Two thoughts:
1. Unions play politics too, and they may decide to sacrifice you "for the larger good".
2. If you get a lawyer, the company will get a better one.
Not saying you shouldn't do any of the above, but you'll have to weight your risks and benefits carefully. Very rarely it is a good idea to get confrontational from the very start, especially against a larger opponent.
I agree with your assessment that it is not a good idea to get confrontational from the very start. We differ on assumptions though: I understand the situation Dominic describes not as the start (when hopefully correcting the error is first priority) but rather as the "end" where blame is assigned.
So yes, I agree with your points but the way I understand Dominic, this is a preparatory stage for an eventual confrontation. He is going to trap you in lies. This is disguised by an outside location, lack of managers, tea and biscuits and his politeness. But you have already been singled out and Dominic described several reasons why this may be wrong as in the lack of a CVS etc. Hopefully outside help does realize eventual flaws in this selection process. I am not sure I would.
And yes, I have not seen the pointy end myself (yet) but I have seen this situation. It put enormous stress on people. I do not think that I am going to make very good decisions in that situation and outside help should (again: hopefully) prevent me at least from making the worse mistakes (as in telling lies, not spotting obvious defences etc.).
Taking legal advice about an issue as serious as your continued employment is a matter of competence. If you are not earning then mortgages, school fees etc don't get paid. If you resign under a cloud you will struggle to be re-employed and for an added kick in the teeth forfeit any income protectection insurance that you have in place.
Dominic's examples are unusual in that he is talking one on one. More often there will be a technology group manager, business manager and an HR rep trying to browbeat you into accepting a decision which was made before the meeting. You will be made to feel in the wrong, that you are holding back your colleagues and that it would be best for everyone if you quit without making a fuss. Having an external ally in any such meeting will massively reduce this peer pressure.
Finding an employment lawyer is not diffcult (Yellow Pages if nowhere else!) and you might have legal cover bundled with your household insurance.
All this will not guarantee your continued employment but at minimum you will walk away knowing that you didn't just roll over.
Forensic IT Interviews can only give you facts and evidence. In a modern business environment that's something very unpopular. Lack of facts gives you freedom to do whatever you think is right, unconstrained by empiricism. If a manager would have to act by facts and logic he could be replaced by a computer, or the next best person. He'd simply have to choose the best solution. The more facts a manager has, the less important he is.
The only reason I can imagine a company paying for that is to create the appearance of something being done. In that case the result is utterly unimportant.
have it easy
Oh dear I've just deleted the entire customer database, the boss is going apeshit, the headhunter has been called in to take heads... oops conduct in depth interviews to get to the bottom of the missing semi colon and fire the person responsible.
In my line of metal bashing, oops we've missed a crack in a hydraulic coupling casting and there's 1000PSI oil going everywhere and the aircraft's rudder is stuck pointing 5 degrees to the left, not a good thing at 24 000 feet, or in setting up a new robot cell, we miss out verifying the door interlock switch in the PLC, some poor fool opens it while the cell is running and gets grabbed by the robot and stuffed into a machine tool.
Then the brown smelly stuff really hits the fan.
Along with the guilt of "did I f*** up and kill someone"
Not to worry, some of us work with systems that can produce similar results. Admittedly, the one I'm paid to work on can't kill anyone directly, but the right kind of bug in it could be a major contributing factor to a fatal incident.
Picturing myself on the receiving end of a fact-finding interview from someone like Mr. Connor, having to explain my actions in detail, makes an excellent counterbalance to the temptation to give in to the JFDI mentality and take shortcuts.
"Not to worry, some of us work with systems that can produce similar results"
Not to start a pissing contest or anything like that, but one major difference, assuming that the guy above is a licensed aircraft engineer, is that if a developer fucks up badly enough, he will lose his job and will have to find another one elsewhere, whereas if an aircraft engineer fucks up badly enough, he will lose his license and therefore, his livelihood.
Likewise, both are liable to get people killed, but the engineer is much more likely to end up in prison for it.
For the reasons above I have immense respect for these guys.
The flip-side of this is that the management in the aircraft case realise their staff are dealing with serious stuff whereas I'm not sure IT management are. From the article:
"One thing that non-IT types find hard to grasp is that the scale of the consequences and the cause don’t correlate at all - although RBS management probably gets that idea now."
I very much doubt it. Facts are scarce but from the outside it does look like a slightly larger screw-up by RBS's IT could have tanked the bank. Bankers probably realise (though Lehmans, and Barings before them, failed to protect themselves, even so) that traders can destroy the whole company in an afternoon. Trader working conditions (supervision, pay packets, etc) reflect that. I see no evidence that IT staff are treated the same way.
And if a bank goes tits-up, taking a hundred billion of your favourite currency unit with it, who's to say that there aren't deaths and similar consequences down the line. We just don't know for certain which deaths to attribute to the loss. Society is awfully exposed to the banking system.
I find the greatest hinderance to getting the truth is often an inability to involve the correct indiviuals in a proper evaluation. In some ways the article reassures in that it states that "binary" answers are not provided. However, I have concerns that an individual is engaged in this activity. When things get to the point described in this article then checks and balances are absolutely required. On a number of occasions in the past I have been asked to explains actions that I have not taken. Why ? Well because an individual with an inflated opinion of his own knowledge and interlect has made a critical assumption early in the analysis and then proceeded to complete the story on that basis. In all cases I have told the involved management that I would not accept any mopre instances when that individual was involved in any review without my participation.
My impression from the article was this is about determining whether it is likely to be accident or malice?
I realise the two are hard to separate but, as others have alluded to, things like having somebody else check your work mean you're less likely to the the guy who accidentally removed a disk from the RAID5, went to fix it and hot-swapped the wrong drive, rebuilt it, then tried to restore from a tape you accidentally kept appending to for ten years and have been storing next to a speaker.*
If Dominic Connor's job is merely to "find who is to blame" then it is, indeed, a very sad reflection of corporate culture and a chilling reminder that IT staff are not the ones who get golden parachutes. However, if he's called in to give some indication as to whether an individual responsible for a chain of cock-ups is unlucky, an incompetent liar or bent on destruction I'd say there's less to fear and more articles like this are needed.
*yes, I know, data centres aren't like that.
...I question the validity of the approach.
1. If the person in question is incompetent to that extent then it's his management's fault for having left him in place to do the damage in the first place
2. If he's malicious then, again, I'd question what was going on in the workplace; there should always be checks and controls in place in any organisation (including monitoring motivation levels in your staff)
3. If it's a mistake then initiating a blame-game process won't help as the only winner will be he who covers his tracks best. Better to find out what happened, why and how it can be prevented in future.
If I did want to perform a forensic analysis to dig up some real malicious / incompetent behaviour then I am not sure that a HR consultant / recruitment agent (sorry, head-hunter) would be my first port of call regardless of their apparent self-believe / abilities. I'd probably start with an investigation's agency along with a forensic examination of any machines / logs. By the time Mr HR has had his chat there's time for all the evidence to have been removed.
Nice piece of self-promotion here though. Hope it works out well for you.
Just my two pence worth.
@Adamwill: Unions *were* a good thing, where we part company is that they are all that useful today given their tendency to follow agendas that workers either don't support or dislike.
As for the point about outsiders coming in to mess with staff being stopped by unions, the reverse is observed to be true. If you look at the public sector, much of which in unionised consultants crawl all over that assessing staff and "improving" their work practices.
Also they do play politics, one guy I used to work for was terminated because a union rep became his manager and they'd fallen out because of a strike.
The word "blame" is tossed around in this discussion a lot and references made to the more formal processes found in engineering. This part of my work exists because most IT groups have no such process, they get me instead. It follows that sometimes I have to work out whether the thing that happened was bad at all, one report I wrote basically said "I can see why this scared you shitless but actually nothing much really happened and it is gone now".
Backup tapes do fail, I recall reading various stats about how often backups can't be restored, and numbers vary from 10 to 50%. The vast majority of s/w bugs are stupid, not just mine either.
One issue in some code written by IBM was where someone had taken an error code, coerced it to a memory address and copied data into it. I will admit here and in public that I've called the wrong function by mistake. That's the nature of IT.
A big difference between safety-critical systems and corporate IT is navigation around black boxes.
Our daily work consists of dealing with large lumps that every so often fail to do what we expect and give diagnostics of the form "Operation failed". Which is less than useful and isn't predictable and where the vendor will flatly refuse to tell you why.
I have read the RBS is suing CA, someone like me is going to make some money out of just this sort of situation.
>I have read the RBS is suing CA, someone like me is going to make some money out of just this sort of situation.
That should be good. Surely the CA end user agreement will have the usual this software is crap clause. If it doesn't do what we say it does or you lose money by using it then tough shit, you can alway pay us more to correct the errors and introduce some new ones so we can repeat this process at a later date.
And as to who is going to make some money out of it, you shouldn't need to ask, as always the lawyers.
@Jon72 : Yes, of course, I hope you will forgive the blatant absence of specifics in my replies but for instance:
In one system, it was claimed that specific person had made a change that caused a major issue. I was able to demonstrate that the change itself had never been made. In this case I was able to demonstrate that a change had been made to the s/w that talked to it, causing the failure. This change itself was rational, being something the vendor itself recommended. No one was to blame, just random IT shit and when run on a system without the patch the problem simple disappeared.
In another, it was said that the developer had done something bad. Code reviews are tricky in this context, however I realised after staring at it for a while that the code wasn't big enough to be what they said it was, it would require more lines of code to do that particular bad thing.
A third was a guy was deemed to have hacked into the system, normally that would be forensic IT not me, however I realised that the stuff he was supposed to have accessed wasn't on the hacked system.
Even with all technical detail expunged you can see that these were unarguable proof and I'd stick my name out that in general I'd expect to more easily find clear proof that the behaviour was reasonable than proof it was bad. That also goes to the sort of work that bounces my way, because if someone has very clearly done something bad then the firm don't need me.
Frequently.
Just a few weeks ago, I had to remedy the initial effects of a spam relay attack, then coordinate efforts to prevent it happening again. I also had to prepare the incident report of what happened, how it was resolved, how it was handled and what could have prevented it.
I also got a major grilling on that report from my bosses and our liaison at the customer. It was pointed out that we'd raised this very issue as a weakness for the latest a few months before.
It may not sound much, or as important as RBS, but when it involves a global firm with a market cap of $3bn, & 20K+ users worldwide, potentially having all outbound mail blocked by their external perimeter mail host, it can be very bad for that customer and us.
AC to prevent identification of the client and my employers.
When being pressured by the boss to get something done quickly, ensure you are on CCTV when you hand him the keyboard, and tell him to have a go.
Then leave the room, make a telephone call, under another CCTV camera, and when you get back your boss will be responsible for EVERYTHING!!!
All's fair in Code and War...
@Joeman
"When being pressured by the boss to get something done quickly, ensure you are on CCTV when you hand him the keyboard, and tell him to have a go."
Aaargh no!
You've brought back bad memories of a company secretary who would wander in while I was running weekly payroll and do something out of sequence.
He always disappeared sharpish once he realised what he'd done, leaving me to clear up the mess.
then your manager (who has access to his manager and whom you don't have access to) is quite within his rights to say that he does not have confidence in your ability the next time a prime assignment comes up. He has not lied, he has just inferred there is a short-coming in your skillset (personal or professional). 2 or 3 of these inferrnces and you get a bad reputation without even knowing its happening.
Access is power, unless yours is better than his, you will nearly always come off second best.
Oppressive, obsessive, obnoxious overachieving supervisors, managers and vice presidents who have reached the very top of their Peter Principle position. They can't let the person work by constantly probing and asking/demanding the work be done in the shortest amount of time with the least resources.
Sounds like SOP to me.
>Sometimes it is quite hard to objectively tell the difference between incompetence and malice
It was only after I'd quit my job and seen the company go down the pan that I realized it was incompetence rather than malice.
..so incompetent, that the MD (who I'd though was being malicious) when sacked for his incompetence found out, he tried to email himself over 3GB of sensitive company data using the company's email system
..so incompetent that competence wasn't part of the MD's job spec, so he managed to sue successfully for unfair dismissal.
AC to protect the guilty.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2025
