Two clichés, one headline: 'No good deed goes unpunished' and 'It's always DNS' Beware the perils of being too helpful as we kick the working week off with a Who, Me? starring both DNS and the need of management to put someone's – anyone's – head on a spike. Our tale takes us back to 2016, when "Sam" was working for an IT service provider. Sam's services had been rented out to a company that had just been …
Do not start changes early. CAB won’t have accounted for that and you may break someone else’s change or testing.
Have a test plan. This includes testing the broken and fixed functionality pre and post change.
Any decent organization will have the help desk log the start and end of changes. Always ask them for permission to start work and have them update the change to “in progress”.
If you finish early, you just tell people it’s still being tested and you don’t mark it as complete until the users have told you it’s working.
Mailbox Permissions Issue
Team mailbox configured with very eclectic permissions, e.g. Some folders open to all, some to 1 or 2 people etc., some allowed to edit, some not. Stupid but ours not to reason why.
We got a request in to update a couple of permissions for new staff, colleague used a tool we had to make the changes but made wrong change at top of mailbox and removed everyone's write permissions.
Grovel from colleague to PA to get an idea of permissions again and all quickly sorted. Screenshots taken for future reference and supplied to PA.
Next day permissions had gone screwy again but at various sub-folder levels. Wasn't any admin staff this time.
PA - You obviously didn't do it right, I'm fed up of you IT monkeys making my life so difficult
Us - Well we sent you screenshots of how it was all configured, here are new ones showing the changes somebody has made
PA - I know it was you lot, couldn't have been anyone on my team we're all perfect, just f* fix it before I report you all to CEO, etc. etc.
Us - Sure, here you go, with new screenshots to prove all good (+ mailbox auditing turned on in background without telling her)
PA - It's broken again, you lot must have done it, I've already lodged a complaint
Us - Really? Here's the audit trail for that mailbox showing that 'Joe Bloggs' who she had insisted had to be an owner of every folder despite asking us to make permissions changes, her manager if I recall correctly, changed XYZ at time ABC
PA - Not a word :-)
Indeed never even take an interest, show curiosity or ask a question.
If you do then everyone else will metaphorically take a step back, and suddenly it becomes your problem and your responsibility. At least until it's fixed, at which point everyone else will of course take credit (or try to).
Exactly. We have a particular customer that is a bottomless pit of problems. I won't do anything without a ticket. I wouldn't even close a door for them without a ticket. I won't talk to them via phone because that conversation won't match what goes in email. A web server took a dump recently and they had no support for the app from the vendor and it was very old stuff. I figured it out, so now I will be the "expert" in IIS and basically anything referred to by three letters.
Had a previous manager who was the exact opposite of that.
Wouldn't take any heat for team, would be on phone or at our desks hanging around and demanding instant solutions to anything that affected anybody senior even if it hadn't been logged as a priority.
Less patience than my kids and less happy to take responsibility than them.
Not a Manager, a Colleague I will call Reg in honour of our gracious hosts.
He would wander off every so often, for a cigarette or a coffee or a visit to the restrooms, or lots of other excuses during the day. After these forays out into the real world he would often ask the rest of us to do little jobs he had just remembered being asked to do some time previously while he got on with the boring stuff - since it meant we didn't need to do the boring stuff, we were happy to help.
Until the day one of the Managers came down from their ivory tower and asked where Reg was, as they wanted to thank him for something he'd done that had saved the PHBs a lot of heartache.
"Oh yes?" asked the team lead, "What was that then?"
"Oh," said our esteemed visitor, "He..." and proceeded to describe something one of us had done while Reg was out having a coffee or a cigarette or a toilet break or whatever his excuse had been that time. Further questioning elicited that our good colleague Reg, so eager to do the easy but boring crap while we helped him out, had been schmoozing the bosses and getting all the little tasks, getting us to do the work and then going back to the PHBs and taking all the credit, saying we all seemed quite happy to let him do all the difficult stuff while we sat around doing the easy bits.
Reg left shortly after, but his final words were not an apology for taking all the credit for the work we had been doing or how good it had been to work with such a good bunch of people, but complaints that *we* had stabbed *him* in the back over all "his" special projects for the managers...
Stabbing him in the front was tempting, as was giving him a good smack upside the head with a giant cluestick - but no we had much more delicious revenge because they arranged it that we were in the area when they called him on it...
T'was sport indeed to watch the cheeky sod get asked exactly how he had done something, and then asked why it had someone else's name as author in a comment in the code...
These good managers exist.
We had a problem where there was an urgent fix needed and we kept having our chain yanked by management, and by people wandering round offering to "help". About half of our time was being wasted answering calls and saying no thank you. Our boss had the knack of wandering by whenever the volume of noise/frustration from our office increased. He said he would handle it, and told management that all communications had to come through him. It was a good feeling telling a senior manager to go and talk to Pete, and hanging up.
Pete came round every hour and took us for a tea/coffee/cake/chocolate to a) get status, b) get us away from our desks to talk to each other.
We fixed the problem. Afterwards Pete said there was a huge customer demo needing the fix. Management didn't want to tell us about it in case it scared us.
Later we were given a "thank you lunch" to which the senior manager came, and paid for it.
Even worse are conference calls for panics. People (mostly management) joining randomly all wanting to know status. Since they're all bored, they keep asking every 2 minutes "Are we there yet?"
More than once I have responded "I can either fix this, or answer the constant questions. Which one fo you want?" Generally results in deathly silence.
More than once I have responded "I can either fix this, or answer the constant questions. Which one fo you want?" Generally results in deathly silence.
I know that feeling. Back in the mists of time, working for a telco beginning with 'B' and ending with 'T'. It had a nice escalation structure going through management levels 1-5, with many 1-2's that wanted updates every 10mins. After a couple of those panics, we just installed a message system so we could record a status message, then get on with fixing the problem.
Now in IT we have progress. So panic/crisis calls became bridges, or IM chats, which can lead to fRANTic message spam demanding to know why the engineer's status shows them as 'away'.
About 04:00 AM on the forth day of a sev.1 problem involving recovery from a SAN server meltdown that killed our entire office environment, I finally made some progress (I should only have been nursing the system on the graveyard shift while waiting for the support centre to engage while the one-off support call details were checked, but I thought I would spend some time looking into the problem while I was waiting).
Not only did one of the design architects stand guard keeping people off my back while I confirmed that we were able to move forward, but he kept bringing me tea and coffee to keep me alert and focused while I followed the first steps in the recovery procedure. And once I had confirmed it was working and we could access the backups, he told me to knock off and go and get some sleep while they called others back in to continue the recovery.
Although I was instrumental in the recovery, I always put it down to me being too stubborn to just give up and wait for the 'experts' (and it was me missing out a step by mistake in the online fix info that allowed me to make the breakthrough). I guess it was this seld-deprication that did made the managers not think of including me in the end-of-year shout out to people making contributions to the business!
It's one of my favourite lines with various of our managers around here (thankfully rarely my own boss, who is definitely on the decent PHB list) -
"Do you want me to take time making a detailed PowerPoint to explain the problem, its history and where we are now so you can understand it, or should I just waste the time by actually fixing the problem?"
It normally gets the point across...
I had one like that, he wasn't particularly good at any form of IT/Design/etc, but deliberately only employed people who were. He said his job was to protect us from all the corporate crap and in return, he'd bask in the reflected glory of our (hoped for) successes. He was as good as his word.
When I was bogged down with a particularly tough problem that I just couldn't suss my excellent boss stuck a note on my now closed door stating that I had the plague so zero visitors pls - we were a fun relaxed site with plenty of time for social chat & coffee etc so a note saying Keep Out was a massive blessing. He backed it up with more formal email to all involved requesting No Disturbances for the time being.
On top of this he also took all the flak that came from other sites until deciding who would deal with it. He retired a few years ago and I'm two jobs down the line now, but he was a top boss.
There are some good guys. I once destroyed the email server with a window's update (many moons ago.) The timing was to say the least extremely bad and email was vital to find out what was happening to people in a sensitive part of the world that was becoming unstable -- the boss and I did the disaster recovery in record time, we had put a notice on the server room door that said "We know" no interruptions -- a really good boss who has our backs.
I would tell the manager to go swivel. Change management is there for a reason. I would also insist on him issuing that order in writing.
At least where I work, it doesn't matter who asks for it, nothing goes to production without it having gone through change management and change being approved.
Ah change control. Yep, everything thru them. Disapproval granted for lack of full stop in final sentence, wind is in the east etc. And when approval does come, its about 2 days after change was required. Emergencies ? Your fault, you should have planned for that.
> Because it was requested by somebody above him (and he didn't know any better at that time, I would at the very least have asked for written instructions as I have learned the lessons about CYA).
I got to this point in the story and thought that he was going to back out the change. That would have been ... interesting!
I had a good manager. Supported his staff took the flack from the loons and allowed us to get on with the job. He moved on and his replacement was a dick! If it broke his number one priority was we tell him we had a problem, describe the problem and offer him solutions from which he would choose the action we should take. (Oooh look the ups is on fire, the options are......Yes I did use this as an example, cos I'm a twat!)
ITIL and change management became a tick-box exercise. One day an inexperienced tech removed the root entry in the internal DNS. And the change management team said this was a good idea. Turns out this was a bad idea! A couple of days later I discovered this whoopsie, fixed it and wrote it up.
Then got a bollocking for not telling the birdbrain manager and setting up the original, inherited legacy DNS incorrectly. DNS was set up just fine, but I'd made em look stupid. Head block chopping on you must go!
Anyone seen my ITIL manual?
It's in the basement. And unless you've completed your leopard handling certification, that's where it should remain.
But ITIL often becomes a crutch. People write procedures that often aren't the people that do the work. But it then provides a shield & weapon against anyone who's not following procedures. So I once read a safety manual for safe working at heights. Ladders must not be climbed until the top of the ladder had been fully secured. Or they don't handle exceptions very well. Why was there no procedure for what to do when a volcano on Sicily erupts, and sends lava flowing downhill towards a major cable landing station?
But that's all part of the fun. Having a documented crisis process that deals with exceptions is fine in ITIL-land. Spin up a bridge with relevant VPs or C-levels on it to approve stuff as you go, and a note taker to document as you go. Then it can just be a matter of fending off middle managers who feel like they should be involved in the crisis, but don't really add value. If they need to do something, their manager will inform them.
I thought sharing stories here WAS therapy.
And it's free.
But we're here to support each other to avoid/mitigate mental illness, not cause it. If it needs to stay down, by all means we* understand.
(*Maybe not all of us. I'm speaking ideals and generalities; exceptions surely exist.)
For you ==>
Shrieking from head office demanded that the change be reversed
Its a pity that didnt happen , then Sam could clarify his fix
"yes the email is not working again because you demanded that I remove my fix"
.
"those who have worked in the cesspit of outsourced IT, I'm sure they'll get it!"
I have , and those the levels of shouting , buck passing and ass covering occured often.
Way back when, I was asked to integrate with new customer. They had a SOAP API and we were .NET so pretty easy to get going. I had a test endpoint with fake data, all seemed good. Then I started experimenting and discovered their credential service wasn't over HTTPS. I got told that the mistake was mine, so I wrote a 5 line curl script that hit the HTTP endpoint to prove my point. Someone in the email chain forwarded that onto someone in senior management their side who understood enough tech to know it was bad.
Customer went quiet for 2 weeks, my tech contact their side wasn't replying to my emails. Got an invite to a call, me, my manager, a junior who was shadowing me and about 4 from the customer. The project lead their side tried to talk around the subject, putting back dates etc before someone senior to them butted in saying the architect their side had been fired and the team (a handful of tech) had walked. I remember my manager mouthing the F word and the junior went white. We were asked if there was anything we could do our side - you know - to secure /their/ service. My manager said something along the lines of "Is there anyone technical over there that you can talk to?"
The project got canned and they continued with an ungainly passing of Excel reports.
I would have to say that I would have told the guy to do his job and keep me out of it.
I would also have found it quite suspicious to come to me and specify that "it didn't have to wait until 3, you can do it early".
Nope, no way. You're not going to weasel way yourself into an excuse for blaming me.
Of course, I say that with the benefit of 25 years in the industry. I've seen enough office backstabbing happen around me, and sometimes to me, to not have a sixth sense about it.
Add in an Antwerp-based marginal 'digital transformation manager' who talked himself into a position as CIO and has no idea what he's doing.
His default reaction to being embarrassed by staff is to fire that person and let another contract to Accenture at 3x the price to cover for himself.
I'm hoping the CEO and CFO figure out his game before it's my turn.
I had a great boss for 14 years, at a Stockbrokers. He worked on the theory that our job was to fix stuff and his was to run interference with the C suite and other random "users" to the extent that when we had an emergency, the first thing we did was take our phones off the hook so they would not ring. Luckily the IT area was behind a security door and we were the ones that controlled who had access to the area. So random people walking in and bothering us was not an issue.
One day I happened to be nearby when he was chatting to the Chairman and CEO who opined that he never saw us so maybe we weren't necessary. My boss came back with, " we work on the theory that the only time you see us is when there is a problem, therefore not seeing us is a good thing".
Wonderful boss, and I was sad to be stood down when the GFC hit. Unfortunately, when he had an opportunity for us to work together again, it was shafted by a twat who had been a "Projects manager" at the Stockborkers (sic) and shown the door when it was proven that IT could run our own projects without his input. This twat was good friends with a replacement CEO that took over the organisation we were going to work for, and poisoned the well for us :(
Before I retired I ran a small specialized software company. Amongst other things, we had written a product used by a number of local authorities. One site in particular was "special" as it was the first one. I had done "them a favour" by saving data from an American based system that had been badly corrupted. The original supplier did not want to fix it, as we suspected that each of their sites had a different "bespoke" system and the original author had retired. MS Access 2 had just come out and we cobbled them together a simple database containing the original data. Several of the "staff" were volunteers and the manager wanted a "friendly" front end and some management reports. The whole thing cost them a couple of thousand dollarpounds. We started to get phone calls from other users of the American system asking us to sell them a copy. Over the space of a few months we had sold about twenty. We were then asked by a national government body to expand the system so that it could be used widely. After suitable workshopping and user feed-back we had ~200 sales, and the option of a SQL Server back end...
Then the problem with the "special" local authority... They had removed much of their IT (3 people) and replaced it with a single in-house support person and a large national contractor. After a few months, some genius noticed that the department where our software was installed was about a mile away from the main building and connected by a 128k ISDN. Our stuff ran well with a local printer and a dedicated SQL Server. The staff used (almost exclusively) Outlook, Word, Excel and PowerPoint managed by a small server with a shared printer. The staff managed nightly back up tapes themselves. This arrangement meant that the new contractor had to send someone to the local site on occasion.
Now the bit of real genius thinking - The contractor decided to manage the local office from the main building by joining the local site to the main PDC, connected by the ISDN. The local staff discovered that sending a MS Word print job would now take minutes to complete. The local office manager asked if there was anything that could be done to fix that. I suggested putting the main printer on the receptionist's PC and making it a local share. Apparently, from time to time the contractor removed the share so that it could be "managed better". The office manager would then put it back.
A couple of weeks later I had an accident and badly broke an ankle and was "no load bearing and no travelling" for 16 weeks. After about 14 weeks I got frantic phone calls from the local office and their senior manager - Everything was broken. I got someone to drive me and my wheelchair to a meeting that was set for the following day. When I arrived I was sent to the conference room where the contractor had five staff down one site of a long table, the senior manager was at one end, the office manager was at the other, and I was on the other side of the table from the contractors. After a number of years of working in the public service I could recognize a "put up job", so I politely said good morning and let them talk. After a brief intro from the senior management I was then subjected to about 5 minutes of unpleasant ranting from the senior contractor, with additional barbed comments from his staff. I waited until they had stopped and asked how long this had been going on, and what was actually "broken" to cause the meeting. "Nothing has worked for the last month" I was told, "You keep going in there and fiddling with stuff and breaking it. The gist was that we were a small company and don't know what we were doing, and that they were a large national company and were considered to be an industry leader. "OK", I said, "We were asked to supply our software by the main government body, and now have about 200 sites - This site seems to be the only one with a problem. I have been doing this stuff for about 20 years, the staff that you send appear to be juniors. You can see that I am in a wheelchair - This is my first day out for more than 3 months, so what has changed on site?". "Nothing", I was told "Your stuff broke it!". I asked "Could I make a guess that you have included the SQL Server in the main domain and moved it to the main building? If you have, every bit of traffic now needs to go up and down the ISDN. If you want to work that way, did you consider using "Terminal Services"? It went very quiet on the other side of the table...
It was agreed that I would "help" the large contractor sort things out. After they left, the local and senior managers asked me to stay and called in the remaining local IT person and then asked what was really going on. The story was that the contractors were coming in every few days, their manager had said that he was not impressed by our software. They thought that they should rewrite it (and presumably take over this small market, of which we were now the main supplier?). The senior local authority manager knew me; and after everybody else had left, told me he thought that I could help sort out what was actually going on. I suggested that maybe their big contractor had tendered for a low-cost basic on-site support contract, and that "additional hours worked" were charged at a (much) higher rate?
It seemed to sort itself out. The SQL Server went back to the local site, the LAN was sorted, the in-house person was asked to "supervise" local back-ups; I received a nice invite to the Mayor's cocktail party; another local support person was employed; the new contract was with a different supplier for a fixed up-front cost, and site presence *only* after a call out with a <24 hour response (for about a half of the real cost of the previous arrangements). The new owners of "our" company still have the site as a customer along with about 600 others, but the bad news is the contractor is still in business, and some still consider them to be "an industry leader".
I was working on our company's WISP gear in a cabinet at the bottom of a transmission mast when the power went out. Our UPS worked perfectly keeping our subscribers online, except they had no Internet. The company we bought the Internet through, their UPS had failed.
Wanting to get everyone back online again I opened their cabinet and ran a power cable from our cabinet to theirs and powered them up. Everyone came back online. I phoned the other company to let them know the situation. Their engineer turned up in a right rage and said I was the one who had caused the power cut. Clearly not me as the surrounding properties were without power. They threatened to sue me for putting their service off line when in fact I had got it back online.
1) Nobody at my company would ever behave that way
2) Been on the receiving end of such bureacratically-inspired nonsense
3) Been on the receiving end of such bureacratically-inspired nonsensemore than once
4) I don't see the issue - change management is a sacred process and Sam should have been sacked
(anyone answering 1 and working at a company over 5000 people is a liar, and anyone answering 4 should be sacked but will probably hang on to their job long after the Sams of the world are sent packing)
I worked at a place where one manager was trying to get rid of the hardware manager.
After a Tuesday patch day went badly, we had an after action meeting.
She asked why it was not tested. Umm MS does not give them to us beforehand.
Umm our dev environment is not nearly as complex as production.
The manager in charge of the hardware left within a month and the toxicity level went up.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
