Customer never gets told No.
So i a work for a company who's global reach includes the recent NHS Fiasco....yup that would be them. Having shifted and changed directions and paths over the years to further develop my skill set I currently am at the position where the thought of doing day to day sysadmin work fills my head with thoughts of suicide and murder. Not me of course, just assisting the customer in their final options.
So over the years i have experienced much and seen much, and I'm currently overseeing a group of Sysadmins for Wintel and Vmware. These are people who still think that the customer is always right and that the customer gets what he asks for, regardless of the limitations of the human body to go without sleep. (124 hour week anyone?) So i'm taking a moment to reflect, when i hear (for a better term) one of the PFY's agreeing to a raft of demands from a customer about patching the server environments. Patch the entire environment, tonight, in 6 hours. That would be over 500 varying servers in 6 hours....a bit of a push to say the least, yet wait, we are to catch up with the missing patches for the last 6 months.
Now any good sysadmin will decry ' you have not been patching the servers? how crap are you!!' but hold on a minute... these are customer servers, we can only patch when we get direct permission to do so.... which we had been chasing, for 6 months oddly enough. So its agreed (minus my vocal dissatisfaction and protest about time frames) the servers will be patched.... in 6 hours.... with 8 different OS's as minimum, after doing a normal day (14 hours) the poor PFY has been targeted as the victim who will patch the environment that same day till 6am... Of course it fails, the patching server cant cope, the environment is too messy and some of the time frames for some of the servers (shortest being 30mins for 10 servers) has left the pfy a broken man (with RSA and other conditions) and sends an email stating the successes and failures.
Being the kind man i am, i let him crawl into a bed and proceed to die. Yet management decide that 8am is just about a good time to give the PFY a direct call to discuss the email he has just sent. And then again an hour later.... and then again 2 hours later.... and then again at 1pm. This must of been the time he gave up on getting any sleep and came into the office. His eyes told me of his pain.....The phone call i got a mere few moments after that even gave me a moment of pity for him....but that soon passed...he was, again.... to be patching till 6am. THIS time every server had to be patched, so, giving him no work for the rest of the day.
I allowed him to prepare his assault on the server lists and the patches required. He did well and manged to get all the servers patched... emailed out his victory success and stated, that yet again he would crawl into bed and to please do not call him..... but the management team, who i can only assume had lost the ability to read an email... or anything for that matter, decided to call once more!... this time it was because the customer hadn't received a text stating the success of the patching. Supposedly this had been agreed by management ( and completely failed to be filtered down).. but that wasn't the best part, the customer was complaining about the fact the server was missing patches.
To stick up for my team and the PFY's honour I decided to take a look myself. Nope the customer was wrong from date x to date y the servers had been patched and was considered fully compliant. Oh how foolish i was to think it was that simple, no, the customer was complaining about patches that were not on the server, from 2003!!! that's right, the tool the customer was using was the Microsoft security tool to check which patches were needed. The customer was outraged at the lie the PFY had emailed out.
I went to bat on his behalf, asking the management team to kindly not call the engineer again (disturbed from 7:30am onwards) or i would view it as employee abuse. After a significant amount of wrangling and the discussion of the use of correct language ( i.e not foul) when asking for something to be done, management once more collapsed to the indomitable might of the customer and agreed to their demands... which now expanded to all the applications to be patched too......so that meant the patching requirements had just increased about 10fold.
The PFY comes back to the office... eyes like black holes and a face so gaunt i could shine a torch through both cheeks and see the other side, and he was beginning to smell a little ripe too, but only just. Again he is tasked by management with the job to patch, he pleaded with them, i pleaded with them, our co-workers pleaded with them. But to no avail, management were determined everyone else was far too busy. The look on the PFY's face was that of a man on the verge of emotional collapse. But soldier on he did... a longer patch window had been agreed.....24hours constant patching with servers blocked into 2-6 hour windows in which we were ordained with permission to patch and reboot.... His 4th day of little or no sleep was at hand and reaching into a 5th, already he had clocked up a fantastical 82 hours of work and more was to come.
The end came, he released emails, reports, screenshots, text messages everything......and STILL management called him to find out the status. That was it, he snapped (i don't blame him) i heard the phone call... i heard the language...i smiled...Management were shocked at his 'attitude' and came to me to complain. Oh for the 30 minutes of ear bashing i gave them, made what my PFY had said on the phone seem pale in comparison. No proper procedures, no proper practice, the customer had been told yes yes yes yes, and why? why was this all necessary. Company audit and the customer forgot it was coming and didn't plan to have the servers patched.
The PFY isnt allowed to speak to the patch management and incident management team anymore. I have been given a polite warning by my manager.... the patch manager (term used loosely for someone that emails the customer asking permission to patch and doesnt actually patch or plan or do any work) is not allowed to accept any patching requests from the customer that are not 3 weeks in advance. So its a minor bonus i guess... :D
Biting the hand that feeds IT
