An IT emergency during a festive visit to the in-laws? So sorry, everyone, I need to step out for a while Much of the world may be on holiday, but On-Call – The Register’s weekly tale of readers being asked to rescue tortured tech, is still hard at it. This week, we offer a topical tale from “Bill” who spent one festive season in the company of his in-laws and had decided to bring his work laptop along because previous experience …
Long ago, in another life, I looked after an inherited NT4.0 file server built around the infamous EDS.
No RAID setup or adequate supply of back-up tapes, for reasons best known to the beancounter director.
What could possibly go wrong with that?
One day Murphy decided that a corrupt EDS and a worn out tape at the same time made for a lot of fun.
As a result, I had to stay in a Thursday till 03:00 and the succeeding long week-end to fix that.
O.
> What was to be done? Would Bill need to abandon his family, and his holiday?
It seems to me that the people who caused the problem are the ones responsible for fixing it. If they cannot, then it might be time to replace them with others who could. Or better: with true professionals who would avoid the problems in the first place.
I have worked for outfits that have absorbed other companies. Including their less capable IT departments. It would not surprise me if those depts. lack of ability and (frequently) their antiquated systems was a factor in their parent companies owners wishing to get rid of the problem.
Whilst I agree with the sentiment, sometimes its the bean counters who made the problem or short sighted Directors who wouldn't approve the fix before it was needed.
In the case of having the equipment but not bothering to set it up sensibly or not asking for the correct solution, then yes hang them high.
I've been 'lucky' enough to inherit several failing tech support teams over the years. In both cases case it was pure bad management, in the first the guys just needed to understand the basics of change management and logical problem investigation in complex environments, it took 18 months but I turned them into a world class team with completely stable production environments and we went on to implement several major technology changes (change from novell to windows server,implementation of Unix servers, email implementation then migration from group wise to exchange etc) successfully after that. In another case the team was split between the incompetent, the lazy and the ill informed. It took longer to get rid of the incompetent who had been the 'team lead' but I managed to bring the other two up to speed by putting my own team to work mentoring them and helping them to develop, all they needed was exposure to a good team culture, one their previous manager had deliberately poisoned
Our lot did a similar thing, consolidating as far as possible into a single stack.
Basically they looked at the nightmare we had of third-party supports passing the buck to each other, and decided they needed to do something about it. (hypothetical example: Novell blames HP, HP blame Oracle, Oracle don't respond to attempts to contact them but our own oracle DBAs blame 'networking'...).
It's political, sure, but arguable still a technical benefit to get as much as possible in a single supplier. Problems get fixed quicker, there's fewer compatibility issues (and the ones that do happen are clearly the fault of that one supplier). If there is a problem where the MS Outlook team blame the MS Teams team and visa versa - it's on *them* to sort it out. We don't need to spend half as much Incident Management time on making them talk to each other.
I've worked with some who absorbed MORE competent IT departments and ended up pushing out the better staff because they made the larger company staff look bad by comparison.
I remember working for one company, when we merged they did a survey of both companies in relation to IT support (as they wanted to outsource it entirely). Our company had an approval rate internally over over 90% which is insanely good, the other, large company was under 20.
The poorly performing one was kept as oddly enough the person getting to make the decision was in charge of that for the previous 10 years, spouted some nonsense about "learning to do better" then went back to sleep while his best staff left.
At my last place we audited ourselves perodicaly, checking for permissions etc, the one thing we would do is record any account changes & the date of execution in the account "Description field" with the history kept intact &w ho executed the change.
When we were let go & the other half of the team kept on, they didn't want to do that.
I'd love to know how the PWC auditors reacted on seeing our nigh on perfect record plummet & the errors made.
So what do you do when a company decides to replace its generally functional IT department with an outsourced version that is far less capable? Saw that happen on my way out at a company some years back. The IT department wasn't what I'd call top-notch, but given the amount of resources they were given, did a pretty good job. Then someone in beancounter central decided that it would be a good idea to outsource all of these functions to HCL. I left in the middle of the transition, but kept in contact with a few of my former colleagues who said how things... did not go well. In basic outsourcing fashion, HCL brought in a bunch of people so green they needed mowing, and the one or two people who they took on from the client company had their salaries slashed and so they didn't really give a fuck.
There are two options, really
1) run. Run away quickly. Assuming you don't accidentally get a job somewhere with worse problems, it helps you *and* sends a message to the bean counters that there are costs beyond the old team's salary.
2) befriend the new guys and help them learn as much as you can. I can't promise it'll work, but in my experience it certainly seemed to make a difference to the underpaid, undersupported outsourced team to have *some* friendly faces who don't rage when things break, and respond well to clarifying questions about issues.
I had one boss that was great. Sure I got called for a few things, but he knew when to chuck in the towel, and when to keep the upper manglement off our backs when we had to recover from a case of intense idiocy... Funny, it's only been a year since I worked for him, some days it seem like 20 years ago, and some days it was only yesterday.
I once spent an hour and a half at the bottom of a slope watching all my mates ski on (initially) pristine snow while I did remote support for a piece o SW I'd had the misfortune of taking over mid-development some weeks earlier. Of course none of the other members of my team, who were not on holiday, could take the call...
Adding insult to injury, when I came back, I spent days knocking on doors until someone at my company "magnanimously" signed-off to pay me back the roaming charges
So you were on the ski slopes, back in the days of roaming charges, got it.
I am surprised you had reliable service. I would think service was a bit spotty. Dropped calls, incoming calls going straight to voice mail, that sort of thing. Let's not forget short lived cell phone batteries.
No, I am not suggesting a course of action, but possibly an opportunity for creativity.
/block
Someone once did that to me - the caller was told to "go away" in a very direct biblical sense and a formal complaint made to the only people that could have shared it.
They were told to make their request properly through the formal channels.
I've done nothing ever for them ever again - even when their clients would be paying me for their referral. Nor in my day job - the department they work in is toxic and I refuse to to deal with them. Doesn't help that they are openly critical of my departments whole service. Not the boss. SEP - painting it pink!
I once, many years ago, had my boss at the time call me on my personal mobile when I was at the bar in my local pub on a Friday night, to try to give me a dressing down, for a reason I can't even remember (but probably related to overwork and under-resourcing).
Since then, I have made it a policy decision to not provide my personal mobile number to work. On the rare occasion that it has been absolutely necessary, I have demanded written assurances that the number would only be used for that specific purpose, not passed on, and not kept for longer than required. My direct boss does actually have my personal number, but I can actually trust him with it, and I think he has possibly contacted me on it, once, via a polite SMS.
Now, if I get calls that I am not expecting from an unrecognised number, I don't answer them and I block the number. If I'm being charitable, and it's a mobile number, I might send them a "who this" SMS before blocking them. Anyone who needs to contact me urgently will have other means of doing so. Long gone are the days when a phone call gets treated as a priority interrupt.
The likes of Amazon who demand a phone number for their couriers can have good luck trying to call 00000000000.
Same here, I was in Lautenbrunen area way up on the hills when I got called by work while on Vacation. Luckily the family could take an hour walk while I fixed the issue (being the only storage Netapp guy left) fairly quickly. They still sing my praises after that one. Heck, I also took a call while sitting on a chairlift and solved the issue for my boss. He was *very* happy that day.
I've never, ever come across a situation where a senior manager said thanks carry on with your vacation. During my time in tech support / diagnostics I've been forced to carry on trying to recover data off a completely dead server, performing pointless reboots until 3rd line support arrived ad 07:30, driving into the office a 3:AM just so 'someone's working on the fault', had to baby sit an engineer through dissembling a £1m mainframe to get to a faulty fan, he hadn't received any training as 'field engineers wouldn't need it these machines are so reliable'. I was a software specialist in the diagnostic center, alone on nights but had the hardware manuals, he was alone on site with no documentation, he'd just managed to get to get to the part when a trained engineer arrived 10 hours later who managed to complete the work in 2 hours as he actually knew which order to assemble parts in).
It always seems senior managers can sleep better if they know a technician is awake and 'working the problem'
It might not be your area of expertise, but you are "available" (i.e. cheaper than the real expert).
You just reach the point when you feel you are making progress on the problem and...
Somebody in Manglement insists that you spend an hour on a Conference Call to discuss what progress is being made. You might get another 15 minutes getting back to the pre-call state, and....
Somebody in Manglement insists that you spend an hour on a Conference Call to discuss what progress is being made, and why the Service Level Agreement has been breached.
Andy A,
You have earned your stripes if your scenario has be experienced 'once'. !!!
If you have experienced it more than 3 times you are starting to work on your sleep reduction & life/work balance problem !!!
[Time to get another job where you are paid more and have less 'idiocy' to deal with]
:)
Had something like this happen.
Called into a meeting that lasted two hours by the clients management and told how to suck eggs.
Came very close to walking out of the meeting.
Afterwards I told the clients project manager that I nor any other engineer from my company would attend such a meeting ever again.
Now we send a manager with an engineer and charge for both. The manager either sits around and does nothing or acts as a buffer between the client and the engineer.
It is quantum management.
As long as you are providing them with updates, you can't solve the problem as that requires your full attention.
As soon as you give the problem your full attention you will be able to make progress and solve it.
There will be no management updates.
So it is either, but never both.
In a different life, I was one of the "Senior Security Incident Handlers" at a large financial institution. Large, as in > 150 billion USD capitalization.
The process to handle incidents was very well documented and maintained. There were maintained contact lists, conference calls at set times, and fully documented steps in the process. By being a dedicated "Incident Handler", I coordinated the needed work and reports, but mostly acted as a buffer between the people fixing the problem and the manglement screaming about it.
Of course, you would often find a new senior manager or junior VP who would attend a call and bluster demanding to take over the entire process, since they were *important*. Fortunately we also had a list of SVPs we could contact to come on the call and tell the blustering mangler to knock it off. We had good cooperation with these senior people: They knew the process Worked and trusted it. We never had to bring that hammer down twice with the same clown: They very quickly learned that the "Senior Incident Handler" may not have a fancy title and parking spot, but we had the trust of the highest level of management so they needed to back right off.
The trick was to have the entire process developed and document AND signed off at the highest levels. Then when it hit the fan, we just ran the process. Doctrine defeats Doubt!
You need a better Boss. The purpose of the Boss is to be a proxy for you in these meetings. You keep the Boss informed of the progress and when you will have more progress information, and he keeps the wolves at bay without demanding 15 minute check-ins.
That's when you need this script:
"I can fix the problem, or I can answer the phone and tell you what
I'm doing about the problem, except that while I'm on the phone,
I'm not doing anything about the problem. Choose one."
and as others commented, it's your boss' job to keep senior management busy while you actually fix the problem
Been there, seen it, lived through it.
Only my backup nightmare was due to someone changing a UNIX backup device from st0n to st0 at some point, without change control, causing portions of data to be erased. Long time ago, before there was any commercially available backup software from what I can remember. I remember finding out the backup was corrupted, affected multiple backups going back weeks, and being summoned to the boss's office to explain. Luckily for me I was able to point out what the problem was and that it wasn't anything I'd done!
Fun times.
I'm at work today because two of my team quit last week, they were on temporary contracts which I'd been trying to get made permanent for six months. I've warned executives repeatedly that if these staff go it will cause major problems for us in the new year and have a huge impact to our work plans as recruitment is painfully slow and the specialist systems we operate require many months of training. So realistically at least 9 month delay.
No decisions made over emergency December meetings before the execs all left for their holidays. My staff both decided to pack it in last week, I can't blame them - they have found alternative permanent roles elsewhere and despite wanting to stay with us because they like it here - paying the mortgage has to take priority.
If I'm lucky they will both re-apply if/when the posts are ever made permanent but I've got my doubts.
So now my planned Christmas break has been cancelled at short notice so I can cover these two posts that end shortly anyway. Got to love the NHS.
Put a manager hat on for just a second.... What exactly is the problem?
>"I'm at work today..."
Sounds like management has staffing covered just fine. There are "alternative permanent roles elsewhere" which have competitive compensation because they are "paying the mortgage". And yet...
>"I'm at work today..."
My friend, it seems to me the solution here is obvious. And if you choose to stay, there will never be a better time to renegotiate your contract than right now.
Not suggesting you do this if it's a customer/patient facing system, but let the system fall over. At the inevitable emergency blame session, sorry solution finding meeting, produce all the emails where you said you needed those staff members signed up as permanent before the management all went on holiday, and then adviser them that those staff have moved on because the management didn't do their jobs. Be sure to be the one taking the minutes of meeting. And then make sure those minutes, accompanied by the full email chain, find their way a long way up the management chain.
This might seem like a great way to kill your career, but if you're management is going to screw you so badly over something as simple as making staff permanent, then all your doing is creating more stress for yourself. You may as well go out on a high...
Take a recorder of some sort - and keep multiple backups.
Sure, policy is no recorders, no lawyers, etc. - what are they going to do? sack you?
You have one chance to keep this right.
DOI - been in lots of near career ending meetings in the NHS. Got tapes of them all. The notes/record bears little resemblance. "the formal record" - playing the recording blows this out of the water.
Powder dry for the tribunal.
You DID BCC all your emails to your personal account didn't you, as they are really poor at finding stuff (ICO focuses minds)
Sorry if you are logging in remotely to do so, but you really need to.
Got the powderburns. Accumulating evidence.
to the day after heart surgery when my boss called, not to see how I was (sore , pissed off, and full of morphine), but they had a problem that needed sorting and only I had the ability to sort it(which was a lie as I'd left instructions with several people about all sorts of problems including the one he was complaining about)
When I got back to work I blamed the drugs and pain for me telling him to f*** off
I had a manager in the same vein. I was new to the job, and had gotten a call from one of our plants about a control system issue keeping the production line from running. Boss wanted me to call the experienced engineer who was out that day for his father's funeral. I said no, I'll figure it out. He said "he'll welcome the distraction". I refused. It may have taken me 30 minutes longer, but the issue was resolved without making that call.
I worked for that manager another 15 years. In fairness his job was his life and he just didn't comprehend any work/life boundaries, applying the same rules to his personal life he asked of others. He also took no for an answer if you stood your ground.
It was corrupted in the server. And the data on the previous week's backup has been deleted when they automatically tried to backup the corrupted data.
Though from what I understand, data from two weeks ago should have been properly backed up and safe, so two weeks of data loss instead of one.
Ah, backups and the holidays. Back around the turn of the millennium, I was responsible for the care and feeding of an old AS/400 system. I was way in over my head, but the powers that be weren't going to hire a full time admin for that machine when users were moving from dumb terminals to PCs (running Windows 3.1) and backend was moving to Novell servers. My role with the big iron was more palliative care than administration.
Of course, the MRP software still lived on the AS/400, so it was still business critical, so when a hard drive died it was urgent to get the machine restored quickly. Two or three years in a row, we had failures right around new years eve, to the point that I joked that the machine and I had a standing date for NYE.
We had a service contract with IBM, so when a drive died it was a simple matter of opening a ticket and coordinating a time for the service engineer to be on site. I'd usually pull all the covers and vacuum out the system while it was down and we were waiting for the drive to arrive.
Swapping the hardware was pretty quick. For you youngins, a hard drive on this system was a 3U 19" rackmount box that held a whopping 400 MB. Our system had a rack full of those (I think we also had 2 different capacities).
Once we physically got the hard drive swapped, it was restore time. I'd do the ritual trek to the fire safe (stored in an adjacent building) to retrieve the tapes. IPL (Initial program load, IBM speak for "boot from tape") was done from a set of 4 1/2" tapes. Each one ran for about an hour, so I could catch a quick nap in my chair before swapping them. The data was saved to an 8mm tape, so much less swapping was needed.
I think our record was about a 24 hour turnaround from the time a drive failed until I had the machine back up and running again.
No emergency callouts here but noting the various feuding family and no Christmas cheer comments. We have both Christian and Hindu family members. Far from non-seasonal cheer, that means we get to celebrate both Christmas AND Diwali, (plus the other main festivals of both religions).
Oh, and the (often very extended - for several years we had to hire a village hall) family get togethers are great.
2012 nd just went to bed on Xmas Eve, I'd had about 4 hours sleep and then at 1.30am on Christmas Day the phone rang....
Ops: "This process has failed and we don't know what to do?"
Me: "What's the error message?"
Ops: "Says blah blah blah..."
Me: "For the love of fecking Christ! Ignore it! Merry Xmas!"
Phone down.I swear to this day they only called me to prove they were still working at Xmas.
I may have commented on a similar tale before, but about 10 or so years back I had a call from my minion who was manning the fort (just the two of us) while I was on my way through France as there were network oddities that he couldn't fathom. So I spent a couple of hours drinking coffees and using the free WiFi in the medieval city of Carcassonne while I connected back over VPN and remotely dumped network traffic and switched bits of network off and on to pinpoint the problem. Nice location to just sit in a fairly empty outdoor cafe courtyard enjoying sun and coffee... I don't think I even claimed for the coffee on expenses!
To this day I still don't actually understand what the network was doing. In short, Cisco switches were passing IPv6 traffic over a spanning-tree blocked port, causing a network loop and associated storm. Resilient link from the stacked pair of 3750s at the core to two units in an unstacked (as they didn't support stacking and thus couldn't do an aggregate to different units) cabinet of 2960s, using spanning tree to block one of the links. The port correctly showed as in BLOCKED state and all config was correct. IPv4 traffic was not passing through it, but network monitoring and port mirroring showed that IPv6 was passing. It had worked fine for several years prior to this but waited until I was driving to the south of France. I administratively disabled one of the ports rather than using spanning tree until I got back, which sorted the problem. The faultreoccurred a month or two later, after which I just kept the spare link disabled with a note to enable it should the live link or the switch it was connected to fail. Was clearly something triggering it which wasn't present in normal circumstances but never got to the bottom of what that trigger was, but it did instill in me a fear and mistrust of spanning tree...
I managed to avoid a full Boxing Day of shopping with the family after Leeds flooded around 10 years ago taking down the Vodafone DC built stupidly close to the river Aire. That wasn't my problem, but the gov department I worked for used the VF DC for a service so I had to spend about 3-4 hours in a quiet McDs with a cup of tea on a conference call earning double-time while they spent it faster than I earnt it...
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
