Re: Well that was an invisible problem
It's pronounced to rhyme with "lorry", but I get that a lot... ;-)
(I wrote this up a while ago for submission to El Reg, but was never quite happy with it. All names changed to protect the allegedly innocent.)
My first job was in the mid 90's, working for a big company with serious political dysfunctions. One of the best demonstrations of those issues is something I refer to as "That time I was told to steal a £14,500 switch"...
I was working as tech support in a building that was full of outsourced telephone support desks for some very big IT names. One of my colleagues - let's call him Dave - had just moved from working on one of those helpdesks into a project management role. We got an email from Dave saying that a new network switch had arrived, and could we please locate it and configure it?
(In the mid 90's and a network switch was an exotic bit of kit. These were the heady days of the new 100Mbit "Fast" ethernet. Switching wasn't a feature on hubs as it is today, it was a function for dedicated hardware. And in this case, it was a 12-port 10/100 3COM switch which, including taxes and delivery, cost a little over £14,500.)
Dave explained that a new helpdesk was going to go live, and there were concerns over the performance of the database server that handled scheduling of hardware engineer visits. That was millions of pounds of business each year, and therefore probably the most valuable server in the building - possibly in our division. Analysis had shown that the server's performance was fine, but it was homed on a network with at least 100 clients and yet more traffic from a WAN uplink. Network congestion was most likely the issue.
My boss dispatched me to find the switch. It wasn't at Dave's desk. It wasn't with reception/facilities, who said that they had delivered it to Dave's desk. Dave had only recently changed jobs, so it was likely their information was out of date - I went to check his old desk...
And it had been there.
But it was now with Terry.
Terry was a man of initiative, and had decided that as this had been delivered to Dave's old desk the switch therefore belonged to Terry's department. A short but futile conversation left me certain I wasn't going to leave with the switch, as Terry had repurposed it as "something I can put on my CV". (He didn't quite phrase it that way, but the meaning was clear to us both.)
To complicate things, Terry's department was a flagship project. Big name client, used as a case study, on the tour route for all visiting potential customers - they had serious political clout in the company.
Dave was on leave, and this was 1996 so he didn't have a mobile phone. But after some calling around we managed to get hold of him, and he confirmed that the switch was ordered under his new budget code. He was very unhappy to hear that "his" switch has been poached. It was made clear that there was a deadline for setting up this new helpdesk - his main concern was that Terry might plug the switch into his department's network (they managed their own IT to some degree), and that would make it hard to get back due to their political capital.
My boss assured Dave that this would be handled. We hung up the phone, and I was ordered to go and steal the switch.
Not exactly how I'd planned my day.
It was approaching lunchtime, so I slid round to a helpdesk adjacent to Terry's, and began to very slowly diagnose a non-existent fault on a PC. The moment Terry went to lunch, I pounced. Swiftly repacking the switch and disconnecting it from a serial port, it was soon retrieved. Now we had to decide what to do with it. My boss decided to lock himself in our small office, and read the manual. I was sent out to distract Terry, and do all our pending jobs in the process. On my way out of the door, I grabbed the empty box.
"What are you doing with that?", my boss asked.
"Decoy" was my response.
The ground floor server room was a repurposed meeting room - so it had glass windows. I dashed in, sat the box on the workbench, and then left - making sure to lock the door as always.
I spent much of the afternoon running around the building in as unpredictable a pattern as possible. I kept dropping into conversation that we were more busy than usual, and I had to go to $department next - knowing full well that I was going elsewhere. On returning to one helpdesk, I heard that Terry was looking for me. Eventually I bumped into him, and found out that he too had been busy - he knew the switch was in the ground floor server room. Eager to help, I went to fetch the key - but never returned, having been diverted by a faulty computer on the way. Anyone who's done desktop support will know the kinds of distractions that can drag you somewhere unexpected. That afternoon, I made sure that they all did.
At six in the evening, I dropped in to our office. My boss was still reading the manual. I was sure that Terry would have gone home by now - his helpdesk closed at five - but I headed back out on the distraction trail anyway. At seven thirty, I got paged (remember pagers?) and returned to our tiny office to hear the plan my boss had come up with. Then we went home.
The next day, shortly past nine, Terry dropped by our office.
"I want my switch."
"It's not yours."
"It's ours, we're a flagship desk, and I want it."
My boss adopted a soft, conciliatory tone. "OK, let's go and fetch it."
We walked to the server room, unlocked the door, and ushered him in.
"There it is. Help yourself."
Terry was both livid and crestfallen at the same time.
My boss hadn't just read the manual the previous day, but had also written and uploaded a configuration for a switch he'd never seen before.
We'd been in since before six, and had racked and cabled it and cut all services across to it - a WAN uplink for the building, a link for each of the local hub stacks (remember 3Com 100Mbits backplane connectors?), a link each for the Exchange, IIS and File/Print servers... And a link for Holly, the multi-million pound database server.
A single network cable whose traffic was worth more money than most people will earn in their entire career. If Terry wanted his switch, all he had to do was unplug that cable.
Terry left without his switch.
That long day and following early start was worth it. Not just for the satisfaction of a job well done, but in other ways. For example, one of the helpdesks ran Doom/Quake servers at lunchtime to help relieve employee stress, and apparently the switch made a noticeable difference to their performance. I was gifted many, many free beers for that.
And finally, I should note that Dave showed great promise as a project manager.
He took all the credit for our work.