Epson
Is that the London suburb just outside Fujitsu?
Welcome again to On-Call, our weekly look at readers' trials and tribulations when asked to go out and fix stuff for clients. This week, reader “SH” shared the story of his time working for an amusement machine operator as a service engineer. SH spent his days maintaining fruit machines*, pool tables, juke boxes and video …
I wasn't exactly on-call, but I had spent many months on-and-off trying to resolve an intermittent bug which just occasionally caused a customer system to miss a critical security-related message. We couldn't work out if there was a real problem, or just less-than-competent local support staff. Eventually the irate customer insisted that if he didn't see a development engineer by the end of the week he was returning the equipment (well, he didn't put it quite like that, but you get the idea). Fearful of losing future deals, my boss put me on a plane.
To Australia.
One of the nice things about booking economy class tickets half way around the world at 3 days notice is that you have to pay full fare, and so I got bumped to business class.
A day's work on-site, with code I knew well, finally located a real bug, a typo. Unfortunately it was in code I'd written, so my boss was not well pleased when he read the trip report that went with the expenses claim...
Ah, those were the days before the internet bubble burst. These days they'd just have written off the deal :(
Well not exactly on call, but did get sent to survey sites in places Like the Shetlands, Hebrides etc in Winter, which meant a one and a half hour survey, few hours of CAD drawings, used to take quite a few days because of storms marooning you there. (If Ferry crew get off the boat looking pale, then as far as the architect I was with was concerned there was no way we were getting on). Luckily our employer was of the opinion that they were making so much money from the work we could drink during the days we were stranded on the company.
A company I used to work for got a contract for support of PCs in various radar installations round the country. My manager had an easy area to cover, but his friend in Scotland had it tough. The company had decided that Aberdeen was a good centre to deal with the whole of the Highlands and Islands.
We looked at the ferry timetable and worked out that an engineer could get on one of the 3 ferries a week, spend several hours at sea, drive half an hour to the site, press the "on" switch, then see the ferry disappearing as he exited the building - assuming the weather allowed the boat to sail at all.
There is no chance of hitting any targets when you have an average call duration of 3 days.
Your friend may have visited some of the places I did. I can tell you trying to draw a site sketch while standing at the highest point in the Shetlands next to a radome in December the wind chill was brutal. Upside is if you are stuck in those places the locals are so bored of spending the winter looking at each other the pubs are really welcoming.
Also do not go drinking with oil rig workers who have just got off the platform and fancy a drink.
"So, the island's big enough that it's a half-hour drive from the ferry to the site, but his vision was good enough that he could see the ferry from there?"
island roads tend to follow the coastline rather than straight lines, so he could have been looking across a bay..........or alternatively looking down from a mountain top (where else are you going to put a radar?)
In a previous job working in support, we FAXed some instructions to a users *Systems Manager* to shutdown some Databases and OS and then reboot their SCO Unix box (so you get an idea of the vintage of this story, right). The instructions went on for a couple of pages, but on the header page, I wrote a note asking them to return the FAX to me as it was the only copy we had - which they dutifully did...
"To Australia"
Ahem I got a call one morning from the MD of our South African company. "Our major customer (a bank) has an unresolved problem. Our engineers can't fix it. The customer is going to throw the machine out if it isn't resolved by the end of the week". It was already Wednesday.
I have certain skills - like finding the guy who wrote the suspect software. "Fancy a trip to South Africa to fix it" I asked nervously. This was in the days of white rule and the anti-apartheid calls to boycott. I was asking him to do something I'm not sure I would do myself though I was already an unwilling accessory. Fine he said (it was winter here and summer there as he advised me).
"Would tomorrow be OK?"
"I'll just check". 5 mins later, "Yep, I'm up for it".
What a relief. Called travel department and he was booked on the next day's noon flight out of Heathrow. Rang back the MD to assure him the only man who could sort the problem would be there the next day. He could tell the bank to relax. Sat back in relief that I had triumphed again and everybody knew it.
The guy rang back a couple of hours later. I forgot to say I haven't got a passport (nerds in the 70s didn't all have 'em). S**t - I could see my P45 being enveloped.
What to do? It was already late afternoon. Then I remembered that as the company did a lot of secret work with obscure government departments we had a secret office that dealt with sensitive matters in a very quiet way. I rang them, I explained the situation to a reassuring ex(?)-military voice.
"So it is a matter of national interest that we maintain this strategic economic link at all costs?"
"Yes", I spluttered
"Leave it with me I'll ring back in an hour"
He did. "Ring the chappy and tell him to drop into Petty France (then the passport office HQ) on the way to the airport. Its all sorted.
"Don't you need his details, photos ..?"
"No, we have that already".
And yes that guy made the flight, fixed the problem and that bank continued to bust sanctions with the aid of our equipment. My job was not only saved but enhanced.
I made an inadequate donation to the AAM in penance. It is something I am not proud of. But I do remember it as an example of what government can do when it wants too. And how scary those people in the shadows really are.
Not Australia but Italy. I had a contract to write some reporting S/W for my client's industrial control system running in their client's factory in Italy. The development was in my client's office in England but I had to go out to install it.
Once I got there I couldn't get through a run of the reports without random crashes. On reboot fsck kept leaving files in lost+found containing random fragments of memory contents. The end-user client didn't want me to leave until it was seen to run and it was still crashing when I should have left - and I was running out of Lira. I'd also had a call from an agent to see a new client on the following Monday with a view to starting contract on Tuesday. Finally the suite ran & so did I. I was told the consequent hardware call identified a bad memory stick; maybe without the extra S/W running the machine never used that area of memory.
I got as far as "Once I got there I couldn't get through a run of the reports without random crashes." and thought hardware - probably overheating CPU, memory, or hard disk.
I guess software engineers look for bugs, and sometimes it requires a step back to look at the environment it's running on.
That's easy. You turn it on to make sure it's working & then turn it off again because users will only mess things up if you leave it switched on.
Actually it was off in the first place I think (lightning storm seems to have upset it from memory).
But yeah should have added after twenty minutes switch it back on again, that way they get a brief spark of internet before it goes down again and then comes back for good. So ensuring your leet skills for your truimphant drive back to Scotland after bringing a factory back online. :)
I have a mains radio in one of my barns, If I've not been in for several weeks I can turn it on and get four or five steps away before it fades as the well made capacitor drains into the device. And I have to go and turn the mains on...
I feel this is the reason why most digital devices are full of capacitors made of wishful thinking - so you can turn them off and expect them to reset.
Sounds like my amp, though it earths it's self so instead the caps drain with satisfying thump from the speakers (vintage pioneer upgraded with new speaker terminals, resoldered main boards and single strand wiring) still better than couple of grand modern amps.
Sometimes you just turn it on.
This is a few decades ago, and it didn't happen to me, but to a colleague. I know, I know, it always happens to a colleague, but this time I saw him drive away, return the next day and I saw the visitors report.
So the customer is in a frenzy because they want the printer to print, and it doesn't print. Is the green light on? We are assured the green light is on. Are any other lights blinking? Is there enough paper. No, yes, yes & yes.
So we send someone over. The green light is not on, the printer is not on, it is turned off. Also, the plug is disconnected from the socket. Customer explains that perhaps he saw the reflection of the sunlight. Except there's no window in the room. Anyway, 2 days work and one embarrassed customer.
A couple of months ago I visited a customer and had to print a report there. He insisted that the report should be printed on that thare printer. Since it was the default printer that everybody used in the office all the time.
Printer was turned off, the electrical cord was missing, there was not a single sheet of paper in it. It was covered in dust, it was clear it hadn't been touched in at least year.
Took me 1 hour to get it working again, changed to report to charge the extra hour. Customer pretended to be surprised. I pulled procedure on the prick with enthusiasm.
Sometimes you just turn it on.
So the customer is in a frenzy because they want the printer to print, and it doesn't print. Is the green light on? We are assured the green light is on. Are any other lights blinking? Is there enough paper. No, yes, yes & yes.
Pretty much this exact scenario happened to me when working for an ILR station. News stories would come over a satellite link as a serial stream straight (well, almost) to a dot-matrix printer that used fanfold and was about as robust as a typical tank.
Unfortunately, being somewhat junior, there was no call-out fee, no reports to file, no comeback on the numpty who couldn't see the power light was "off", let alone that the thing wasn't "on line". As it was a Saturday, I was bleeped out of bed at some unearthly hour (5am?), spent 10 minutes on the phone explaining which lights to look out for and which buttons to press "yes, that's ok, yes, I've pressed that" and then 30 minutes driving into town, one minute to walk from the back door up to the newsroom and a total of, oh, I dunno, five seconds (about as long as it took to walk across the room) to spot the missing lights and turn the blasted thing back on whereupon page after page of overnight news and sport started spewing all over the newsroom floor.
Certainly beat that time when the flat roof was being replaced and the company doing it had just finished scraping the old stuff off when it started raining, so they decided to go home.
M.
Vercotti: [...] Anyway I decided to open a high class night club for the gentry at Biggleswade with International cuisine and cooking and top line acts, and not a cheap clip joint for picking up tarts -- that was right out, I deny that completely --, and one evening in walks Dinsdale with a couple of big lads, one of whom was carrying a tactical nuclear missile. They said I had bought one of their fruit machines and would I pay for it.
2nd Interviewer: How much did they want?
Vercotti: They wanted three quarters of a million pounds.
2nd Interviewer: Why didn't you call the police?
Vercotti: Well I had noticed that the lad with the thermonuclear device was the chief constable for the area. So a week later they called again and told me the cheque had bounced and said... I had to see... Doug.
This post has been deleted by its author
This post has been deleted by its author
Well, given the note above that it would need to be an ordinary fission bomb only:
The warhead is indeed rucksack portable, weighs 51lb. I do like the "Handle with care" label. I do dislike the sheer raging insanity of the whole nuclear weapons program that made it possible to make over 2000 of these things.
We frequently get support calls raised with us by the end user rather than the companies 'technical' team. They're always a bit miffed when you ask them what they've done to fix it themselves. Explaining they're going to get a bill if we come and unplug then plug back in a phone generally makes them try themselves. Never get a thanks from them.
One customer refused point blank to pay for our time once, having taken a server down themselves which was working to try to fix another issue. The person who had done it was on holiday and when they got back freely admitted it was them.. Nice bill for the time taken arguing with them as well.
An HVAC engineer friend of mine once had an emergency callout where the air-con pump had stopped. Knowing the device involved, he hit it with a lump hammer and it started working again.
On presenting the customer with a bill for £100 emergency callout, he was told "we're not paying you £100 for hitting it with a hammer"
He said "OK, let me re-do the bill" and presented them with a bill for "Resolving issue with pump by hitting it with a hammer: f.o.c.; Knowing exactly where and how hard to hit pump to get it working again without causing damage: £100"
They paid.
@Doctor Syntax
SU electric fuel pumps had a set of mechanical contact breaker points (like an old distributor) which liked to stick open or closed which stopped the pump running. Quick bash of the top of the pump and you're away.
(for those who are really nerdy, look at the first item on the 2nd page of this: https://www.holden.co.uk/cataloguePDFs/cat10/Fuel_Air.pdf )
Remember the mechanical ones, not really? I've taken the electrical ones apart and put them together again for fun (replace valves and clean contacts)
Once the ignition was on they'd start ticking away trying to push fuel to the carb float chamber - this was in the days before return loops on the fuel line.
Generally they were sited somewhere inconvenient - up under the rear wheel arch - where they would get covered in mud and underbody spray.
"I remember hanging out the back of a Mini (45 years ago) banging the fuel pump - was on the Marylebone Rd in London - in rush hour."
A bit earlier than that I had a field trip in N Ireland. I had the hired van with all the kit in it but no heater. A few others were supposed to be following in the departmental Mini. After a very long while they got there. Having spent a few miles periodically driving over the roadside verges to jolt the pump into operation they'd given in and taken it into a garage to get it fixed. Whilst it was up on the lift they could see where lumps of the cooling fins were missing from the sump. That would be where we'd driven it up another mountain a few weeks earlier.
"Remember the mechanical ones, not really?"
The toyota pickup/hilux with carburettor (with 20R or 22R engines, and 18R and before I bet) have a mechanical fuel pump, mounted conveniently at the front of the engine head. It's almost trivial to replace, but it hardly ever fails. Though after market ones as always may have varying quality so it's best to get an OEM one (Kyosan for example).
"a mechanical fuel pump, mounted conveniently at the front of the engine head."
Our wonderful British Leyland produced the 1.8l Diesel on which the mechanical fuel pump was so mounted that if the diaphragm broke, fuel poured into the sump.
The effect of the hot fuel reaching the sump was for fuel vapour to make its way up past the rings into the cylinder, causing runaway (remember that Diesels do not have throttles so taking a foot off the governor has no effect.)
The fuel dilution of the oil then causes the bearings and rings to wear very rapidly indeed while the unfortunate owner is still trying to work out how to stop the damn thing, which eventually does stop due to complete mechanical seizure.
It didn't happen to me...but after I got an account from the person it did happen to, I replaced the damn thing with an electric pump.
> An HVAC engineer friend of mine once had ...
Ah, now that reminds me of a fairly recent incident with a customer - in which I got to polish my brownie points a bit more.
Said customer had built a new place a few years earlier - having outgrown their old offices. This new one even included a dedicated room for their IT gear - and it got the inevitable air con unit which was installed in exactly the place I suggested it wasn't installed (ie where it was easy for the installer, not where it would work best.) But in spite of this, it worked well enough.
After a couple of years the air con broke down - so the emergency fix was to prop the door open and position some fans to move air around using the large and currently unused top floor as a heat sink. So of course, they called in the air-con people to fix it - and this is where I get involved.
I get a phone call to say they "have the wrong sort of roof" ! WTF ? Customer is relaying information he's been told by the air-con guy who is adamant that this air-con unit cannot cool the room because it has the wrong sort of roof and it's getting too hot through lack of insulation from the heat upstairs. I had a nice line of choice adjectives to call him, but (unusually for an Aspie) managed to keep them to myself. I might add that by this time, the customer had already sent someone down to the builder's merchants for a roll of loft insulation to put on top of the server room ceiling (which I'll point out was boarded top and bottom with a 4" sandwich of glass fibre insulation between the timbers).
Now, no matter how much I pointed out that this very same air-con unit had been working fine for 2 years, including much hotter weather - it's the room that's at fault. I'd even checked the specs of the unit before it was installed as I know many units cannot cope with the "dry heat" in server rooms.
This had been going on for 3 or 4 days with the air-con guy adamant that the unit was working fine but the room was faulty. So I rang the air-con company, spoke to the service manager, somehow managed to avoid unloading a load of adjectives about his service guy, discussed the matter and found the service manager in agreement that it didn't sound right.
2 hours later - the call came back that the unit was faulty (stuck reversing valve) and the room was now cool again !
As an aside, that same customer called up up another time and were "expressing great displeasure" that the faulty UPS we supplied them was beeping madly - you know, all our fault, why did we sell them a pile of crap, why didn't we spot it was faulty before it failed, and all that. Oddly, we never got an apology when after a few questions it turned out that someone was working in the upstairs space and was a bit cold - so they plugged a fan heater into the nearest socket they could find. Yes, that socket came off the UPS which was now overloaded. We had a bit of a chuckle about that !
It is surprising how many of these fit the bill:
FOC may refer to:
Father of the Chapel, a trade union official this is the only one that doesn't (unless it was housed in a chapel.)
First order condition, an equation used to calculate a minimum or a maximum
Flag of convenience
Free of charge
Workers Front of Catalonia (Front Obrer de Catalunya)
Freight Operating Company UK Train System code (maybe?)
Full operational capability a term in military acquisition
Festival of Code, a yearly computer programming event for young people
IATA code for Fuzhou Changle International Airport (you could make that sound like FO charlie with an accent... (Ok two.))
Communications and technology
Fiber optic cable (3? damn!)
Firm order commitment, the scheduled installation date for telecommunication services or scheduled transfer date in local number portability.
First Class CW Operators' Club, an Amateur Radio Morse code organisation
Field-oriented control, "a maths-intensive technique for controlling brushless dc and ac induction motors"
the Faint Object Camera, as used in the Hubble Space Telescope
Fraction organic carbon in the field of hydrogeology Doesn't make a great deal of sense but not a fourth.
Fiber Optic Components
Entertainment
FOC (album), debut album by Australian rock group Far Out Corporation
Flag of Convenience (band), rock group formed by former members of Buzzcocks
Transformers: Fall of Cybertron
Flight of the Conchords, a Grammy Award-winning New Zealand comedy duo composed of Bret McKenzie and Jemaine Clement.
Star Wars: Empire at War: Forces of Corruption, an expansion for the game Star Wars: Empire at War
"We frequently get support calls raised with us by the end user rather than the companies 'technical' team. They're always a bit miffed when you ask them what they've done to fix it themselves. Explaining they're going to get a bill if we come and unplug then plug back in a phone generally makes them try themselves. Never get a thanks from them."
Living in a small village in North Wales I seem to have become the defacto IT support service for everyone. I've learned that making them so "something" towards sorting a problem (finding out their email/ISP account password for example) does seem to sort the genuine cases from the idiots.
I used to work freelance giving IT support to schools (something I now do permanently in just one school). Each school would book a day a week or similar and I'd go in and sort out all their problems.
I was once paid a full day's fee - on emergency rates after swapping another client's day for the emergency call-out - to go to a particular school.
"The server isn't working, we can't get anything working, you need to come in".
So I arranged a last-minute day-swap with my other clients, telling the school in question that I would only be fixing the emergency and then leaving (that could work for me or against me - I could have been there all night).
I traipsed across town in the middle of winter, freezing cold, at a mad rush, got to the school, and - indeed - all the clients were down and couldn't see the server (the school was quite small, primary, and this was back in the days where schools were just starting to run their operations on computers). It was so cold, I left my coat on while I started looking at things.
The server being down meant that they couldn't pay wages, register children properly, etc. so it was quite critical. There was no "server room" (IT things were too new to them for that), so the server sat underneath a workspace in the main offices, tucked out of the way.
I could hear it spinning, it was certainly doing something, but nothing was working. I turned the monitor on. As it warmed up (yes!), there was a single white line of text on a black console screen (AARGARGHH!). It was at this point that I was about to reach for the backup tapes when I realised what the message said:
"Press Enter to continue..."
I pressed Enter. The machine did a quick chkdsk (no faults), and then booted into Windows. Everything started working.
For some reason the server had gone off and then sat in the BIOS until someone confirmed the boot. Confirming the boot was enough to fix it.
At that point I'd been paid a full day's money at emergency rates to press a power switch and tap Enter.
The rest of my investigation consisted merely of why there was an extension lead near the server under the office worktop, strung under the worktop to where the office girls sat, that wasn't normally there. Then I found a suspiciously warm electrical heater tucked behind a chair at the other end of the worktop, unplugged.
Yes, they'd got cold, looked for somewhere to plug in a heater, and decided the server plug was one they could pull. The server went off, or the fuse blew, so they then panicked, unplugged it, hid it, and had said nothing while the head and other staff ran around like headless chickens wondering why nothing was working. When the power was restored, the machine had a BIOS setting (it may have been a duff clock battery from being under there so long, I forget the exact reason for it) that required you to press Enter and told you so. But the monitor was always off to discourage them trying to log into the server directly so they couldn't see that.
However, going home before 9am, after only pressing a button and tapping enter, having been paid more than norrmal, and having happy clients - you can't beat that.
When at Uni, I provided first level support to a department. One afternoon I received a call that a professor's computer had a virus. I dutifully wander across to his office and asked if anything had occurred since it was last working correctly. The reply was nothing and described the problem as keys not working or two letters (e.g. R & T) appearing when a single letter was typed. I verified this, but it didn't feel like a virus, so I replaced the keyboard. The problem was fixed.
Talking to his secretary on the way out I found out that he had left his window open during a dust storm and had asked her for a bucket of soapy water earlier in the day.
It's not just school staff, I think it's common to anyone who sees a computer as a necessary evil rather than a valued tool, and therefore doesn't take as much care of it as we would hope.
At the aforementioned radio station I washed plenty of keyboards under the tap in order to get all the sandwich crumbs and sticky sweet congealed coffee out.
I had a major victory with another office wonk who had a floppy disc containing "vital data" that wouldn't work any more. "How long since you last used it", "erm... a couple of weeks", "and you've never thought to back up this vital data?", "What's a backup?"
She dredged the disc out of her desk drawer and it turned out to be encased in dried hot chocolate (i.e. the accident had happened quite some time ago). So well encased that I couldn't open the slider (3.5" disc) and the disc inside was glued to the paper runners.
One sacrificial disc casing and a lot of running under luke-warm water later, I recovered very nearly every file from that 1.44MB floppy. Fortunately the one or two files that wouldn't copy weren't "vital".
And then there was the time one DJ on his last show spilled about half a pint of cider into the (almost brand new) studio desk, soldiered on and didn't tell anyone. I was called out at 11pm by the guy doing the following show who was struggling with two CD players and two microphones out of action but even so couldn't be persuaded to hoik his boxes into the adjacent, spare, fully-functional studio and preferred that I dismantled the desk around him, live on air. Washed most of that out, too, but did use quite a lot of our stock of spare parts for the Penny & Giles conductive plastic faders.
Bloomin' expensive cider, and no comeback on the culprit who was leaving anyway.
M.
The small Scottish site had a problem. Someone dug up the road outside and hit the power mains, so everything went onto UPS, but was then gracefully shutdown when no power came back on after 20 minutes. Unfortunately, it didn't come back on until the next day.
When the power did finally start flowing, the UPS had nothing left, so nothing in the cabinet would start. It should have restarted automatically, but for some reason we never uncovered, it didn't. The site was too small to have an IT person, so we tried to get one of the staff there to turn on the UPS. They refused as they were worried that they "might get electrocuted"!
I had to phone around some local electricians; a couple refused, but one guy was prepared to tackle this monumental task. He actually apologised for submitting the bill; 30 mins of his time, minimum charge, plus driving 10 miles (with VAT, I think it came to £34), just for pressing a button on the front of the UPS. As I pointed out, it was cheaper than me flying up for the day.
Hands up if...
...You've been the one to have to do a long drive to switch a box back on after you got confused between "Restart" and "Shutdown"
Never sure if it's more embarrassing explaining to the boss why you suddenly have to drive to the other side of the country (UK only), or explaining to an onsite technician that you fucked up, and please can he wander along and restart it...
...You've been the one to have to do a long drive to switch a box back on after you got confused between "Restart" and "Shutdown"
I have never actually had to do this. Largely because after a close call one day where I got as far as the "are you sure" prompt (and actually had my finger on the enter button) I have set a group policy to remove "shutdown" and "restart" from the start menu of anything server like running windows, and do my reboots via the remote shutdown interface (run -> "shutdown -i").
Not quite... But I have had to phone a guy to get him to unlock the office so I could get the keys and phone another guy who phoned a guy to get security to let me in to the hosting suite. Because I accidentally triggered a remote shutdown on a live server.
It was 3am and I was in the wrong terminal window...
Because I accidentally triggered a remote shutdown on a live server.It was 3am and I was in the wrong terminal window...
This is, IMO, the one practical use of command blacklisting in the sudoers file.
I had a group of users with a group of servers. Against my advice, most of them had sudoer privilege. And we had regular shutdown accidents...
I blacklisted the shutdown and reboot commands - that won't stop the users being able to perform the fundtions, of course, but it does stop the doing it accidentally...
Vic.
Posting anon for obvious reasons ...
I was within earshot when one of the helldesk guys was talking a customer through power cycling a frozen server. I think the instruction was along the lines of "press and hold the power button on the bottom server in the rack" - a simple enough statement for someone who knows the difference between a server and a UPS. Hmm, I think you're ahead of me here !
Of course, the customer didn't know the difference, and so pressed the power button on the UPS - at which point things went a bit quiet making the ensuing language (on both sides) audible.
I've done worse than that. I was at a large corporate establishment, needing to shut down a system running a bespoke application.
In front of me was a large desktop with a monitor on top, and a keyboard and mouse in front of it. It was displaying the application. I initiated shutdown, and it worked fine, sitting at the shutdown complete message.
I pressed the power button on the desktop. The desktop switched off.
The shutdown complete message stayed ON.
'oh, that's not the server' said the customer 'we put the keyboard/monitor for your server on top of this desktop because it's convenient. Your server is next door'
Always ask, even when it seems a stupid question.
Years ago I cleared the routing table on a firewall, which of course meant I could not connect to the firewall. Luckily it was only about 2 miles to the data center so I hop in my car to drive down to the data center to fix thing. The cop didn't think doing 50 in a 35 mph zone was a good idea. I was early on a Sunday so there was not much traffic. He must have been in a good mood as I didn't get a ticket. I don't know how I could explain that on the expense report.
I've never done the wrong server restart but I've got a very similar t-shirt to that, and not so long ago either!
The company I work for had a 3rd party system that I was the SME for. In a nutshell said system consisted of several Windows servers running services which would connect to our IVRs and telephony systems and grab the CTI events being chucked out of said systems, then write them to a very large SQL database. My company would then use that data for popping customers details on to agents screens as the customers call hit the agent's phones.
Just side tracking a bit, screen pop, for those who don't know, is very widely used in contact centers as it can give better customer experience, (for example you know who's calling and can answer with "Hello is that Mr Smith"), and it can also drastically reduce the average handling time of an incoming call, as your agent don't have to look the customer up in your customer DB etc.
So back to said system, it generally worked well - until there was a WAN flap or local circuit drop at a site. Then the CTI events would be missed or collected out of order causing all magnitudes of chaos, such as no screen pop for that site!
When the shit hit the fan, fixing said system was generally fairly simple, just restart one of the windows services on one of the several boxes. There was one exception to that, one service we never restarted! But more on that later...
So now you know a little about the system and it's general importance, let me set the scene for my t-shit moment.
I'm at a team meeting down in Birmingham, the whole team were there including the boss - and the boss's boss. Now, at our team meetings, one of us normally does a mini workshop on one of the systems we either spent lots of time supporting, or are SME on. Just a quick overview, a 'get to know it a bit better' for the rest of the team.
It was my turn that month to do 'said system' that I'm supposed to know so well!
So the boss is waffling on about process or something when we gets a fault in for my system. Boss suggests that rather than do the overview later, why don't I go through how I'd fix said fault now - live - with the team.
"What a great idea" I thought, it gets my bit out of the way and it means I can now have a pint at lunch to.
So I takes a look at the fault, it's simple enough, a circuit fault at a site has caused that site to drop the CTI connection and the site isn't getting screen pop. All I need to do is to restart the service on box 3, it's a ten second job!
After hooking the laptop up to the projector, I start explaining the system a bit to the team and RDP in to box 3 to show the team the service we need to restart. After waffling a bit I then log in to box 10 via RDP to point out some related and useful bits on there.
Now remember I said earlier than there was one service we never restarted? That was on box 10 and it logged the agents logon and log off events to the phones for every site. Now screen pop would only work if there was a phone logon record for the agent, and for reasons way too boring to go in to here, restarting the service on box 10 cleared all those logon records!
So if that service was ever restarted, to get screen pop working again, every agent in our company would have to log off their phones then back on. Doesn't seem like a big thing, but it would cause call queuing, customer experience issues and large headaches to the call centre managers who''d have to coordinate said exercise and would complain bitterly to upper management.
Oh one other thing about this system, while each box had a different function, the one service for said system which ran on each box was called the same thing!
But back to my t-shirt....
After waffling a bit more restarted the service to clear the issue, then I logged off RDP feeling smug that I'd done my bit for both the fault and the team meeting...
Which was when the new fault for no screen pop across the entire company came in!!
And I realised (as did the whole team) - that I'd restarted the service on the wrong box, oops!!!
How many agent had to log off and back in?
Oh about 7500.
I've done the "shut down remote server without restart as intended", and gone over to sort it out, yes.
I have also freed up disk space by deleting a stupidly large file that some unimaginative colleague had named "/unix". I think that server was fine until they tried to reboot it. Should I have known? For that matter, should I be admitting it now?
This is not exactly on the topic, but half of our fleet of remote site cabinetted servers were fitted with a cheap modem (the telephone kind) for support and file transfer, which occasionally went funny and stopped answering our calls, or something like that. (If I remember right, the ones the supplier installed the previous year were fine. Or maybe that was the other supplier.) We had between 50 and 100 of the wonky ones. On site staff weren't comfortable being asked to press the button or pull a plug as required to restore service. Supplier wouldn't replace them or couldn't be asked to for some reason. Probably because the equipment worked some of the time.
So I recommended buying household clockwork time switches to turn each modem off and then on every day. This wasn't popular but eventually management decided to buy a box of these.
But there wasn't room in the cabinet to plug them in. I was told; I wasn't hands on for this. Power strip in the wrong place, presumably.
It was still a good idea of mine though, wasn't it? It would have worked, if it worked.
Technically we could have bought a similar number of our own modems, but this game does have rules and that would have broken them.
Brings back memories of something that's happened around here a few times. I work for a company who makes semiconductor manufacturing kit, and said machines have two controller screens (one at the front for operators, and one at the back for maintenance technicians when the tools are under PM). Both controllers show the same info, and only one is active at a time, switchable between front and back with a nice button just beside each.
Over the (too many) years, I can recall at least 3 occasions where we've had emergency tool down customer calls and escalations that a machine has "just stopped responding". In each case of course we tell them to check that the controller they're trying to use is the active one, and they swear blind it is. Hence we send an engineer (sometimes involving a decent length drive, or even a flight) just to find that they of course haven't understood a damn word you said (or at least bothered to actually listen and check), and simply pressing the display switch button brings control to the screen in-front of them and suddenly everything miraculously works.
It's become a running joke over the years, given how much the hourly rate and travel expenses are for our engineers...
I worked for a well known mainframe company, I was a field engineer team leader with responsibility for about 6 customers in central London. When I was appointed to this role I spent about 6 months in constant contact with 'my' customers educating them on how to fix common trivial problems, how to pre-diagnose problems so when they called me I had a better idea what was wrong and could go armed with the necessary resources, parts... After all this my team call rate went from about 5 a day to 5 a week and my customers were as happy as Larry- I was well chuffed.
My manager wasn't happy - because we weren't running running around like blue-arsed flies all the time doing stupid things like cleaning read/write heads on tape drives, removing paper jams from printers, bursters etc.. he called us a load of idle bastards and we all got poor annual reviews!!!
Shortly after I told said manager he could stick his job where the sun doesn't shine and left.
So, don't curse those 'trivial' calls, they may mean job security and good pay rises.
"my customers were as happy as Larry- I was well chuffed.
My manager wasn't happy - because we weren't running running around like blue-arsed flies"
It's outputs that matter, not inputs. Politicians reminding us about how much they (they? - we!!) spend on whatever are amongst the worst for failing to grasp this simple fact.
A long time ago I had a phone call from an important customer in London. The screen on the messaging server we supplied was blank but the light was on. I jumped on the train, rushed into town (only a 35min train journey so not too bad) and got to the client's offices an hour later. Sure enough the screen was blank, now in those days we had manual controls for brightness, contrast, etc and I assumed that they had just turned them down. This was not the case. The screen just looked like it was off (apart from the green light to say it was on), strange. For some reason I forget now I ran my finger down the screen and revealed a layer of perfectly formed dust so thick that it blocked out the entire screen. Turns out that the office next to the server room had a fanfold printer and an extractor fan to remove the paper dust. This fan just dumped it all into the false ceiling and the aircon in the server room sucked it through and deposited it over the 6ft racks and of course the static on the screen did the rest. At least it was easy to fix.
So why did the technician not ask what the issue was on the phone before he drove all that way if it was so minor a problem as to not warrant a call out? Sounds like poor time / job management to me, not a problem with the pub land lady. In any case the machine was not in full working order no matter how minor it might seem to the technician and why is traffic on the m25 something to moan about, it is normal and should have been expected.
I'd installed a system for a customer of ours in Cyprus. A few years later, and I was working for a different company and the original customer calls me and says his system had broken down and would I come out and fix it as I was the only one he would trust to touch it.
Business-class flight was arranged by customer on a new Airbus A310, and his wife collected me from Larnaca to his home in Nicosia.
A 10 minute inspection of said system identified that it failed to power up, and a quick inspection, with a DVM, of the power supply found the fault. A trip to a local component supplier found the correct power transistor which was duly fitted.
My flight back wasn't scheduled for another day or so, so that left me some time to fo some sight seeing.
I knew Cyprus was divided, but I didn't think it extended to Nicosia....so, I was walking around, minding my own business and turned into a street, which proved to be a dead end and was patrolled by some UN troops, with assault rifles. I'd found one of the "green lines" and I could have almost shat myself with fright, having never been that close to armed people before. A quick "hi" and a hasty retreat ensured. Scared the daylights out of me for about 10 seconds.
I wasn't on this one, but a colleague had to go with a trouble-shooting group to a destination in Northern Ireland. One of the group was an American from head office.
They hired a car at Belfast. After a distance the American started asking about these "troubles" - where was it all happening? It all looked quite normal to him.
One of the group was a woman whose husband was ex-Army. She pointed and said "Well, there's a sniper there and someone there and someone there and that looks like a reinforced OP..." at which point they entered a chicane and came under observation. The woman promptly wound down the window and shouted "Keep up the good work lads". My colleague reported that from the smell the American immediately had some kind of nether region accident. I think if I'd been there it would have been two of us.
"It all looked quite normal to him."
That was the problem. It could all look quite normal but unless you knew the area you couldn't be sure. I remember driving along some country roads with a couple of SOCOs who started getting quite nervous. Seeing that these were roads I'd happily driven along quite often taking the wife & kids out at weekends I started to worry. In the end I decided it was just that they were from the other side of Belfast & didn't know where they were.
Call from customer saying that they can't get our software to work and that it always goes wrong at the same point. Very upset, insisting that someone attend NOW (four hour drive) or else...
We check the software in the office and, sure enough, it works perfectly. A friend gets in his car and drives to the customer's site.
They run him through what they've been doing to the point at which it is reported to fail. "See, we get to this point, but pressing the right mouse button does not do what it says on the screen". "Ah" says my friend "I see the problem. Do you have a marker pen"? He then puts an "L" and an "R" on the mouse buttons...
I was called about a malfunctioning printer. I investigated. Found no issues on the pc, walked around the desk to the printer, with a flashing light and a display that said load paper tray 1. When I told (as polite as I could) the user that it was only out of paper, they asked if I could fill it....
Reminds me of the time I was managing a services contract in Manchester.
I was called to urgently deal with overheating issues in some government agencies office and had travelled a couple of hours to get there.
When I arrived, sure enough, the office was roasting and the staff appeared almost dead. This appalled me, as earlier that week we had hired dozens of portable AC units and planted them around the many offices to assist. It was a good will gesture towards the client and totally beyond our contract.
I checked the units and each one was bone dry.
"They require topping up with tap water every day" I said to the manager.
"Yes, so why haven't you been doing it?" was his reply.
Glancing around the office I spotted an entire garden centre of plants and enquired who watered them.
"Well the staff of course"
"So, wouldn't it be in your staff's interests to also water the air conditioning units at the same time?"
"It's not their job"
"Well, all I can suggest is that the AC units are off hired due to improper use and your staff go home once the temperature goes beyond union agreed limits"
Lesson: Never ever try to nurture good will with government organisations as it will be undone almost as soon as you leave the office.
About the worst trivial thing I've ever dealt with was when I got dragged into work while trying to take a sick day with stomach flu. The boss (Note, not my current boss) called and said it was an emergency. Her computer wouldn't turn on and she had to have something done that day. So in between the frequent involuntary obeisances to the porcelain god I made my way in to work. And plugged in her monitor.
For example, a mains powered, battery backed up remote server (e.g. raspberry pi), with a 3G dongle, and dynamic DNS, running a remote shell, should allow you to remote in to their infrastructure and remote fix.
Even having a remote PC on their network with TeamViewer can help.
Couple this with remote ILO/DRAC providing remote hardware control console (as found on HP/Dell servers) then many issues never need a site visit. Everything that has a power button on it, should have a plan B for being able to activate it (on/off) without having to press it with your finger, and at the same time have the ability to see the boot prompt and interact with it.
I worked on a secure UK Government project for many years. All the server kit was either Dell or HP with DRAC/ILO installed, with the servers in a remote secure location with no onsite staff and the sort of security that is "tooled up".
Whenever I went to the site management office to do a software install, I would ask the local guys to check several days before my scheduled visit to verify ILO/DRAC access, because the stupid things would apparently "pop" in their interface slot, and fail to work for remote control. Getting someone on-site to visit the server to reseat the DRAC was a multi-day process, and I didn't want to waste my install time waiting for this to be addressed. In addition, the DRAC and ILO software did not play well with the terminal server jumphosts used for remote management, so the entire desktop session would regularly die leaving more hassle trying to reclaim the DRAC/ILO session from a still present but no longer working virtual desktop.
Of course, 3 times out of 4 the pointless jeremys didn't do the DRAC logon test, so I had to waste several days anyhow.
Regularly, a straightforward 2 day install turned into a multi week-long frustration fest ...
...possibly the same one as SH. I never got a satisfactory answer from my boss as to why they didn't use 30p LEDs instead of bulbs, and would consequently almost never need changing.
The reduced current consumption of LEDs would probably have stopped the lamp boards catching fire (yes, really) when too many bulbs were switched on, as well. When my colleague asked "how many bulbs is it safe to have on at once?" all he got was the answer, "we don't know, just don't turn too many on at once."
Whilst in the middle of setting things up and training staff at our new contract site I was called to a meeting with one of the clients departmental managers.
It was a government organisation, with thousands of employees at their national headquarters.
The manager was introducing hot desking for several dozen employees and needed my urgent assistance in setting things up.
Apparently her staff came in all shapes and sizes and the flat screen monitors needed to be adjusted for optimum viewing height.
I quickly demonstrated how the screen was fully adjustable and she nodded knowingly.
Then the bombshell, as she insisted it wasn't her staffs responsibility to adjust the screens and I would need to have my staff virtually resident in the department to accommodate each and every coming and going of her musical chair policy.
I did react rather badly, as I couldn't hold back the laughter.
Once the giggles had subsided I pointed out that was totally unworkable, beyond the contract and billable.
Only the word billable made any impact in this ludicrous stand off and had her reaching for marginally less moronic ways to drain my staff of the will to live.
Eventually after knocking back suggestions like, someone being here first thing in the morning to play mother or setting up a range of desks with tall, medium and small signs on them she came to the only logical conclusion possible and the only one I was willing to accommodate. She gave up on the idea of hot desking.
I later asked why, when all her staff have a desk was she so determined to have them going totally gypsy. Her reply perhaps answers many questions that are usually asked of such incompetent agencies. "Well, they either get along with their neighbours and spend the day chatting and not working, or they learn to hate them and spend the day bickering and not working."
Dealing with an annoying PA of a client of mine over the phone. Her laptop "wouldn't go online". Spent 30mins on the phone with this silly person explaining how to locate the WiFi switch on the side of the laptop. She denied the switch was there. Claimed the laptop was broken.
No matter how I described what she was looking for, she kept insisting she was an "IT Expert" and that there was no switch.
Next day, two hour long train journey, walked into office, turned on switch, walked out of office, two hour train journey home. Being paid £350 to read a paperback is nice. And I am glad to say the company soon got rid of her as that wasn't her only mistake...
Oh - and then there was that other time I had an hour long trip up to London. Getting to London at 09:30 the tubes were out so had to take a bus to Sloane Square. Got on site to find nothing actually ready so turned round and walked straight out the door again after being on site barely five minutes. And walked straight into chaos. It took me four hours to get home that day. The date? 07/07/2005. I still put in the full invoice.
(To protect the guilty parties I've decided to post this as AC. This is a tale about the support of semiconductor production equipment. Some of you might be able to guess who I am from that)
The research and development have been involved in a support call with a customer site for about 2 days. In which they promptly managed to break things badly enough to need replacement (R&D guys and hardware... I don't know how but they ALWAYS end up breaking things). Ofcourse, downtime in a semi-con fab is a matter of hundres of thousands if not millions of euros a day, so some people start getting nervous here. So late in the evening on a friday we get the call if one of our department can be on a flight to asia the next morning. 3 of us spent the entire evening hunting down people who know what is wrong, what type of machine it is, what type of parts we need, what spares we'll need, what spare-spares we'll take just in case, if we need special tooling for installation and qualification, how and/or what we need to qualify, what the deliverables are, etc. Because ofcourse everyone that could potentially have told us this knocked off 30 minutes before they tell US we need to do something...
Ofcourse we also had to fight tooth and nail to get those "just in case spares to spares" shipped because why would we need them? Then packaging everything in a way that'll survive the usual dropping off of trucks and throwing around that packages marked "Caution: Fragile" are usually subjected to.
Eventually my co-worker is on the plane the next afternoon. When he gets there he finds out just how stuffed they got the machine (well and truly stuffed) but miraculously we had managed to scrounge and scavange all the parts he'd need for the repair. The next day the parts arrive and first thing in the morning they are unpacked. Remember that part about these parts being very fragile? My co-worker starts neatly opening up one of the parcels while one of the fab-workers start on another package. A cleanroom product is always "double bagged". The outer bag keeps the contaminants from travel and package out, while the inner bag protects the part right up until it is about to be installed. The RIGHT way to open a cleanroom packaged product is to carefully cut off the outer layer of packaging while leaving the clean inside layer on until the part is about to be installed. This unnamed fab worker just RIPS all packaging off. Bending the legs on a rather fragile mechanism in the process. The mechanism he had just FUBARed was also exactly the stuff needed to get that machine up and running.
Some quick thinking and a lot of unsanctioned language and work methods later my colleague managed to frankenstein a working system out of the spare-spares we didn't need to bring, parts from the old mechanism and the broken new one.
The moral of the story: If the onsite support engineer tells you he'll need something, don't tell him otherwise. We tend to know what we are doing.
My main job in education was as a highly trained literacy specialist, travelling between schools. Quite well paid by teaching standards. But we were a small team, in a lot of demand with a full caseload and enormous waiting list. (I did my IT support role out of core teaching hours).
One week I arrived in a school where I'd been called in to assess a child who was an urgent case, causing all sorts of concerns and so forth.
When I got there there was a terrible pong coming from a sink in the open plan area in the middle of the school. I was intercepted by the head teacher who asked me if I would fix the sink.
I pointed out that I wasn't employed (by the LEA so it didn't come out of his budget) to do a plumber's job and he had a caretaker to manage that sort of thing.
"Yes", he said. "But the caretaker is busy".
Once had a fun one in Academic Computing. Most of the campus was PC, but of course the art dept will have their Macs... (this is a story so old it's pre-windows!)
So of course, art dept chair wants to know how to protect his work on the 40 MB (not a typo, this is an old story) Mac SE. We orderd him a copy of Redux (popular backup software of the day) shipping to his office, and told him "Give us a call when it's in and we'll send someone over to your building to get you set up."
A few days pass and we get a call - not, as one might expect, asking to have someone over to set up the new software. No - this professor called to complain that the software wasn't working!
A little investigation revealed the reason - somehow he'd gotten to the point in the manual where it described how to run a backup (place a blank floppy in the drive, when it fills up, take it out, label it, and place the next floppy) while entirely managing to miss out on the part where you install and run the software!!!
So he had been sitting there for an hour, swapping in blank floppies, waiting a few minutes (for what I'm still not sure), taking them out, labeling them, and going on to the next one... at some point he'd started suspecting that the software (still sealed into the envelope, still in the box) wasn't working and placed the call to support.
We gently explained that in order for the software to work, you have to actually install it, and then, yes, even run it.
I used to work for a company that made image-processing inspection equipment. We had a number of cameras pointing at a flow of product (usually foodstuffs), some DSP to analyse the video, and a set of solenoid-controlled valves to blow defects out of flow using compressed air. It was a neat system.
There were two of us who did the field-service work (alongside our main jobs). My colleague got a call from one of our installations in Germany - their machine had completely stopped working. They had no idea why, nothing had changed, it was obviously faulty. He took them through the usual diagnostics, and it was clear their cameras were out of alignment; this is something all customers were trained to correct, but the re-alignment process was failing. So he ended up with an overseas visit to re-align a camera.
On arrival at site, it was clear that the machine would not be realigned today. The substantial, stainless-steel frame that held the cameras had a suspiciously forklift-shaped ding in the side...
Vic.
I used to work for a company that made image-processing inspection equipment.
OK, another anecdote from the same company.
We'd put some kit into a frozen-foods company in Grimsby. It had been a long hard slog to get the sale, but we'd got there in the end. They'd even made a special TV advert about how this particular product had been "carefully selected" to give you the plumpest, nicest, etc. Specially selected by code I wrote, that was. Anyway, I digress...
We got a phone call form them one morning - the machine was broken. Nothing workd - it was just throwing air around like it was going out of fashion. I went through the over-the-phone diagnostics with the guy on site, and there was clearly something wrong with the unit's vision - it couldn't even see the alignment target, let alone get any use out of it.
So I agreed to drive to site to fix it. A quick mental calculation later, and I told the customer I'd be with him in four hours. "That's not bloody good enough" was the swift retort. I had to explain to him that, even if I disregarded speed limits and left within the next half-hour, four hours was what it was going to take, and if he wanted me before then, he'd need to charter a helicopter. He calmed down a little then.
So I got to site. I went through the usual high-care routines for site access - this was frozen fish, so there was a bit more than some other sites. And they took me to the machine.
The machine had been installed in a chiller - that runs at approx -4C. The product remains frozen throughout, and there is enough time for processing. The machine - an optical sorter, remember - was made from stainless, and so would soak down to that temperature. And then they used a hot hose to wash down the whole line between shifts.
I walked into the chiller and decided I couldn't set the alignment, on the grounds that I couldn't actually find the machine in all that fog...
Vic.
My best ever wild goose chase was back in the late 80's.
We had a big customer in Hong Kong who seemed to manage to hit every bug in our DBMS software harder and faster than everyone else. in '87 I'd been out there to track down a particularly nasty little fault that was causing a DB corruption part way through the running of a simulated workload which constituted part of the UAT on a multi-million quid project.I helped track it down and then ended up loafing around Hong Kong for three weeks wearing a pager until the final sign-off was obtained.
Anyhow, when this same site was live in mid '88 they hit another issue. The distress flares went up and my presence was demanded, so I jumped onto a paraffin budgie at Manchester on Saturday morning, landing at Chep Lap Kok on Sunday to be greeted by an old mate who'd been hauled up from Oz who greeted me thus: "About ten minutes after you took off from Heathrow we found that they'd been running an un-patched version of the software. Problem solved. Shall we go to the Pub?".
Mistake.
Gentle readers. If you fall off of a 13 hour long haul into a time zone 8 hours adrift from the one set in your body clock, the last thing you need to do is sink several lunchtime pints and then crash in your hotel room mid afternoon. My body clock obstinately stuck on UK time and the next 5 days until my return flight were a hell of jet lag. Sleepless nights and dozing off at the desk days.
I once worked for a company that had just hired a somewhat cute "star developer". These were the days where mobile applications on Palmtops were getting all the rage, and she was hailed as the best programmer that ever existed. Two days after the started working, she ordered some "necessary equipment", which the boss promptly paid for and got delivered: a couple of high-end Macintoshes with all the latest expensive peripherals, a really, really expensive digital camera, scanners, and a monstruous laser color printer that stood almost 2 meters tall, with all the bells and whistles. All for "GUI visual concept building", e.g. spend the day playing with photoshop on some lame sketches she could have done on a piece of paper. A toner kit for that printer would easily have paid for my salary at the time. We programmers had slow and crappy PCs with CRTs, of course. Talk about charm.
Anyway, she starts to pester me about the printer one day. "It's not printing at all!", to which I reply, "I'm busy" for four consecutive days. Then, my ever-so-helpful pointy-haired boss calls me and tells me to drop everything and help her. Okay. Let's see this damn thing.
As I approached the printer, the smell of molten plastic was suffocating to say the least. There was a small puff of smoke coming from the sides. I started pulling things apart. When I got to the toner, it was stuck. I pulled it hard, and it kind of exploded in mid-air. Toner everywhere. It turns out that she left all the blue plastic tapes everywhere. The ones with big red yellow tabs that read "REMOVE BEFORE USING". It had melted against the drum and shut the toner output slot. The drum was a mess of sticky plastic. I went back to my boss with the melted toner and said he needed to hire someone with a bit more clue than her, and to get a new printer. For the next three months I was looked down as the arrogant leper bastard. She stayed on the job for another six months, produced nothing, got a raise, then left the company for another stupid boss. She really knew how to use her assets.