Nothing's working, and I've checked everything, so it must be YOUR fault The customer is always right. Except when they're not. Here we have a story from the On Call archives concerning connectors, telephones, and a user blessed with a little too much confidence. Today's tale takes us back to the 1990s and comes from a former employee of a now-defunct telecom equipment vendor. We'll call him Felix …
Remove loose card -- leaving the slot orientation plug behind in the slot.
Reset loose card -- external device STILL DOES NOT WORK.
Remove card, examine slot orientation plug on the card. Confirm that slot orientation plug, now on the card, lines up with slot orientation gap in the slot.
Repeat repeatedly -- external device STILL DOES NOT WORK.
Eventually, get educated by someone who's had to have hardware replacement to fix exact same problem. Remove slot orientation plug from card, and re-apply it at the other end of the edge, and reseat the card 180 degrees around, the way it was before you started reseating the loose card ....
That would be the time an engineer moved a 1000A, 80V PSU (about 1m x 1m x 2.5m) without unplugging it, which ripped the three-phase socket off the wall, causing the phases to short together. Luckily, the protection for the circuit kept the bang reasonably local.
Another time, the same guy was having trouble getting some thermocouples to stay in place on a power board (open) he was working on (it was connected to one of above PSUs, but it was only set to 48v). He decided that a large lump of solid brass would do the job, so put it on top. The PCB was a charred, smoking mess by the time the block had stopped jumping in the air and throwing molten metal in all directions...
Sounds like one of "those guys". The kind whom disaster stalks daily.
I once knew a farm-hand / grounds-keeper at a large stables who, among other things, managed to tip a tractor twice within as many months. No-one else had managed it in the memory of those currently working there, and he did it two completely different ways. He was a nice guy, eager and hard-working... but his decision-making left a little to be desired.
I remember a very funny TV series where the detective started claiming "Trust me, I know exactly what I'm doing" every time before disaster started. Found it: "Sledge Hammer" (that's the name of the program and the officer). Would probably be not politically correct enough today; would offend someone.
So says Inspector Sledge Hammer!.
Coincidentally, I am currently re-watching it on DVD. It is absolutely hilarious.
The writer was so convinced that it wouldn't be renewed for a second season that he killed everyone off in the last episode with Sledge trying - and failing - to defuse a nuclear bomb. It was renewed, so he had to set the second season five years before the first.
There was one colleague I knew who had the infamous phrase "It'll be all right..."
Long ago I found myself supervising an archaeology student (project on numerical classification of objects) whose ritual greeting seemed to be "Wanna hear a tale of woe?".
His unsurpassed offering was the water-skiing weekend when firstly he got up from steering the boat to go back and sort out some problem with the tow and secondly one of the group managed to write off the boat on its trailer when they were pulling out to go home - right into the path off an off-duty police officer who was the father of one of the girls in the party & had come looking for her.
Ah, I worked with one of those. His job was providing tech support to our customers but between calls kept regaling us with stories about how his motorbikes needed constantly repairing and rebuilding, and how he'd had to strip his home PC, replace parts in it, reinstall the operating system..
It was always a mystery to us that he could never 'fix' something and then just leave it working for months or years, like the rest of us.
>> regaling us with stories about how his motorbikes needed constantly repairing and rebuilding..
Ah, must have been a Harley-Davidson owner.
I swear that there are more Harley's in pieces in garages than drivable in the world.
Obviously bled over to everything in his house. Probably rooms half drywalled, plumbing sticking out at odd angles, etc. etc.
Not IT-related (other than us all working together in IT at the time) but myself and a bunch of colleagues went on an offroad driving day at Mallory Park years ago. On taking our places in some offroad gokarts, we were told to "be careful but have fun and go for it - these are perfectly safe and nobody has ever rolled one of these that we're aware of". One of our lot promptly did so within 5 years of the start...
At Mine's a Large One, my friends & I were at a local amusement park many decades ago. It featured a "tank" field where the "tanks" were just modified gocarts with body paneling to make them look right, plus a glorified potato gun on the roof that shot tennis balls filled with talcum powder. The "round" would strike a tank, barf the powder out in a giant cloud, & the "dead" tank would be easy to spot from the giant splotch.
The folks that ran the park told us not to worry about the stability, "you can't make them flip no matter what." Silly buffoons.
All it took was causing the gun to swivel in one direction at the same time the tank turned the other. The torque imparted was enough to cause the tank to tip, the barrel to hit the deck, the opposite wheels to lift off the deck, the barrel to bend, the tank to crash back down on all four tires, and the turret to try & reset to center-forward as the internals tried to recalibrate it's sudden inability to aim. Result: the tank would flip over the other direction, wind up on its side, & skid to a screaming stop.
The operators come running out to make sure we're all ok. We were, we were too busy LOFAO to feel any pain. The operators made us get seen by the on site medics while they tried to figure out how we'd manage to break their tank.
In the end we were all physicly fine, the operators refunded our money for the game, then banned us from ever playing again. All because a bunch of teens managed to do what others claimed was impossible.
Best. Birthday. EVER! XD
Many eons ago, for my 40th birthday, my wife bought me a day at a 'tank driving experience'. Great fun. I even got congratulated by the ex Armoured Corps sergeant who said he'd never seen anyone get an SPG (Sexton?? Abbot?? memory fails me - big bastard anyway) sideways before and would I please not do it again.
We had a Stalwart for some tests once - a big 6 wheel drive, 6 wheel steering amphibious armoured truck thing. See https://www.youtube.com/watch?v=3mr_pCrhTkk.
We were using an old airfield for the testing which had big white crosses painted each end of the runway to stop people landing on it. The squaddies we borrowed to drive it were loving the easy duty and were tearing about happily in this 'unstoppable' beast.
Turns out if you drift it 180 using the slippery painted cross then when it hits concrete again all three wheels grip at once and flip it neatly on it's side...
You can turn anything over if you try hard enough :-)
I used to be into trialling Land Rovers - not serious "customised for the job" stuff, but "drive it there, do a trial, drive it home and hose it off, drive it to work the next day" level. One year I went to "The Nationals" which were at the army training ground at Bordon - and during the brief were told not to go into the deep holes or to touch anything we found lying around in case it went bang.
The cut a longish story short, as well as competing, I ended up giving a couple of other people lifts between the sections - it's a big place - as I had back seats and the Land Rover they were double entered in didn't, and there's a rule about not carrying anyone unless they are in a seat. This guy used to instruct there, and was quite happy talking about the techniques they use to recover bogged down 60 ton tanks. Basically, exactly the same way we did it with Land Rovers - hook up another (or 2 or 3) with an elastic rope, set off with great gusty to build kinetic energy, and with a bit of luck that gets converted gracefully into tension in the rope which unsticks the stuck vehicle.
There's been accidents with Land Rovers (circa 2 tons). The rule with the tanks (circa 60 tons up to it's belly, and 3 off (180 tons) doing the recovery was "no-one anywhere near unless inside a tank with the hatch shut !
Unfortunately getting video evidence of any of my exploits is not only on My Skippy's List of shit I'm not allowed to do, it's highlighted, bold, italicized, underlined, in 99 point type, and somehow, even though the list is printed on paper, the damned entry blinks.
*Comical pout*
All the fun shit is forbidden!
=-)p
Besides, if my last ten clips (posted before MSL was updated to include said aforementioned enteries) then I tend to wind up on Earth's Most Wanted. Sheesh, like the Powers That Be have never seen someone doing GTA maneuvers in an Abrams before? ;-D
"Blue flash" reminds me of a student holiday job working maintenance in a department store. Me and another temp were tasked with stripping the wall boards out of a now disused storage area. We got the job because it was directly under the re-upholstering shop and full of every kind of dust you could imagine. We were equipped with pick and axes to rip the wall boards off - and not something that would be allowed nowadays (asbestos or not). We had dust masks and overalls to keep the worst out.
Anyways, at one point I noticed a thick cable leading in behind one wall and kept clear of that area until we could check what it was. Not my buddy - he swung his axe into the wall. Luckily, his axe had a long wooden shaft that was longer than the blue spark that shot out from the wall! Lucky, too, that the spark didn't ignite the dust. The blue flash was the last of our light as the warehouse (and, as we soon discovered, the rest of the building) plunged into darkness and electrical inertness. Fortunately, that building was not part of the main store complex and the power-down didn't affect any customer-facing operations.
Happy times!!!
We had a chappy doing some landscaping with a very large digger in a neighbouring field when he cut through an off record mains cable with a blue flash that was visible for miles and a bang to wake the dead. We we playing footie in the yard overlooking this and saw the driver climb out of his cab and leap from the top of the caterpillar tracks a good 20' horizontally to the ground around 5' below the track. His boss commended him on ensuring he didnt allow himself to become part of the circuit by climbing down off the top of the track. Turned out the reason he jumped was because he couldn't see to climb down due to the several hundred joule flash and was in serious flight not fight mode due to the flash bang and wanted to get the fuck out of there asap.
Not quite the same level of importance obviously, but my father-in-law swore blind several times that he had removed the orange sticky label thing from the head of the new Canon printer ink cartridge, yet it was still not printing.
My round trip to remove said orange sticky label thing was only about 50 miles and it was during the day, but at least he had the good grace to be completely and utterly embarrassed. And, yes, we still bring it up every now and again!
My father asked for help as the stupid new cartridge he'd put in his printer wouldn't come out at all. I eventually managed to extract the black cartridge from the colour slot. In his defence the cartridges on that printer are the same size, though slightly more damning is that he had needed to force it past the plastic part that is there to stop you putting one in the wrong slot.
In his defence the cartridges on that printer are the same size, though slightly more damning is that he had needed to force it past the plastic part that is there to stop you putting one in the wrong slot.
Ah... one of those who believes that if all else fails, get a bigger hammer.
I like the stickers you get on button batteries these days, which say there's a bitter tasting sticky ring on the battery to stop kids swallowing them which "MUST NOT BE REMOVED" - as it says on the protective sticker. Only problem is that unless you read carefully, it looks like the protective sticker is the bit that must not be removed. I've been paid to fix that a few times by now.
We had some idiot in manglement (Not in IT) come up with the bright idea of putting all Inkjet cartridges in one central location including "pulls" from printers about to be decommissioned (1/3rds through the site downsizing/closure over a 5 year period).
Within weeks we started getting tickets about inkjets printer issues:
My HP printer isn't working & I have changed the cartridge.
Type 1 Resolution:
Before I come half way across site have you got the old one there?
Yes (No)
Whats the printed label number on it? (OK let me google the right number, its 57)
57
And the number on the "new "one 63
Your'e fitting an incompatible cartridge, get one a 57.
But it looks the same!
Type 2 Resolution:
You fitted the new cartridge, are the numbers the same
Yes
& it's new right
Yes
One trudge down the hill to the building in pouring rain later, pull the cartridge out, its as light as a feather.
I thought you said this was new, it's empty.
Can't be it came from the new box
Was it a new sealed box or just in the tray?
In the tray.
(Pull a cartridge from elsewhere & hand it to them) Feel the weight difference?
Yes - Why was it in the new box then.
Someone had the bright idea to centralise inkjet cartridges to one location & pulled cartridges out of machines going to be scrapped, along with new ones & sometimes people dump the empties in there as well, when they get the replacement.
Type 3: A mix of the two above.
Oh joy. I used to work somewhere that swapped to "remanufactured" inkjet cartridges to "save money". Trust me, it didn't. The fact that we had massive networked colour laser printers should've killed the local inkjets off, but, they were used by the executives, to print whatever they wanted, without it "counting" or being recorded on the network. Hilarious! Thousands of hours, across a Blue Chip company.
"My round trip to remove said orange sticky label thing was only about 50 miles and it was during the day,"
I had a the same experience with a commercial customer. In my case it was a 250 mile round trip. But it was a nice sunny day and the customer location was a nice seaside resort :-)
5 minute fix, a couple of hours on the beach "for lunch", then a leisurely drive back home over the moors :-)
By a strange coincidence, 1988 happens to be the first year that I fucked something up. I had been working at a data center in West Germany for the US Army, and we ran massive cycles on the mainframe for supplies and requisitions, as well as financial.
These cycles had dozens of jobs that had to be started when a previous job had passed a certain job step otherwise things would quickly go TITSUP. By automating the job release processes, we were able to automate a lot of this, which reduced mistakes and sped up the processing of the cycles. I also made lots of changes to automate generation dataset processing and free up lots of space. There was nothing special about this - it was basic MVS JCL.
n 1988, we had just finished setting up the last cycle for automation - the monthly financial cycle which took more than a day to run.
All of these cycles had one thing in common - a job at the end of the cycle to print the spool files and produce the output for the customer. Since this was pretty simple and never really had any issues, it was an afterthought. No one, including me, had really looked at.
So when this cycle that I had modified was scheduled to run, I was out of town. And of course that was the first time we had ever had in issue with the print job. I don't remember the details, but it failed to print, issuing a return code, but not abending. And the file disposition for that JCL step was set to delete the spool files.
So the job didn't print, the spool files were gone - and weren't on any backups because they had just been created. And the person who just made changes to the cycle - me - was out of town when it ran. And had I been available, I could have easily avoided the problem.
The client wanted his printouts - and my head on a platter. We had to restore from backups and re run most of the cycle. I owed lots of people beer after we got those printouts. And to this day, I am always around when any production system I made changes to runs for the first time to ensure it runs smoothly.
reminds me of a story from a friend of mine. His dad had some problem with the computer that couldn't be resolved over the phone so friend asked the dad to bring the computer over next time he visited.
Dad turned up carrying just the monitor which he assumed, since that was by far the biggest component, that was where all the gubbins was.
450 mile round trip to Penzance for me about 15 years ago, to press the power button on a data backup server, when someone from our headoffice remote shut it down instead of signing off.
They obviously didn't admit it, even though I could see the shutdown request in the logs.
@elaar
A former manager of mine regaled us with a tale where he was flown out by helicopter, a long time ago, to an oil drilling ship somewhere in the North Sea to fix the 'faulty' software that, no longer, kept the ship in place with the help of a couple of gyroscopes. My manager's diagnostic was the software worked as designed but both gyroscopes had been turned off. The officer on duty, when the ship started spinning around the drill, swore on everything he held holy that he had only turned off one of the two gyroscopes not both. The computer log said otherwise so a now fired officer was flown back to land with my manager.
About 15 years ago I was on-call and client calls from a middle-of-nowhere "highway oasis": one restaurant computer isn't connecting. Client declines to troubleshoot beyond rebooting and asks for onsite service (per agreement).
After driving the company van for 6 hours and 500 km I arrive only to immediately discover that someone had unplugged one end of a patch cable in the comms room cabinet. Re-connected, checked the operation and explained the problem to the manager. 15 minutes and off I go for another 6 hour trip back to HQ.
A tiring day, but at least writing the bill gave some satisfaction.
---
On another occasion I was to install the computer equipment at a restaurant even further away. Waking up before 5:00 for early morning flight, and arriving five hours later I can see (and I'm told) that the builders are about a month behind schedule with missing walls/electricity/network/everything. Can't do anything so back to the hotel I had booked, spend the day and take the first available flight back.
Apparently the construction company had communicated the change of schedule several months earlier and the news had reached my inept manager who kept it to himself and tried to blame me afterwards. Feckin eejit!
"Client declines to troubleshoot beyond rebooting"
When I was on phone support for Dixons (while Dixons phone support was still controlled by Dixons and not Capita), that kind of customer was the bane of my life.
The thing was, we wanted to help, and if the machine was genuinely broken then we would readily book a visit by a repair technician (it's why we were there in the first place). However, there was internal aggression between the repair department and tech support, since it was not that uncommon for visits to be booked for ridiculously simple software/driver problems which could readily have been fixed over the phone, or because the wrong fault had been diagnosed because of poor support by someone in our group and the tech would turn up with the wrong part. Partly understandable, I guess.
The repair technicians were only supposed to fix actual hardware issues - replacement boards, hard drives, etc. - and it was our role on support to diagnose such issues by elimination through various steps. And even then, we began offering a DIY repair service, where parts were sent out and the customer could fit them with step-by-step instructions (with a visit if it went titsup). But some customers don't like having to help in the diagnosis.
I remember one really annoying call from a young woman, and the problem she had needed to be properly identified before we could send out a technician with the correct parts. I explained to her we wanted to fix it, but we had to find out what the actual problem was. She was amenable, but in the background there was a gravelly voiced harridan who kept interrupting and repeating 'tell them we want someone to come out'. Every time, the girl changed her attitude when this happened.
I must admit that in the end I snapped and said 'will you please tell that person to shut up'. She said 'that's my mum'. I apologised profusely (but I didn't mean it).
While providing telephone support for my brother, whose printer was not working, the conversation went like this...
Me: OK - first let's check the basics. Is the power cable plugged in at both ends?
Him: Of course it's bloody well plugged in at both .... oh. It's working now.
I once called my ISP to complain the router was completely fubar, and having done a reset the ISP default username and password didn't work to log in.
After extensive troubleshooting I remembered that a couple of years previously I'd updated the firmware. Having done support for a while, I started laughing when I realised, and then apologised for being that customer.
I've had a case (in my own home no less) where a 486 - that will tell you how long ago it was, kept giving me minor shocks when i moved the keyboard. Long story short, the earth/ground pin on the IEC inlet was loose, and had cracked its solder joint inside the filter assembly they used to fit, so the chassis was floating at about half mains potential (120v here in 240v land). The metal baseplate of the keyboard was connected to chassis ground. There was enough leakage through the remaining components of the filter to deliver a tingle, and that odd 'corrugated' feeling if you rubbed your finger over the metalwork.
Took me a while to find because replugging the lead would fix it, for a bit, till the solder joint moved out of contact again.
Lost track of how many time that, per contract, I had to bill "portal-to-portal travel + 2hr minimum" for a site visit that showed on the "Resolution" as "Troubleshoot/Repair open circuit".
This was our code for "Drove 100 miles one-way to plug phone cord back into wall jack". We couldn't do it for free, but really hated to see some poor receptionist get canned over not noticing that the plug wasn't fully seated in the jack.
-----
On a more technical side, the old AT&T/Lucent/Avaya SCC cabinets (and the newer G650 carriers which replaced them) in a stack configuration have a ribbon cable which connects the carriers together.
More specifically, the architecture is a common TDM bus. Each carrier has two TDM bus connectors. A single-cabinet has a terminator plug on both bus connectors. Adding a cabinet requires removing a terminator, installing the TDM Bus cable between the cabinets, and moving the terminator to the unused connector on the new cabinet.
Both the terminator plugs and the ribbon cable have an arrow embossed in them. Unfortunately, it is physically possible to install a terminator or cable in the wrong orientation, resulting in dead cabinets, really weird system behavior, etc....
-----
Site visit for a totally down PBX. As we are heading for the comm room, my escort mentions something to the effect of "Bad enough that the network servers are down due to a failed UPS, but just too much to lose the PBX at the same time".
Me:
"So you lost the (huge single "room") UPS and all your servers went dark, and the PBX in the next room went down at the same time?"
Escort:
"Yep. Servers tend to do that when they dont have power...Oh F**k. Bet I know where the PBX rack is powered from..."
Me:
"I think I have a pretty good idea as well."
About a month later I was back there with a truckload of rack-mount UPS units, PDUs, and EBMs to install. Seems that the "Whole room UPS" crowd lost out to the "Multiple UPS systems with bypass switches" crowd.
.....some poor receptionist get canned over not noticing that the plug wasn't fully seated in the jack....
No No No. This is so wrong.There is no way a receptionist should be responsible for troubleshooting a system. Not even for the simplest of issues. Their job, the clue's in the name, is to deal with people. How were these issues phoned in? Who's job was it to do so? If the receptionist is told "Any problems call this number" then that's what they should be doing. If not, then who does make the call?
Either there's a tech support contract and it's not her/his responsibility what the fault is, or there's a first line support locally who should check the obvious. Maybe the receptionist ought to have checked, just plain common sense, you might think.. But it's not their responsibility to do so.
Having spent some 17 years on a variety of help desks I did develop some questions specifically for this sort of issue, for printers it usually went like this:
"Ok, so you have turned the printer off and on and it still isn't printing?"
"That's right, can you send someone please?"
"Can do, can I just get you to tell me which lights are showing on the printer?"
"Which lights?"
"Yes, as long as it is turned on there will be a power light and a couple of other indicators - that will tell the tech what needs to be fixed, so they can bring any parts with them"
Pause, followed by the click of a switch being powered on, then the "Whiirrrr..." of a printer powering up.
"Oh, I think I might have fixed the issue after all"
"Really, that is excellent, I'll close the ticket. Have a nice day!"
I've probably recounted these before but they're from back when I worked for a company doing support for the consultants at a major insurance company.
There was the user who swore blind she'd checked the power lead, including unplugging it and reconnecting it so we called an engineer, her boss swore when he received the call-out invoice for "plugging in power cable!"
Then there was the one that insisted the power to her computer hadn't gone off (we always knew when it had, due to the corruption of the database). An hour later she phoned back and apologised, her consultant boss had unplugged the computer for some reason then plugged it back in, restarted it and not told her.
After nearly 37 years in the industry I've learnt things rarely go wrong for no reason, it's nearly always something that's been changed, and I'll confess on some occasions it's been me that's made the change.
"things rarely go wrong for no reason, it's nearly always something that's been changed"
My ex would so often complain that the computer had "just stopped working" or "just started doing this" and swear blind that she hadn't done anything. Invariably I found that some new piece of software had been installed... which she'd deny installing (despite the fact that I'd set it up so that to install anything needed the admin password to be entered, so couldn't have "just installed itself" as was so often claimed).
I was on a help desk working with a customer who was having problems getting some software configured. The problem had been going on for about a week. In the end I went back to the beginning and the conversation went like
Me: Do step 1 and send me the output
Him: Here you are
Me: Do step 2 and send me the output
Him: Here you are
Me: Do step 3 and send me the output
Him:I don't think we need to do that, as it has no effect.
Me: Please do step 3 and send me the output
Him:Ok then but you are wasting your time, just wait and see
...
Me: Now start it
Him:It works!
A month later I met the guys boss at a conference and we had a conversation. The manager said the guy was often like that - not following instructions, and then it not working, and blaming it on the products. I wrote an email for the manager describing the situation - and an estimate of the costs of my time etc. I believe the managed had a career changing conversation with the employee, because we never heard from the employee again.
Circa 2000, everyone's getting online, lots are buying domain names, lots of them can't cope with how to set up an Outlook / O. Express mailbox, even with the most basic instructions provided. A helldesk colleague regularly resorted to: "Don't do anything unless I tell you to. OK, click this. Now write that in the field. Now... WHAT? NO I DIDN'T TELL YOU TO DO THAT. DON'T DO ANYTHING UNLESS I TELL YOU". Thankfully he did it in such a way that he never got complaints.
Wow! That sounds like me!
When I was on support, and people called in and we had to go through a routine, I absolutely emphasised that they mustn't do anything other than what I asked them to, otherwise it wouldn't work and we'd have to start all over again.
It began after I noticed that when I asked some to press a certain key or click a certain icon, I could hear 'clickety-clickety-click' in the background and I knew they weren't doing what I'd asked.
To be fair it sounds like an easy , bloody obvious thing ,but is also something you'd never guess over the phone because its ... too obvious / simple / easy
Maybe if its a "user" you'd check there wasnt a powercut in progress , but this was some kind of "tech lead ".
Maybe I'll get replies along the lines of "never underestimate idiocy " etc
But you simply cannot treat all comers like morons all the time .
There is a science to asking idiot proof questions without embarrassing the customer / higher up
Sometimes you just have to "assume" some things as a given
This will doubtless lead to a very specific reply:
“assumption is the mother of all fuckups"
well let me say in advance "fuck off 50/50 hindsight wielding bastards"
"There is a science to asking idiot proof questions without embarrassing the customer / higher up / Sometimes you just have to "assume" some things as a given"
It's hardly a science. Just needs a moment's thought. Anyone who has ever done front line support should know how it goes. IME it's easier with higher-ups, because they have experience of 'oops, we forgot to check the basics'.
It is fine to say things like 'OK, that sounds reasonable, now before I get in the car and drive to you, we're just going to run through all the simple stuff in case something was missed'. You can even add 'please suggest anything _I'm_ missing', to get buy-in.
Sometimes it's not the user/customer who's wrong.
Phone and internet became intermittent one afternoon. During an interval when the phone was working ask BT helldesk is there's work going on in the area.
There isn't. We can send out an engineer, £80 charge if it's not our fault.
Offer declined, walk down to the village where the footway cabinet is located to find:
2 x OpenReach vans
2 x manholes opened
2 x OpenReach bods occupying manholes, redoing everyone's connections.
Virgin are my faves for this. They seem to never tell their phone staff or update the service status when there's a problem at their end. So you can find the internet not coming through. Reboot the box, and it still isn't working ( but had been an hour or two before and nothing has happened otherwise). Check service status- nothing.
Phone 'em up and they say they need to run a check. Then it's "We'll send an engineer a week on Monday" time.
Usually by then Downdetector has caught up.
Once, just once, a phone jockey said "I've been getting a lot of calls- I'll check". He asked his manager - who'd heard nothing but went to make some calls. And in due course I was told there was work going on in the area to fix a problem they'd known about for 4 hours already.
I think VM managers are afraid to admit publicly if anything goes wrong.
That's BT / Openreach's biggest talent: nobody knows what anyone else is doing, has done in the past, or might do in the future.
I've had their people out to fix the same fault 6 times in a row. It was an iffy connection up a pole which only caused problems when it was windy. The first 3 visits happened days after I reported the fault, by which time it wasn't windy any more.
The 4th and 5th guys arrived, announced they'd need a cherry picker (which I'd said when I reported the fault) and left again.
The last pair of blokes actually ordered the cherrypicker and waited till it turned up.
How does BT actually make any money?
I think this is endemic in call centre staff. I don't think I've ever had someone from a major company come to fix something, who'd been told what the call centre knew and brought the right bits. I'm not meaning technical stuff here. What part of "The freezer handle is broken and needs a new one refitted" is too complicated to pass on so that he arrives with the dratted handle?
I suspect these sorts of problems are down to everyone, call centre & field staff, being incentivised to close a ticket ASAP. Not everyone is.
Back in 2020 we wanted to get some trees felled - old Christmas trees that had been planted out and grown into monsters. The wires for the street lights up the road ran close to the canopy so we would have to get them powered off. A guy from the distribution company turned up and took a look. He did a thorough job, walking up the road and discovered that due to an unfixed fault elsewhere the circuit had been patched to supply some houses. He also noted that my neighbour's trees were also growing round the wires and they - the distribution company - would have to trim them. Due to this and that and Covid there'd have to be a delay before we could get the work done. He went to his van to write out a report.
He must have spent a couple of hours on this all told. I was quite impressed with his thoroughness. Less impressed when, having finished, he drove round the corner, parked up and was still there a few hours later when I drove past. I had to chase up an appointment to get the power off and the work on the neighbour's trees has still not been done over a year later.
I think this is endemic in call centre staff. I don't think I've ever had someone from a major company come to fix something, who'd been told what the call centre knew and brought the right bits.
I once worked for a very large medical device company who sent a text (not MMS text) with the engineers job for the next day. The problem with that was they sent it the night before usually after 7pm so you couldn't organise spares for the job and the text message being limited to 160 characters usually said something like
"10th January 2022 - ref. 56789-455667433455 St Elsewhere Hospital. Main Street, some town, some county. Hosp ref: GFY-5678-6546-6579-6544. Problem:The customer says that the I"
Of course you got to site with zero information about the job. Management just weren't interested in changing their stupid system.
The problem with BTOR is that the guys who come out are contractors paid to come out - but only paid if they close the fault and either NOT given past information or don't bother reading it (despite assurances given to the customer (the ISP) that the techs have been briefed before coming out)
What they do is close the fault regardless as soon as they step offsite, meaning the whole cycle has to be started over. The entire system is unfit for purpose and essentially being milked hard by unscrupulous contractors due to the very low per-rate payments
OFCOM are utterly useless. because regulatory capture
I'll bet the last pair of blockes were actually BTOR employees, unlike all the previous ones
I have had a similar experience. No phone/internet service, so I call the telco to get it sorted out. No issue at their end, apparently. We'll look into it.
Whilst I'm on the phone, I look out the window to see a telegraph pole being lifted out of the ground. Ah, that would explain it. Thanks for the heads up, Openreach!
You think that's bad? I worked for Openreach (for sins committed in a previous life).
Gets a message to attend a number of subscribers (customers) whose 'phones were all NDT (No Dial Tone AKA. fucked).
Gets to site to see my erstwhile Openreach colleagues pulling the stick (pole) out of the ground with all the attendant dropwires (overhead cables) coiled and laid neatly in the subscribers front gardens awaiting reconnection to a shiny new stick.
Lesson, if your 'phones off look out of the window first.
Toodle Pip.
Quality British Customer Service - the kind that made British Leyland a worldbeating sucess
This is why commonwealth customers beat paths to everybody ELSE'S front doors as soon as they had an opportunity
But of course, it's all everyone else's fault and our company needs protection from those EEEEEVIL foreigners and their national-security-threatening ability to get things done right first time and/or not gaslight the customer
If my service is out, I'm calling. No, I DO NOT CARE that I can see a crew fucking up the pole in my front yard.
This is from long experience. If I look out and see that work is being done and assume that my service will return when it's finished, I can guarantee that I'm wrong, and my service will still be fucked. If I call and am told "we're working in your area, things will come back when we're done" there's a decent chance they will. If I call and they claim no knowledge of work and send someone, that is a. on them, and b. a sure sign that if I hadn't called, I'd be calling later and would have an extra day of outage.
Blame your incompetent dispatch. Do not blame the customer.
That reminded me of something a couple of months ago.
I'm not in support of any kind anymore, but I am still often asked to help with certain things by people who know me. In this case, the old lady who lives next door whose shopping I do because of COVID and her age.
She came round and told me her TV had stopped working and she couldn't get any of her usual channels. She has a set up with an old DVD/PVR system.
I went round and tested everything, pulling out cables and switching them, running through channel scans (yawn), and concluded that either the TV UHF modulator had broken (i.e. the TV was borked) or there was a problem with the aerial.
I arranged for an aerial guy to come round and check that aspect. A little later the aerial guy they'd assigned phoned and asked me which direction the aerial was pointing. I said 'west'. He said 'the transmitter is down and they don't know when it will be back up'.
I'd never have thought of that, since it hasn't happened before in my own living memory! And I'll be damned if I could fine any definitive online mention of it.
Later in the day I did another channel scan and got everything back for her.
Can't say I have ever heard of an OTA network going offline like that. But back in the late 1990's I watched mostly the Fox network (not Fox news). The Simpsons, Married.. With Children, The X-files and Futurama were all appointment TV. They also had the best New Years coverage of the local networks. And New Years Eve in 1999 as we were getting ready to watch the ball drop:
10..9..8..7..6..5..4..3..2..1 (screen goes blank)
No Y2K failures - it seems that our cable network (COX - insert your own jokes here) had a contract dispute with the Fox network and abruptly cut them off at 12 AM 1/1/2000.
I canceled my Cox cable a few days later and got DirecTV, which had great service until they got bought out by AT&T and became shitty.
Precisely. I've never experienced it before which is why I hadn't even considered it.
Mind you, we haven't used a terrestrial signal for probably 20 years or more now. The old lady next door only has that (though admittedly digital/Freeview, so she has a selection of channels).
Spare a thought for those covered by the Bilsdale transmitter someone mentioned above, though. It got burned down back in August and was to remain off air 'indefinitely'. A million households were affected.
I believe they have new transmitters operating since December which are bringing more people back online. According to Freeview, 97% of those affected are now covered by temporary or new masts, but they need to retune.
User stating in a hotel in the USA in the early 2000s (we were uk) rings up adamant that the modem in the laptop isn’t working.
Talk him through software checks (convinced he is right) but no dial tone.
After a few minutes someone asks how is he speaking to us - at which point he says on the hotel room phone line - then the penny drops as he realises there is only 1 phone in the room………
Well, in my case it was the helldesk bod doing that.
I was having some connectivity issues and he was troubleshooting it remotely while doing a Teams video call.
He kept losing connection because he'd disable something or another like he was troubleshooting it locally, and always seemed surprised when he reconnected that the call had fallen...
I've found that the likelihood of the problem being with something the user has done/not done/not checked even though they say they have is directly proportional to the volume with which they are yelling at you that "OF COURSE I'VE DONE THAT/NOT DONE THAT/CHECKED THAT, I'M NOT SOME SORT OF IDIOT" (excuse the shouting).
About 35 years ago I worked in a laboratory that had a lot of expensive, sensitive kit. Each item drew kWs of power, but they normally weren’t turned off as we could almost guarantee that at least one wouldn’t come back up. The exception was Christmas, when staff were normally expected to take a couple of days off and the lab was closed for at least a week. As expected, one important instrument didn’t restart. The reliable and competent operator called the service department, ran through the standard check list, but the kit refused to work. A service engineer was sent, who was charged out at "only" £95/hr including travelling time. After 3 hours on the road, he arrived and we made him a cup of tea - While it was mashing he walked around to the back of the equipment and turned the power supply relay switch on. The equipment restarted. The operator was almost in tears, "I always turn it off at the front panel, the back is hard to reach" she said - Nobody else admitted responsibility. To avoid unpleasant repercussions (it may have been her boss), the engineer wrote it up as a dirty switch contact (Well, we had given him biscuits too).
What is more it seems to have gotten worse and worse over the years. A few decades back this wasn't an issue at all, then I started to notice that I need to bring these under a bright light and lately it has gotten so bad that I had to add a pair of reading glasses to the toolbox in case I run into these. No doubt this is because the molds or whatever the metal tips made with are crude Nth generation copies of the originals.
Same here but there's definitely a problem with a lot of electromechanical parts when you hold up older vs newer ones side by side to compare
Barely readable stampings aren't anything new, but clearly marked parts often aren't obtainable anymore from the usual (or official) suppliers
The funny part is that you can usually find high quality well marked parts from china - but official channels won't buy them as they cost 1% more than the absolute cheapest they can lay their hands on
Years ago, back when IrDA links were a thing, we'd developed something which ran on a palmtop, and synched to a laptop over an IR link so that the data captured on the handheld could be displayed on the main application running on the laptop. This was ready to be demonstrated to the customer - the project manager wanted to demo it himself, so the dev team spent ages running him through umpteen rehearsals...including making sure he understood the importance of line-of-sight for the IR link.
Come the day of the demo, one of the dev team got a frantic call from the project manager who was in a state of panic. The application just wasn't working - the data on the palmtop wasn't synching to the application on the laptop. Customers were in the room. He needed technical support, and he needed it now.
The developer went up to the conference room where the demo was taking place, calmly turned the palmtop around so that the IR port was now facing the laptop, and departed.
IrDA was great and a pain in the proverbial in equal measure.
I recall once doing the month end tasks on a Sunday morning* while sitting outside my tent, having dialled in to the office. The only problem was the sun kept interfering with the laptop-phone connection so I ended up having to shield them.
*We used to get 1/2 day off in lieu for a couple of hours work, my boss was away and it saved me having to drive home before the lunchtime beer session. When the lieu time was stopped we automated some of the process and handed the rest over to the bean counters who should have been doing it in the first place.
This post has been deleted by its author
I remember a customer reporting a fault with an application - they were making changes to something on a data maintenance screen, but those changes never seemed to actually make their way into the system.
This was before the days of remote support over the Internet so I found myself taking a trip of several hundred miles to go and troubleshoot onsite.
Once I was with the customer, I got them to walk me through their workflow in the application while I had a terminal open to spy on the database itself. They opened the relevant bit of the application, did whatever edits they needed to make and then hit the "Go" button. For safety's sake we'd coded it so that this button popped up a modal dialog with words to the effect of "Any updates to the data will now be committed to the database, will trickle down to related records, and cannot be undone - do you wish to proceed Y/N". Selecting Yes did the business and selecting No cancelled everything and took you back to the menu.
It transpired that the user thought that the warning looked a bit scary, and they didn't want to do any damage, so selected the No option to be on the safe side.
Sigh.
At the time this particular company was running a campaign against sexual harassment in the workplace, and there was a poster in the office along the lines of "when someone says no, they mean no". This particular user henceforth became known as the girl who says "no" when she really means "yes". Thankfully it was all taken with good humour
I had a similar problem with a different solution. Our app maintained a customer list. You could add / delete / edit customers. We got a complaint that customers were mysteriously disappearing. Strange enough every time after adding a customer.
I added the number of customers to the title of the window. So after adding a customer, it would change from say "145 customers" to "146 customers". If you edited a customer, the window title stayed the same. The problem went away. Your guess why.
There's also misuse of the word "delete" instead of "remove". I can't remember the exact example, but something like: I want to remove this shortcut to this application. "Do you want to delete AppName?" Err... no of course not, I just want to delete the shortcut.
I've just checked this PC: it now says "This will only delete the shortcut to AppName. Do you want to delete the shortcut to AppName?" Much better.
ObWhich: a personal annoyance is Star Trek holodecks: "Computer, delete Troy character!" No!!!!! You might want that character again in the future, why throw away all those thousands of hours of work????
I may have posted this before but it seemed appropriate.
Many years ago (early 90s) I was a test engineer for a company that made 'smart' payphones; smart in the sense that they were actually small microcontrollers with support electronics that operated as a payphone*.
I had designed some of the actual electronics on the newer variant, designed the test interfaces / measurement circuitry** and wrote the various tests*** for production units.
The actual build of the PCB assemblies was contracted out (not at all uncommon even then for a company that made perhaps 1000 units per month at the time).
One fine day, the operations manager (who was a total <redacted>) came to my desk and very loudly proclaimed that my test was not working as all the latest build was failing a particular test in the sequence although he had not noted which step it was.
I grabbed a known good unit (in its anti-static bag, natch) and set off for the 100+ mile drive to the build house.
When I got there and observed the test, the unit was failing a check for the escrow relay firing circuit. The escrow relay is a hunk of metal that takes (or refunds as appropriate), the money in the hopper. My test instructed the microcontroller to charge up the circuit, which was then measured (it should be 120V) and this step was failing.
I put my known good unit on the tester and it passed without issues, so I knew the test was fine****. I duly grabbed a few failing cards and returned to the office.
An 'associate' engineer (who tended to look down his nose at test engineers) put one on his bench setup and ran through the escrow relay operation and declared it to be fully functional. I had noticed that the operation seemed sluggish (and his 'test' was hardly scientific) so I took the unit and rigged up the same circuit on a bench that the test equipment used (a resistor!) and probed the firing circuit with an oscilloscope.
When I operated it, the circuit charged up to about 80V and promptly collapsed to about 40V. The energy was stored on a capacitor after charging in preparation for firing.
When I had someone desolder the capacitor that was supposed to hold the charge, it had clearly delaminated so the breakdown voltage of the part was not the specified 250V but more like 80V. This showed me that the temperature in the build process was way too high in that zone of the board.
Replaced the cap and bingo, everything worked.
The operations manager had brought the CEO and VP of engineering, with the associate engineer in tow, round to watch my tests (in the hope of seeing me fail, probably) so when they saw that it was not the test that had failed (and had prevented defective kit going out of the door) the CEO congratulated me for a well crafted test. The operations manager and the associate engineer looked as if they wished the earth would swallow them up.
I got on the phone to the build house and explained the problem (they were quite decent about it) and they fixed the bad batch and the process.
The joys of schadenfreude...
* This was in the USA where it was legal for business to hang their own payphone on a wall (subject to varying regulations) where the local telco had not put one and (hopefully) profit.
** Bespoke electronics in a dedicated box.
*** Using EEPROMs in the game slot of a Tandy Colour computer! 6809E assembly code. The test box and target was controlled by some ports we had added to the game slot interface card.
**** On as previous occasion, engineering had changed a circuit (added a diode) without telling me and the test was failing as it had not taken that into account. Morons.
My wife used to write Java code - along with all of the Unit tests needed to validate it.
The code was sent to India, and she moved to a new project.
A year or so later she was contacted by someone from India who said there was a problem with the test - it was giving an error message. Should they fix the test?
My wife asked - have you changed the base Java program ? "yes" came the reply. "Well change it back again as you have broken it".
I've had similar just in the last week.
A product using assembler code. MOVxx source,dest does the operation and sets EQ/NE (and MI/PL) according to the data moved. Always. Code has been working for over a decade.
Recent report that code is failing on a new system. I narrow down the error to a single instruction pair, MOVW R1,-(SP) followed by a BEQ. EQ/NE wasn't being set by the move. I queried this.
"Oh, we rolled our own implementation of the hardware, and we didn't like the flags being changed when you pushed to the stack, so we stopped it doing it."
I was easier to just sigh and add a single instruction TEST R1 ; *BUGFIX* for faulty DF76.
I had one years ago when PC building. Customer rings up irate that his new machine is completely dead demanding we replace it immediately. So I start running though the usual checks but before I can he tells me with an air of superiority "I'm a Microsoft Certified Systems Engineer" to which I reply ok but I still need to run you through a couple of things. He was clearly not happy with this. So we go over the basics, is it all plugged in? is the cable secure at the back? both with snappy yes answers and then we get to the crucial one. Have you turned the switch on on the PSU on the the back of the computer? Dead silence. Then a rather sheepish ok it's starting up now followed by a little mini complaint that he had never seen a switch on a PSU. Some people...
I've mentioned the example before when I had a support call from some guy who had bought a top-of-the-range Sony Vaio that afternoon (early noughties). He'd paid close to £2,000 for it, as well! I was jealous, because it was a spanking machine.
He was well-spoken, and said 'like a fool, I gave it to my son to set up, and he's set a password which he can't remember'.
I asked how old his son was. He was eleven.
But hoping for a different answer, I asked 'do you get asked for the password immediately or when the Windows logo appears'.
It was 'immediately'. Damn.
After we exhausted every possibility that his son might actually be able to remember the password he'd used, I explained that the only realistic way of resolving it was to take it back to the store and ask them for advice (at the time, there was no way there was any online database of Magic Passwords, and there was probably no jumper inside the laptop even if the casual owner of a £2,000 machine less than 2 hours old could access it it without voiding the warranty). It was certainly something that wasn't covered under our warranty or something that could be fixed over the phone.
He said 'So is there anything else you could suggest?'
I replied 'Have you considered adoption?'
Fortunately, he found that amusing.
or "Must Call Someone Else"
I had one MCSE say they were going to connect up a NT4 fileserver of a major international pharmaceutical company customer directlt to the Internet
I responded in writing that owing to the liabilities involved, I would not continue to keep them as a customer if they were to do so, that they needed a bastion server and my own experiments on NT4 had shown me I could manually break into a NT4 machine over PPP connection in about 5 minutes without much effort, therefore I expected a directly connected unfirewalled system to last less than a week online before being attacked by script kiddies, probably losing valuable IP in the process
This was twisted into me making a direct threat to hack into their systems and a snotty letter from their lawyers was received promising to hold me to account if anything happened to their systems
They subsequently went off to another ISP, directly connected their machines as promised (unfirewalled) and got hacked in a week, shutting down their production plant for 3 months. Shortly after that, they were taken over by an even larger pharmaceutical company and the MSCE concerned was no longer employed
We had a technician installing a data collection system in Brazil back in the 90's based on a DEC PDP-11, they had some problems and worked on the PDP to try and fix them but nothing worked so they came back to the US, I flew down with a a box of boards - I swapped all the boards which solved the problem and then switched the boards one at a time to find the problem. I found the problem board and I looked at the chips and noticed that an EPROM was inserted backwards, I pulled it out, turned it round and saw a bent pin, I straightened it, plugged the EPROM back in - everything worked!
The lesson that I passed on to the technician when I returned was that debugging problems can cause problems, and problems can be educational.
I had almost the exact same experience.
My company provided hardware and software to a certain green suited US government agency. The hardware was custom, the software was both for the custom hardware, and test software running on a PC.
We got a call. The test software wasn't working -- it would drive the hardware properly, but it would not receive the test results. This was over an RS-232 link.
So, I get to throw every single piece of serial port diagnostic hardware I can think of (short of a full-bore protocol analyzer), and get on a plane. I get to the site, and examine the customer's hardware -- both ours and his PC. The guy had a PC with two serial ports configured by DIP switches. Both ports were configured as COM1:. Flip a single switch to configure the second port as COM2: and Bob's your uncle.
So my company paid for a round trip flight, and per diem, for me to flip a single DIP switch.
In my helldesk days my company decided to save some coin by having the internal helpdesk take on support for a recently acquired business that sold laptops to real estate agents. We were a printing company, so the Venn diagram of common issues and expertise was two circles on different sheets of paper.
One user called in because his laptop wouldn't print. A colleague and I tag-teamed with this guy for 4 hours. Plugging and unplugging cables, reinstalling drivers, updating firmware. anything we could think of.
At the end of 4 hours, the guy says "hey- when I plug and unplug that printer cable- does that need to be connected to my laptop? Because I have it plugged in to my other computer right now"
My colleague quit to go to law school. Can't blame her.
I work in R&D for a product where lots of stuff goes over network links, and double up as 3rd level support.
After seeing a plethora of log files pushed my way over the years where the only sane answer is that there is some problem with the customer's network, I asked one of the support people why we keep getting these coming our way.
The answer - the customers will only get their network teams involved if our R&D tell them they have a network problem.
Strangely enough, I never hear about these problems again.
Back in a past life, I inherited a compression product that the company I worked for had bought from some other company in the US.
After spending some time trying to make sense of the code (all IBM assembler), I gave up and rewrote it from scratch, not, for sure, always the best idea, but it was spaghetti, and spaghetti assembler is really not fun to work with; four base registers and yuck!
Anyways, we stuck it out as a new release and sure enough, got our first (and only, IIRC) customer case raised.
This particular customer had an application with used APPC to transfer <large number> of records from an AS400 to a mainframe via CICS. Their complaint was that since installing the latest version, their CICS system had started reliably going short on storage (not hard on a 31-bit DOS/VSE partition), and they were not happy about this.
Ok, said our support people, send us a dump of your CICS partition and we will take a look. Transpires that said dump was full of WQEs (work queue elements), these being the ways that CICS remembers which records it needs to write to it's logs in order for dynamic transaction back-out to work.
Transpires that the original software always returned the same value for the RBA (relative byte address, this being VSAM) so CICS thought it was only dealing with one record, constantly updated, and therefore only requiring one WQE to keep track of. The rewritten version accurately returned *all* the RBA values for the records they were writing so their crap application suddenly failed.
Our suggestion was that they should issue a COMMIT every 100 or so records on the CICS side, which, despite their somewhat grumpy replies, solved the problem and we never heard from them again. Probably because the support guy made a point of calling them to make sure that they were delighted by our prompt attention in telling them that they had a crap application (he didn't put it quite like that, but I would have).
Twice in the last month I've been dispatched to troubleshoot and correct lack of AC power to the alarm panel in a retail shop that has $CURRENCY in its name. At one site, the utility had lost power in most of the town. At the other site, the utility had lost 2 transformers outside the shop. By the time I arrived the next day, the lights were back on, so it was "no fault found" after an hour or two driving to the sites.
You'd think that the company would contact their customer to ask if anything unusual was happening. But probably the phones didn't work, since the phones were converted from POTS to VOIP some time ago. You'd think that they'd call store manager's mobile, or the district manager, or something. But they didn't.
Oh well, the pay's the same.
This post has been deleted by its author
For all the years I've been on call I have always remembered Rule One:
When you've got the customer on the phone do not shy away from asking the obvious questions.
No question is too obvious and if "Felix" had asked "Are you sure the connected in the active position? No really, go and check it now." Then service would have been back up one hell of a lot quicker.
Many years ago I bought a new PC from a local system bulider who had frequently helped me out with odds and ends in return for my expertise where he needed it. However, he had connected the flying serial ports with the leads the wrong way round on the I/O card. He had also not explained how ramshackle the case, motherboard and card setup was, nor warned me not to undo the VLB cards myself. I had to remove the card to correct his mistake, but then found myself unable to get it back in, and the graphics card also adrift. He had to come out and reassemble it all. He and his friend between them often had problems like this - but if they gave me one to assemble it was always correct.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
