Techie ended vendor/client blame game by treating managers like toddlers Welcome once again to On Call, The Register's Friday column in which we share your tech support stories. This week, meet a reader we'll Regomize as "Warren," a network engineer who found himself working on a project team that was trying to clean up a big stack of open tickets that had accumulated during the merger of two …
Those on each side feel like they have enough on their plate without investigating an issue where they are guaranteed to find nothing, should it turn out to be the other guy's fault. Game theory dictates that neither should put forth the potentially futile effort unless something happens to disturb the equilibrium - either it is costing the hospital too much money from being able to charge fewer patients for expensive scans, or the vendor feels they are risking future sales if the hospital is unhappy with the quality of the product/service being provided.
Or someone who sits outside that standoff and takes pleasure from the simple act of solving a mystery is brought into the mix.
Game theory dictates that neither should put forth the potentially futile effort
Except being stuck in meetings with a complaining client wastes more of your time than investigating. And when you investigate, you don't just check *your equipment* you also check the handoff to the other vendor's equipment for daming info, and even if that's not fruitful, you've generated metrics that you can compare against the vendor... e.g. "Data was transferred successfully to you at X, but it took until X+30 for an acknowledgement/response... What is wrong with y is your equipment taking that long?"
Unfortunately, there's also companies with horrible buggy old equipment they won't replace after repeated failures, and who contact the vendor as their first troubleshooting step when anything is slowing down even slightly...
Game theory clearly doesn't include the smug satisfaction you get when you *prove* its definitely the other guys fault. That makes it worth investigating every time, if nothing else to avoid being on the wrong end of the smug satisfaction from the other guy !
Same, have had it literally this week with a vendor trying to use the PTFE shoulder pads to shift the responsibility for a problem with their product onto my team instead of sending out a tech to resolve it.
They've tried the threat "if we dispatch an engineer and they can't find a problem we will invoice for %outrageous sum%" so we've just turned it back on them and suggested if they can't be bothered then we will look elsewhere when the current contract is up in 7 months time
Vendor: "I believe the issue is on your end".
Me: "Have you looked into it? I believe it may be on your end because of reason X and Y."
Vendor: "We will look into it if you 100% irrefutably prove it's not on your end"
I've had this discussion a thousand times... Bless vendor support that actually tries to cooperate.
Me: We have conclusively proven that there is packet loss introduced between router F and router G, both manufactured by you and directly connected with a DAC copper cable delivered by you. Can you help us troubleshoot this issue?
Vendor: It seems that the server hooked up to router A in your data center is connected with a third party SFP. Before we can troubleshoot this issue further, we're going to need you to replace that SFP with a Genuine Vendor Ultra Plus brand to ensure that the signal quality is good enough for the path.
Me: ...that's not how any of this works?
Vendor: Case is now Cust-Pending.
Customer: we will replace your kit with something from a reliable supplier, please arrange to collect yours and close the service contact. Or, as an Australian client of ours once memorably informed us after 4 months of fruitless remote debugging, "if I don't see an engineer here by the end of the week, your kit is going in the river!"
4 days, one expedited visa, and a very expensive flight later, I was in his office, very jetlagged, on Friday morning. Took me a week to find and fix the (genuine) bug.
Similar. We had a sample ink dispenser in a factory in Pittsburgh that would randomly fall over. Months of remote debugging failed to provide a solution so I got sent to the US to baby sit it. Took the better part of a week for it to fail, but ended up being an error caused by the ROUND function in Foxpro (this was many years ago). The solution was to convert the number to a string, convert it back to a number and round it again. I never did find out why, but by that point I just wanted to go home.
Had that years ago at a company. System/36 and a /38, all IBM kit including some of the new fangled PS/2 range :)
Had a problem.
IBM: The PC's are not valid
Me: They are all IBM
IBM: the emulator card is not ours
Me: All emulator cards are IBM and all the software is IBM's - no Rumba
IBM: Your network is at fault
Me: All our MAU's are IBM
IBM: Your cabling is not to standard
Me: Nope, it is all IBM <whatever - big old fat unwieldy Token Ring with mahoosive connectors>
IBM: Oh.....
This was the days where my weekends were often spent with long arse bits of curtain rail, roles of tape and lots of shouting of "pull it","push it" in stupid conduits. So glad when we moved,but with rose tinted glasses they were fun days
This is even more fun when there are two Vendors involved in the system. It's basically that meme of the three Spider-Man cartoon characters all pointing at each other.
Vendor 1: It's definitely an issue with Customer network
Vendor 2: No, we checked that, it's definitely an issue with Vendor 1 widget
Customer: It's definitely one of you two. Or if it isn't it's IT's fault
Me: Girls, girls, you're both ugly. Here's some evidence I gathered earlier (wireshark, netflow etc...) now let me know if I can help you with anything else.
I got dumped (On Call, 6 AM Sunday morning) into a conference call with lots of vendors (I'm not sure the exact numbers) of similar poor On Call chaps (poor because they apparently spent the night on the problem, and I only got called because in the path of the problem there wasrouter I was doing support for.)
When they presented the issue, they told me the call was more a matter of making sure everything was OK on my router, as they had obviously reached the end of a (long, since also apparently they had been investigating the issue since noon on Saturday), and wanted to get reassurances at that point.
I connected to the router and checked it's health, everything looked fine, no logs that shouldn't be there, no obvious issue... After reporting that they all went back into WTF mode and started to plan for some massive stuff with lots of Impact to the end upser (I didn't say it before, but it was for a mobile Network, and basically they were loosing traffic randomly on cells in a quarter of the country... the kind of massive outage that eventually make the news if somebody adds 2+2)
Since It was clear that they had explored lots of potential cases for the issue, I went digging deeper in the router and used some hidden commands that doesn't appear in the documentations... This showed me that on a specific card there was drops (as in packet drops) in the backplane...It's not an issue if it's a drop once in a while, but the number seemed high and I didn't know when it was last checked (getting the value reset the counter... a bit Schrodingerish, you get a count, but you don't know when it started counting [obviously if you loook for when the counter started you won't be able to get the count.And doing either resets the counter)
Since the router was a piece of backbone, beside the traffic that had the issue, there was a lot of other stuff (early days FTTH collection and Internet access, ADSL collection & Internet Access, lots of corporate L2 and L3VPNs) that might be unhappy if the obvious solution that came to my mind to check if there really was an issue was used : reset/reload the traffic card.
Anyway, I mentionned that the only thing that didn't seem right to me was these drops in the backplanes and I told them that my first move would be to reset/reload the line card, but since it would disrupt lots of other stuff, it wasn't my call.
Since they were clearly at loose ends they went with it, and I remember clearly the Alcatel guy saying alarms were disappearing faster than he could see on his supervision platform while the Huawei and Nokia guys both explaimed that they both saw all their base stations and RNCs back.
I didn't check on the aftermath of the whole thing but I think the card got replaced ASAP after that.
the fun stuff, is that when they called me, they really didn't expect I could help them solving the issue, they just wanted to close a door in their investigation.
I feel seen by this post. All too often I have the "It's not us" despite packet captures showing it's them.
"No, it must be something your side does to the packets!" - despite there being proof that the packets sent their way are standard. And then, months later, someone else along the chain intimates that said vendor found something, changed a setting or replaced some hardware, and suddenly the problem went away.
Were we told? Not on your nelly!
Working for a bank.
When cards are used it goes through a complex path to us where we would respond if they had the money in their account. If it took more than 7 seconds for the transaction to make it to us, our upstream provider would auth for a us with a set amount even if the customer was overdrawn. We would have to pay it.
At certain times of the day, transaction would exceed the 7 seconds and the bad customers figured it out and would drain to the max amount leaving us with the loss.
Naturally IT got the blame.
Love wire shark. With a capture, I was able to show their server was busy. They told me there was no way I could know that. I let them know their server told me. Then i showed them the packets where they responded with a small window size of a few bytes, basically tar-pitting the transaction.
At that point they admitted that was when Master Card dumped bat transactions. Sadly, our management did not make them pay us back the losses.
I had a similar problem about 10 years ago. Working for a large insurance broker they wanted to use this system to provide enrichment data for quotes (in simple terms how much could they adjust the price before you went elsewhere - there was more to it than just that but). In test the system worked great but before we went live we wanted to ramp up to production levels at this time it was about 10-30 quotes per second which had to reply within 6 seconds to be useful). It couldn’t cope and yes the developers were finger pointing, I looked at a wireshark trace and lots of ssl connections were terminating at 15 seconds duration and all with the same payload size.
Hmm reading on the ssl specification and this told me that is the timeout when the handshake fails - left it with some rather red faced engineers to go and talk to their bosses about more capacity was needed on the 3rd party side.
My company thought this was a very good outcome.
"We'll look into it if your department signs this NDA." Boss signed, we really needed to know.
It was their fault. Their redundant-everywhere storage array had a single point of failure.
I tried to warn management before we bought it: a quick web search found a company whose business was recovering data from this vendors failed arrays.
But the other vendor's bid was higher. Even though there was no evidence that their arrays had ever failed, they lost the bid. (A year or two later, one of their arrays did fail. It was so unusual it made global IT news, including El Reg.)
Vendor here for a counter point...
I cannot count the number of times I have experienced the reverse role. "Send out a tech for free, this is all your fault, we will never spend another dime with you, we will tell all our tech buddies you suck, yadda yadda".
So a Tech goes to site. Problem is as expected. Customer did not follow the documentation <here>. Which we referred them to multiple times in <these emails>. And talked to them about on <this date> and <this date>.
Just as funny as this story is on El Reg, it can also backfire on the customer. Customer's tech doesn't have the technical acumen to properly implement the product. So the vendor gets blamed internally and often with hostility. Then the big meeting happens with the big bosses. The vendor's tech lays out all the evidence in nice order for the big boss. Either the meeting is cut short, or the customer goes on mute for awhile.
No apologies are ever given, but when communication resumes it usually starts with <new person> is taking over this project and will be your contact going forward.
I can imagine that all too well... not necessarily from the vendor side of things, but dealing with customers in general. Sometimes it's more about managing their internal narrative and adjusting their expectations than actually solving an issue.
It also boils my piss when requests start with "your product is shit and you are shit". 90% of the time the person sending that request is, in fact, shit, and failed to read the documentation or did something silly.
Just this week we got someone complaining that a SharePoint site didn't show up in their favorites, nothing had changed and it just suddenly happened. Turns out the very person complaining had actually deleted the entire SharePoint site himself!
"your product is shit and you are shit"
Good Lord! Those customers are the worst. I work for a company just like they do. I want to take home a paycheck just like them. They will get so much further by using different language... "The product is garbage" instead of "YOUR product is garbage". "XYZ's tech support sucks" instead of "YOUR tech support sucks". Like I personally own the company.
Blame the product. Hate the company. I just work here, don't make it personal. I work for your vendor. I really do want to help you because the company's success keeps those paychecks coming.
Just because the customer wrote the purchase order, it does not entitle them to vent all their other frustrations on the person who took their phone call. Aggression will get the formal documented answers every time. A friendly and collaborative request will get the same answer, but often with a bit extra like "here is an unofficial way to make it work".
I always stay on the good side of the support people when having a call logged with them.
Surprisingly I have found that if you don’t rant at the supplier then you actually get better service. Admittedly I know our service manager, and she usually just says “ok if you have logged the call it must be a big problem” as anything simple I can normally fix. Occasionally I have asked for a ticket to be transferred if I feel I am not getting anywhere but for that I go to the service manager and has her to reallocate and explain the reasoning.
At the end of the day the people on the other end are trying to do a job to the best of their ability so treat them as you would be expected to be treated yourself.
And relax!
This is the same problem as always.
Either the Vendor(s) or the Customer, or at worse BOTH, decide ahead of evidence that the blame is elsewhere & assumptions are made following this !!!
I have been the 'vendor' knowing that the customer is wrong BUT will not admit that they did not follow the instructions given 100%.
I have also been the 'customer' being told that the fault is 100% on our side and that we are at fault somewhere, when I have tested everything possible and still not found the fault.
This comes down to the fact that 'People lie' sometimes it is deliberate sometimes it is because they do not understand something ... this happens with both customers & vendors.
All 'FACTS' need to be checked on all sides.
Vendors can be arrogant just as much as customers !!!
You need to have a relationship with your customers where they trust you enough to 'check' something even if they are certain it is NOT at fault.
This relationship means that you as a vendor must be prepared to do the same.
I have witnessed too many times the 'It is not us !!!' line turning out to be not true !!!
Usually you don't get an apology if proven to be right BUT sometimes you do which is nice !!!
:)
I've been on that end too, @Marty. Either way, instead of shouting at the other end saying they're wrong, having the irrefutable evidence always helps. And then when third parties are involved too, I can show you what's on the wire and what's wrong with it, *not* what the other side or the third party is doing. That's *your* part of the network, and networks don't do telepathy (God, wouldn't life be a lot easier if they did?)
:-)
I have been on both sides of the equation (me career went from vendor to customer to consultant to a non-IT role, soon to go to "from July, you're on your on, boys" aka retirement).
I freely admit to overlooking the (in hindsight) obvious blunder we'd made, but also saved the bacon by thinking outside the box and/or around the odd corner or two - like triggering a bug in the core switches' OS, causing a complete network shutdown on a maritime asset (fortunately still in the shipyard, so the suppliers could do a full investigation and proper fix rather applying a band-aid and hoping for the best).
But I did my est to remain reasonably calm, polite, and professional - the one time I had to raise my voice at a telecoms provider, I profusely apologized in advance to the poor key account manager on the other end of the phone line.
The cloud is no better, I am having a discussion about a managed database as a service that had latency issues. The vendor (who may be a little poorer this week) is pointing at an increase in traffic from my end 3 hours into the increased latency. I've spent countless hours drawing pictures using the vendor's own tool pointing this out to no avail. My last missive has a new support guy, who will no doubt start the process again, directing me to the manual...
Modern networking tends to be reliable and fault tolerant so people have either forgotten, or aren't aware, of how things were "back in the day".
Touching wood I can't think of how long it is since a single network interface failed but in the days of thin (10 base 2) ethernet it wasn't unknow for us to have to go round an entire segment systematically disconnecting individual machines until we found the one with the failing NIC (and that's after first changing the terminators at each end as they were also known to fail). The difficulty was that depending on the failure it could impact the entire segment, not just the one machine.
The plus point though was were were dealing with desktop machines with separate NICs*, not with network interfaces integrated into the motherboard, so it was generally just a case of swapping the failing card for another one (invariably NE2000 based), checking the interrupt settings and installing the driver.
*I think the last failure I had on a laptop the only solution was to disable the internal networking and use a USB - Ethernet adapter.
Back in the late 1980s, there was a company in Taiwan which "recycled" MAC addresses on its clones of NE1000/2000 ethernet cards. When you got a new batch of cards which matched the MAC address of one or more cards on your existing LAN[0], much hilarity ensued. As a consultant, the first time was the worst ... after that, the symptoms were fairly obvious. I ran across the problem at probably a couple dozen small companies between '88 and '91ish, and then again (!!) in the mid-late '90s, when people started recycling old Netware kit for Windows networks at home.
[0] An "impossible event", at least according to Novell and IEEE.
Apart from sloppy network card vendors work (rare, have never seen it personally): I a virtualized environment it happens more often if the mac-pool of the virtualization solution across several hosts is not done right. I had such cases with all solutions. Depending whether the double mac happens on the same VLAN or not the virtualization solution notices such collision, and usually force-changes the mac of the VM, but only when turning the VM on, not when it is already running or only rebooting. Though not seen, but some may simply refuse to boot the machine due to the double mac. There can also be cases on the same mac but different VLANs, which might make switches go weird whereas the actual functionality is not disturbed if the switch can handle such a "barely legit" case.
For an somewhat challenging introduction to the subject, Wikipedia has a good article on "MAC address". It stands for "medium access control", with "medium" meaning the network. "As typically represented, MAC addresses are recognizable as six groups of two hexadecimal digits, separated by hyphens, colons, or without a separator", and every "NIC (network interface controller)" in the world is meant to have a different MAC address. Data is sent on the local network from one NIC to another using the MAC address to select the destination NIC.
This causes trouble when a manufacturer actually releases NICs which have the same MAC address as each other. They are meant not to do that.
If you do have a duplicated card, then you may have to throw it away. "Many network interfaces, however, support changing their MAC addresses" - for instance with a software command while the host boots up or connects - so you could do that and set an acceptable MAC address, or even a random hexadecimal number as the address - I refer you to Wikipedia for reasons to do that. A genuinely random address is extremely unlikely to be a duplication.
I'm not certain but I think that if your local networking connects every host directly to a central device (hub), then the local network may have only your host and the hub talking to each other by MAC address anyway - or something like that - and then it wouldn't matter.
For a while, some software licensing and copyright protection depended on the software being used on a computer with one MAC address that you'd paid for. You could possibly override that by replacing the NIC and then setting its MAC address to the required number. Reasonably of course - PCs and network devices do fail and need to be replaced.
I had one customer who insisted on administering their own MAC addresses (!) on Token Ring and this then carried over to Ethernet. The wanted all theri MACs to be FE:01:a.b.c.d for TR, with a.b.c.d the octets from the IPv4 address, and FE.02:... for Ethernet. Never heard of ARP? "But that's so much easier!" - No, it isn't.
This post has been deleted by its author
My first ever role was as desktop support back in the dying days of the 90's and part of my role involved supporting the token ring network. We would get regular multi day network outages caused by someone tapping a cable with their foot under the desk and I learned the importance of a long handled screwdriver to try to listen to the patch panel to isolate the clacking switches whilst someone wandered desk to desk reseating the gigantic connectors!
Not a story from me, but my favourite networking SNAFU was (again the 90's) an ex worked for one of the UK's premier ISP/Hosting companies who ordered a large number of servers from a big box company (was probably Compaq at the time) and upon installation none of them worked. Upon further examination and a bit of fettling it was discovered that every single network card had been set up with the same MAC address.....
Ahhh, what is it with users, and (at the risk of sounding sexist) particularly women ? Back in the days when I did the "scrabbling under desks for a living" stuff, it seemed that now matter how neatly you fastened the cables, or how far away from the chair they were, someone would manage to drag them out from under the desk and drive the chair castors over them (or otherwise mangle them.)
And on MAC addresses, I heard a tale from a local university where they bought a large number of computers from Rodney's brother. They then had some of the "interesting" issues mentioned above. It took a while to track down, but they found that the manufacturer had a bug in their MAC allocation system which meant that for every 256 machines with unique addresses, they'd fire out a 257th one with a duplicate. Of course, for most clients it would never manifest - they had to buy more than 256 in one order for it to cause a problem, and this university had bought "thousands" for a large tech refresh.
"that's after first changing the terminators at each end as they were also known to fail"
I found a lot of 50 ohm BNC terminators whose centre pin had fallen back into the terminator body. Got into the habit of pulling them off and checking on a new site (and marking as tested) as well as keeping a pocket full of verified ones.
They really were cheap shit with dodgy soldering that had a nasty tendency of fracturing when pressure was put on the pin (such as when it engages the centre conductor of a tee-connector). It wasn't enough to simply look to see if the pin was present. You needed to actually grab it with a pair of pliers and make sure it wasn't freely spinning.
Second most common issue was badly assembled connectors. You'd think crimps are difficult to screw up but I saw more bad ones than properly made ones (Disclosure, as an Ex-RF tech, we got TRAINED how to do them properly. Attempting to show installers or apprentices the correct way (or getting them to RTFM) was very much a case of "I've been doing it this way for years and I'm not changing it now". That's an attitude which started changing when we mentioned the badly made connectors in our report and suggested that the client recover the "making good" costs from the installers
Crimps made correctly will outlast the entire rest of the network and are vastly more reliable than soldered connectors.
Made badly you'll be lucky to get a year out of them before odd things start happening.
One job I had included upgrading by replacing EPROMs. The existing staff used the "grab it and keep pulling until it snaps" method. I made a dozen little tools that you could gently lever under each end so there was no sudden jerk as the pins got just loose enough to escape all at once. After giving them to the staff and demonstrating them, they *STILL* destroyed 90% of EPROMs by just yanking them out.
I can't think of how long it is since a single network interface failed
Just a few months ago for me. Strangely slow performance on VMs on one ESXi host in a cluster, which all performed fine moved to another host. Wasn't any slap-in-the-face signs of the NIC completely failing, but things on the host slowed down over time as network traffic went up.
Local technician didn't take my suggestion of replacing the NIC, SFPs and cabling in one go. Instead weeks and weeks of him pulling out fibre optics and blowing on it... changing which PCIe slot the NIC was in... Swapping the location of the two SFPs, etc. Each time, the tech declaring it 100% working now, then running for days before problems grew to the point it couldn't be ignored, and repeating the silly dance.
We used to fit small 10base2 networks into GP surgeries.
We had one Doc who wanted the network extending to another room. We quoted him for an extra socket but he said it was too expensive.
We got a call from him a couple of Monday mornings later saying his network was down.
I visited to find that he'd extended the network himself with some 75 ohm TV antenna cable and a lack of termination.
Or the surgery where the network was terminated at the wall (no PC in the room) and babies were fascinated by the shiny little chrome button that they often removed it from the socket.
My very first experience of a "network" was a PC (Yes, PC, not an AT or better!) with an 8 port RS232C card in it (8-bit ISA slot) which involved running RS232C cables to each individual client PC. It seemed like such a relief to get 10base-T coax and only have to run a single cable around all the clients. My euphoria was quickly dashed!
I was involved in a set of long standing problems with a customer. We were a networking product, and the customer complained that periodically the throughput dropped almost to zero - and by the time they came to look at the problem it got better.
I think the customer found our support teams friendlier than their own support team.
I got them to have a simple program which every minute did a ping to the the remote end, and capture the response time.
They sent me the output, and I could see normally good response times with the occasional spike.
Our emails crossed mine said "did you have a problem at 0713" theirs said "we had another occurrence at 0713".
The customer asked me to present to their network support team, and it took about half an hour.
After we had finished we got push back from them "we do not see any errors/problems in our logs"
me: "What response times were you capturing for this time period"
tap tap tap.. "ahh we are just going on mute"
Eventually they said "we don't have any monitoring enabled on that part of the network" - which is why no errors were reported!
The networking people found the problem and the throughput problem was resolved.
The people I was working with still kept the ping every minute.
I once pointed out to Private Eye that Trump must read the mag...
During COVID, they had a cartoon saying something like 'we would have better figures if we didn't do so much testing'... a week or so later almost the same words were uttered by the Orange One during a briefing!
It was valid issue. For the majority of the time, the absolute majority of cases were asymptomatic and harmless, yet those huge figures were used to justify endless biosecurity theatre and authoritarian impositions on daily life. If we tested for flu today the way we tested for COVID back then, we'd think we were in a permanent pandemic.
Well, with vaccine USA only had over a million confirmed corona deaths until the virus mutated less harmful, since killing the host does not make good survival. More than 0,3 % of the population. Without vaccine, who knows? Maybe Trump back then wanted more to "rid the population of the weak, please at least 5%", but couldn't really say that in his first term.
I don't understand companies that behave like that. I'm a freelance programming consultant. When one of my customers calls to complain about the behavior of one of my programs, I'm damn well going to find out what's going on and what I can do about it. If I do find out that the customer's network configuration is the basis of the problem, I'm not going to lay blame, I'll just explain the situation and what I can do to fix it.
In an entirely different domain, my shower has recently developed trouble in getting hot water. The best I could get was a tepid shower. I called the artisan who installed the equipement and asked him to come over and and fix the issue. He was there the next day and, following my explanations, had already pretty much figured out the problem. He had come with a replacement piece of equipment but, instead of just replacing the faulty piece, he audited the entire bathroom. From that, we found that the hot water balloon was making the water too hot which, in turn, was dilating the joint in the shower's temperature dial which, in the end, was rendered incapable of delivering the desired temperature. We agreed that, apart from replacing the faulty piece of equipment, he would also set the balloon to a lower temperature that would avoid creating problems in the future.
What I want to demonstrate here is that a true professional is going to do his best to ensure that the customer is satisfied when he leaves. It requires dedication and knowledge of all the areas in connection to the one you're called in to work on, but such a person is priceless.
Companies are not priceless. What does that say about where we're going ?
I guess that the main issue is that you're incentivised by customer satisfaction and the resultant word-of-mouth recommendations that might lead to more work. The support bod on the other end of the phone is more likely to be incentivised by making sure that tickets are closed** or pinged back to "waiting customer" as quickly as possible.
* "closed" doesn't have to mean "fixed", of course.
Your description of the 'support bod' is what makes me angry !!!
Just like the artisan has pride in doing the job properly, so should the Support bod !!!
I have, in my career, worked on many Support lines etc and run multiple 'Support Functions' in various large corps and this is what I found most frustrating.
I had to 'encourage' the people in the organisation to agree that doing the 'right thing' was the order of the day and NOT closing the call as quick as possible.
I take pride in whatever I am tasked to do, and will do my best at all times.
I cannot do 'Its good enough' and 'running for it' to close the job down asap.
My background to date covers both Techie and Management arenas.
I understand that Techies are under lots of pressure to close issues ASAP ... been there done that.
Quick solutions that some else then has to solve tomorrow does not save any time and just adds to the pressure.
Do it right and do it now !!!
Stop setting unrealistic targets to close calls in 5 mins etc ... it just spawns multiple calls over days weeks that create unhappy customers.
'Gaming' the stats to look good gets found out eventually !!!
:)
This post has been deleted by its author
But a guess: You did NOT say "fix this as this given price I quoted from you before you even look at it", right? I hear from time to time that some, for the problem type you describe, want a fixed price ahead, which then causes more problems and cost that just "You hour rate is X? OK, come over!"
This post has been deleted by its author
And I'm not talking about programming languages, where most of us are fluent in half a dozen or so.
1: Regulatorian: This is the language of politicians and lawyers. It sets the mandates on banks, hospitals, schools etc. It contains nuances and terms of art that sometimes make a word mean something totally different to what you would infer if you heard it in general conversation.
2: Beancounterese: Spoken by accountantrs, salesmen and middle manglement. It sounds very similar to regulatorian but is sufficiently different in some of its meanings that it's as big a gulf as between old scots and english.
3: Geekian: The language of hard science, mathematics, real-world realities and the only one to use when specifying what a programmer needs to code. Because they will code what you tell them to, and it will work the way this language describes it.
The same word can mean different things in these three languages.
We have to be fluent in all three to accurately interpret requirements and predict what the emerging software will look like, to take error logs and demonstrate to (sometimes hostile) manglement what corrective action is needed and where it needs to be applied.
It gets worse, as there are quite a few Geekian dialects. I have learnt to speak a couple over the years, and know the word "morphology" can have radically different meanings, depending on whether you are talking to a medical doctor, an astronomer, or an image processing specialist. Great fun when you are in a project with different geeks each speaking their own dialect.
This post has been deleted by its author
This post has been deleted by its author
Yes, some of my Indian colleagues have similarly senior technical job titles to me and yet know Jack Sh1t.
Most of our twice weekly calls with them are teaching very basic stuff that I used to drill into European Interns in the first week. And often the second of the weekly calls is going over what they didn't understand in the first - despite very clear and simple self-explanatory diagrams (shared with them during and after the call) in case it was a discrepancy between the Kings English and the abomination they speak too fast!
And some of them have actually been promoted above my job title level and get relatively higher pay than me!
If they put as much effort into doing the job, as they put into arguing about it, then we would get on much better.
Teacher icon as that is what I feel like for half of the time.
I was having an argument with somebody about "transition elements".... y;'know elements in group 3 to group 12, where the d-block is being filled, iron, nickel, copper, gold, titanium, etc. No, in their head they were arguing "elements needed to transition the energy economy". Sorry, that's not what that word means, you can't just make up your own meaning.
Ping has a very specific meaning in our world, and it has slowly worked into other nearby worlds where people try and use it in a technical context, as if they know what they are talking about....
Ping me an email.
A phone pings the phone mast.
I'll ping a message to the users.
etc, etc.
argggh !
You hit a triangle (musical instrument) = ping. You go submarine, another acutal ping. You use pings in a cave too to get some mapping if light is not enough, though LIDAR has replaced most of it. You ping on a piece of plumbing to know where it goes. You ping on a pipe to find out whether the response time is the expected (aka end of pipe or expected crossings), or you get an answer too soon or one too often (i.e. damaged pipe). The last is the one closest to the meaning of an IP-Ping+traceroute. Same with light-ping, used on fibre optics to know whether you have a damage and if, tell roughly where the damage is.
Yes, some real pings there. Ping generally being an acoustic response to a sudden disturbance, or something like that. Our ICMP ping of course was mutilated from submarine/sonar "pings" when a remote object returns a ping. Not sure if that is real or just in the movies....
But emails do not go ping. Messages do not go ping, and mobile phones/masts do not go ping (unless hit with a hammer). I'm sure some of our readers can enlighten us on the real technical term and process for the latter, I would regard it as being network enquiry/registration and network acknowledgement. I spent a lot of my early career working with protocols that revolved around <enq> and <ack> of remote devices. We had not heard of ping in those days.
Used to have similar finger pointing with our network team. Network failed, so we'd drop them a ticket to investigate. "Replace your NIC" was the almost inevitable response. "No, pretty sure it's the switch port or cabling, like it's been the last 20 times we've had this fault". Nope, "Replace your NIC".
We got into the habit of switching cables to prove our point. I can't recall any occasion the fault didn't follow the cable/switch port.
Or speaking to the storage team - slow disks in the EMC arrays - "Everything is fine our end". Are you sure? "Yup, everything is fine". Well, we're seeing this and this... "Oh, we found a hot spot on the disk and have rebalanced"... *sigh*
Yeah, had the switch port issue many times.
For instance user was reporting intermittent connectivity issues with Outlook in online mode which means it came through as a messaging ticket.
Obviously no fault found at our end and if she used colleague's PC all was fine.
Asked her to kindly try swapping the network port (not the cable) with her colleague who then started having the same issue.
Informed networks team in her building who of course denied anything could be up with their kit in any way shape or form. Must be the patch lead, nope tried that.
Eventually get them to repatch her at the switch end to a new switch and all is well.
Turns out that she'd been on 'one of those' switches which was slowly failing but they couldn't be bothered just getting the thing swapped out, hadn't flagged it up to anyone senior and were just moving people as they complained in the hope the thing would cure itself.
"'one of those' switches which was slowly failing but they couldn't be bothered just getting the thing swapped out, hadn't flagged it up to anyone senior and were just moving people as they complained in the hope the thing would cure itself."
You see something very similar with telcos and street cabling in My ISP days I made myself very unpopular with the Telco's contractors on several occasions by insisting that my telco sales manager look at fault and resolution reports for all customers on a given street (usually on the same drop cables)
More often than not there was a LONG traiil of faults dealt with by "lift and shift" (moving the customer to a random other pair) and too bad for whoever got landed with the faulty pair(s). The result of higher ups actually looking was almost always an engineering dig to replace the cable and the contractors losing the steady earner they'd been using to pump up their income (They _knew_ which pairs were failty, but by lift-and-shift they could close the ticket and get paid)
One town virtually halved its fault rate once the telco realised what was going on. Customers were a lot happier but for obvious reasons the contractors weren't.
Know 'Lift & Shift' quite well !!!
Had it done to me to fix a problem ... then a few years later when they had to fix another problem, in the street, my line was 'accidentally' disconnected at the exchange and I got it fixed with what was 'discovered', eventually, to be a faulty pair !!!
Oh!!! what a surprise ... where did that faulty pair come from !!!???
It was 'Openretch', of course !!!
:)
Yes, similar problems with BT/OpenReach. 1920's cabling between us and the exchange. Paper insulated wire known to be wet. When you complained you got switched to one of the 10 pairs in the centre that were driest (and fastest/most reliable). Then the person, who previously had that pair and had been moved to the outside, now complained.
We talked to each other in the village and could map the complaints, the service visits and the new complaint - that's how we knew exactly how many good pairs there ware!
I took it all the way to a personal meeting the East Midlands Area BT Manager. "We have already upgraded the exchange and have no plans to update the cable to your village. Your telephone work OK and we have no obligation to make broadband work." He seemed genuinely surprised when I got Gigaclear in to replace them!
Palo Alto, that hot-bed of high-tech, still had analog switches in a couple of the local exchanges ... in 1998! They still worked for voice, and that was all the customers contracts paid for, so why change 'em? It wasn't until customers started getting upset that they couldn't connect to AOL at 28,800 "like my neighbor" that they switched to digital. The 326 exchange was the last to be changed over. If it wasn't for AOL, it would probably still be analog ... and some of THAT gear was installed in the late 1920s! At least most of the cable-plant was post-WWII ... (I was a some-time NOC-monkey, lurking under Bryant Street and on Fabian(RIP)).
This was/is also a known "fix" for running out of pairs - see Bloor of Andrews & Arnold ISP's explanation of it
Essentially they steal a working pair from a customer line, set up the new person's connection using it, then the person who has had their pair stolen has to submit a fault ticket and wait days/weeks for the engineer to attend, whereupon they swap it with a third person's pair, and repeat ad nauseam until a new multi-core line is laid or the heat death of the universe, whichever is sooner.
> Used to have similar finger pointing with our network team...
Me too. Our servers had multiple NICs for different traffic types (users, inter-server comms, backups). A regular issue was various switch ports getting "hung". The quick troubleshooting fix was to get the network team to reset the switch port. I say "quick" - the conversations invariably went:
Networks: Please reboot the server to check the problem isn't at your end.
Me: That will take at least 20 minutes and cause wider disruption; We've seen this issue multiple times, and the quick troubleshooting fix is for you to remote in and reset the switch port. I can tell you exactly which switch & port it is. It will take you 2 minutes. Or I can escalate to $TECHDIRECTOR.
Networks: <grumble , grumble> We've checked the port and there are no errors.
Me: Thank you. Did you reset it? Because the errors went away during our conversation.
Networks: No, no, we just checked it. The network is never the problem...
Me: [knowing they had reset the port] Cool, thanks. As you can't find any errors that indicate the server/NIC, we'll keep "reset the switch port" at the top of our troubleshooting checklist.
On one memorable occasion a switch for backups traffic hung completely so all servers using that switch were impacted - Guess what, Networks suggested rebooting all the servers (-‸ლ)
We've checked the port and there are no errors
BT used to be notorious for this. Due to location the connection to one of our plants was iffy to say the least, until we finally got a dedicated fibre line. Invariably after contacting BT it would be "No fault found" but the problem had mysteriously gone away.
Not always their fault. We had a leased line that would go down overnight (when not used) and the current hitting the (as we found out later) dry joint caused by BT checking the line was enough to get it working again. They finally left a 'tap' in place, and the next time it went down they were able to verify and agree that the line had an issue. It still took a while to locate and fix, but that was just the time it took to localise the dry joint.
I work on that side of things and believe it or not, exercising jacks is a legitimate fix that will lead to a no fault found conditon. Passing low voltage power through a connection made of different metals will eventually lead to galvanic corrosion, and this will eventually cause a fault. When the test lead is inserted, it scrapes the corrosion off and the line tests clean. When the test lead is removed, the connection is now good. After 2-3 faults cleared by jack exercising, the tech should be directed to inspect cable ends and change the jack. Sometimes a bad cable end will be found, sometimes it will be loose.
This post has been deleted by its author
"exercising jacks is a legitimate fix"
I take it that's a term of art for unplugging and replugging the connector? If so, that makes perfect sense. It's akin to reseating cards or socketed chips (the latter a rarity these days) to break the oxide layer that might have built up between the contacts.
So satisfying, pushing down on an IC[1] and getting a *crunch*. You know you've just solved a problem -- whether the current one that led you to try it or an incipient one that hadn't manifested yet. Or the corresponding disappointment when you push down and, as Adventure likes to put it, nothing happens.
[1] Only to be attempted with (typically DIP) chips in *non* ZIF sockets! Exerting force on a soldered part cannot help anything, and if you get a noise, you've just broken the card.
Working for a software company some 20yrs ago... An MS windows update broke a very important function of the older DOS based software still in use by venues around the country and Europe for accounting and ticketing purposes.
They found they could no longer generate the files of the finances from sales.
At the time we didn't know what the problem was... That took several days of dedicated research and time logging into all of these sites to see if we could figure out why some went down and others didn't.
After 4 days, I thought I'd narrowed it down to a specific update, so set up a clean test system in house using the data (back then they had to mail us backup tapes) from a few of these venues and discovered I was right.
There was no way to remove this update. So we went through all of those that had yet to apply it and blocked it.... then try to figure out a solution for the others that didn't involve wiping hundreds of systems and reinstalling. A colossal task for a 100 different sites, each one on a shoe string budget without their own IT depts.
There was no chance of a software fix for a such old software that had long been discontinued.
So the 'genius' idea I came up with, was a simple code alteration.
To generate the txt file for the reports of sales and accounts... I redirected the software to open 'notepad.exe'
It worked, it wasn't pretty... but I solved the problem and I had countless tiny venues such as community theatres, museums, council run spaces and cinemas thanking me for solving it.
My employers on the other hand... less than grateful for extending the life of deprecated software when the US based owners who bought them out a few years earlier were smelling blood and the loss of expensive software purchases and support packages.
I was made redundant less than 6 months later under dubious reasons when newer, far more useless, but more butt licking staff were kept on.
Would still do it again though... I worked in the arts with community theatres and music workshops. They're important parts of communities and costing them money they can't afford because some selfish evil mega corp in the US wanted to increase shareholder profits 0.00001% for a billion dollar enterprise by screwing over and probably causing those venues to fail.
Scarily similar - a customer of mine ran their financial env on an IBM power server. A fire broke out on the roof of the building, and power was cut abruptly. The UPS did not do its work, and the server went down hard. After we were allowed back in, the server would no longer boot up and complained about some hardware issue. Daily invoicing done by this environment would easily run in the 6 (Euro) digits a day, so I got an IBM tech in, and eventually he tracked it down to damaged RAM. Swapped out, and hey presto, all was OK again (yes, also replaced the UPS). Downtime was just over one day, cost was about EUR 5k for time and parts. Needless to say the customer was ecstatic.
My bosses were less than impressed, in their view I missed an opportunity to sell customer a EUR 50k replacement machine (which would have taken at least 2 weeks to get into production). 4 months later I was laid off - reason was literally "you are too kind to customers". Had to sign an NDA, so posted this anon.
My sister had the same problem in retail. As a young woman, she worked briefly in a chain clothing store in a mall.
Now, I can say from experience that she's really good at it. We went on a couple of clothes shopping expeditions back in the day, and she just had an eye for the perfect shirt out of what, to me, was a bewildering array of mostly not-a-chance choices -- in retrospect, the one that would push my style boundaries a little but not too much.
In her retail position, she'd be honest with someone if what they were trying on wasn't a good choice for them, or even if the store had nothing to suit their style, body type or whatever.
Management didn't see it that way. They wanted her to just make the bloody sale, and customer satisfaction be damned.
Suffice to say, she didn't last long there. (Good thing too; she's had a couple of more satisfying careers since then.)
Most recent event was a client site where they insisted on having an additional router acting as a boarder security device in front of their primary firewall. They had bought it, had it installed by a third party and promptly mirrored it to their secondary setup.
Queue drama. Interface on the device fails and their traffic is routed to the secondary link which is as it should be. However in some budget related manglement, they opted for a slower link. Now traffic for a complete organization is grinding to a halt.
Queue blame. As we provide some hosted services for them and the primary link, it is now our fault. Three hours of diagnosing the issue later we point to their device. Which they now claim we procured and are responsible for. Two quick emails later from years gone by outlining our position that this was a bad design choice on their part and that fact they would be doing this in their own (with an acknowledging email from their then CIO) did they relent. Now, here comes the extra kick to the soft spots. We ended up building a PowerPoint deck outlining everything for them so they could get there current execs to buy in on replacing the gear. They proceed to take 6 months to debate on and replace the device which means traffic has been running (dragging) over their redundant link and then having zero redundancy.
And then complained about quality control of service.
Sounds somewhat familiar, albeit from my side it's manufacturing rather than network.
We have a certain customer who insists on never doing preventative maintenance on our tools if they can avoid it, and so consumable parts which should be exchanged every 1-2 years end up in tools for much longer (current record is 12 years). Of course we (the vendor) highlight this as a risk for product scrap, but mysteriously when they do fail, it's always our fault, always comes as a complete surprise (and of course is usually on a Friday afternoon or during the weekend) and results in an escalation, screaming and "it's the end of the world as we know it" type scenarios.
Suffice it to say, for our good customers we have "run to PM", for others we have "run to fail" but for this particular customer we always refer to it as either "run to complain" or "run to escalation"...
Ooh, had a prime example of that at a previous employer.
We had a customer run by a rather obnoxious bloke who would sometimes handle discussions on upgrades/software by stating "I'm not paying that for anything" regardless of the merits of the ${something} being discussed. So they carried on using a hopelessly under-capable EPOS system, where we even had to resort to two separate instances when they expanded beyond the till count limit for one instance - and that caused some "fun".
But the network itself was equally shoestring - almost literally ! Cables just thrown anywhere, standard internal cable buried or run like a washing line outside, badly crimped ends, consumer grade switches stuffed wherever they felt like putting them. Needless to say, the network was "a bit flaky" - and this being a wildlife park with Lemurs running free, any exposed cable (c.f. the washing lines previously mentioned) was fair game to them. Time and time agin we'd point out the issues - in emails. But every time something failed this obnoxious manager would be ranting at us.
Eventually he left the business when it came down to a choice between staying and being closed down by the council for various H&S failures (his attitude to building maintenance was not much better), or leaving. After that, we got to tidy things up a bit.
Back in the 90s I worked for "Fy By Night Networking Equipment Inc" who made, among other things, a line of quasi-intelligent hubs. (This was the era of half duplex 10BaseT, note). There were a few things that didn't quite work right on it, it was one of those 'mostly OK' things that startups are well known for. This is where knowing your client is really useful because on of the customers who they sold these things to was a big Texas law firm that specialized in contract non-performance issues. The law office were actually very nice about their concerns, they didn't have to mention their reputation and capabilities one bit.
This is where a true class act shows itself. In what was billed as a showdown meeting, dreaded by all, our VP of sales not only bought us time to fix the PoS but sold them a bunch more of the same units. (...and, yes, you guessed it.....I was the worker bee that he was relying on to make it all come right)(.....no pressure......)
Long ago, this advice was given to people looking to interest investors:
"Write on one side of the paper only. Use simple words and short sentences. Pretend your audience are a collection of gin sodden taxi drivers."
As for the poor guy who fixed the problem here: I'm willing to bet he really annoyed some people and suffered some setbacks on account of his hubris.
Back in the early days of modems and TCP/IP, I worked for a VAR providing DEC Alphaservers to customers with our database application. We have one customer who squeezed every penny out of the contract possible. They sourced their own (used) server, declined any networking installation, and purchased an off brand modem and PPP kit for end user access. Customer testing is fine, but after moving into production, users sessions lock up. "The application is broken." Being a large vocal customer, this escalated within our company. The only component we provided was software on the Alphaserver, everything else was customer kit, yet our management accepted the burden of troubleshooting for the company. After almost three weeks of review, including dispatching to the customer's city, trapping traffic on the customer network showed their PPP kit was reducing the window size to each unit until it reached 0, in other words "don't send any more traffic until I'm ready." This was a known bug and their hardware vendor had firmware available to remediate. Did we get any thanks for this? What do you think.
I enjoy those moments when I get to gloat and on a few occasions I have hung up on a support call, and after making 100% certain that the call has ended, have shouted "I ****ing told you what was causing the problem right at the beginning, but you would not listen to me because you were certain it could not possibly be YOUR equipment at fault!". As others have said, I am quite happy to accept that the fault might be in my companies equipment and I fully expect that the customer also accept that it may also be in their equipment. Whenever I hear people telling me that it couldn't possibly be in their kit, I detect the stink of someone covering their own arse, or at best someone who can't be bothered to do their job.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2025
