Techie called out to customer ASAP, then: Do nothing Welcome once again to On Call, The Register's weekly reader-contributed tales of futile and furtive tech support chores. This week, meet a reader we'll Regomize as "Paul" who shared a story about the time his phone rang and he was asked if he could travel to provide tech support on a customer's site – which was about an hour …
Last working archive.org snapshot. So he posted 14 comics which I missed before hitting the nuke button. Overreacting seems to be the current USA state more than ever. Have fun browsing all snapshots!
As I already mentioned (in this news message comments): Archive.org.
And on top of that I script-downloaded all of them from 16-th April 1989 up to 13th of March 2023 until the typical USA overreaction took place on his side after those other's typical USA overreacting screaming at him.
Americans are really getting overly sensitive, and this is not new. Remember the Nippelgate of Janet Jackson? Would be news, but not even close to a scandal, in Germany - and most of Europe too. And it is getting worse.
Or are you referring with "decent stuff" to something else than his comics?
"Americans are really getting overly sensitive"
As an American, l say that's bollocks. You are comparing nations with profoundly different cultures. The Germany reference is almost comical. Having worked for a German company, with Germans everyday, I can say that their culture is profoundly different from ours.
Furthermore, there is nothing whatsoever, "overly sensitive" about condemning someone for making blatantly racist comments and then defending those comments. In the UK, someone hurling racist abuse at others can be criminally prosecuted. In the U.S., that's not possible because of our First Amendment. It's up to the people to sanction this kind of abusive speech.
Yes, there are an ever-growing number of people is the U.S. that spend their time to looking for reasons to be offended, and always finding one. However, most of us spend much more time rolling our eyes and groaning at their outrage-of-the-day than taking their nonsense seriously. Scott Adams' racist comments are not such nonsense.
> Having worked for a German company, with Germans everyday, I can say that their culture is profoundly different from ours.
There is a liiiiitttle problem with this: I am German. We Germans are direct. Only beaten by Danish, tough not much. Way to rude for american standards to be this direct, and all Germans should be taught this before going there. We Germans don't beat around the bush like "Oh, such a nice dress, awesome, looking great, but..." (I don't even know the american-don't-offend-way here), we go directly "Aw no, doesn't look good on you, we should try something else" skipping all that "cushion the blow" nonsense normal for Americans.
In our view this directness is preferred, 'cause we hate nothing more than wasting time and beating around the bush (the latter is even interpreted as offending deception).
When a friend likes a movie and I say "ou, that war horrible movie", and that is fine, it is about the movie. I USA you say at worst "It was Okay", trying not to offend, since "it was horrible" is seen as a great offense since they take it so personal.
A story from "LebenUSA" youtube channel: One coworker constantly failed to fill up excel sheets correctly. He went to her "Please be more careful, or there will be a problem.". He was called to HR, where he discovered that this sentence is "woahhhh don't say this", 'cause it usually interpreted this way: A James Bond Villain or Mafia Boss with a cat on his lap saying "we have a problem". She was really afraid of losing her job or worse.
I know, from the same channel, how this "never offend the slighted little bit" came to, and why it is indeed recommended to behave this way in USA.
If I am wrong, correct me!
Vielen danke. Thank you for demonstrating my point for me. You don't understand American culture and I don't entirely understand yours. Therefore, you asserting that Americans have become too sensitive is like me saying that Germans are rude. Which, by the way, I don't find them to be at all.
You mirror the behavior of those whom you claim to disdain. If you are looking for a single example of corporate HR idiocy to prove your point, you will always find one. In fact you will find plenty. That is to be expected since American HR departments are primarily interested in making sure that no employee has a valid reason to sue the company in court. Different cultures, different legal systems produce different outcomes. QED.
The early stuff was still good, back when Adams was still reasonably fresh out of the cubicle farm himself and had genuine insights into work culture. But that only lasted so long and then he just started cycling through the same set of material over and over, relying on tropes, because he was increasingly out of touch with the evolving workplace having removed himself from it.
I remember an American friend telling me that there was a pizza joint near him in the USA. They had quite high prices but unbelievably fast delivery times and a free pizza guarantee on that to match, He wondered how they did it as this was pre internet ordering and you called to place your order. He went down there one day to collect and saw the operation in action. They had the small selection of pizzas they sold ready to go into the oven, They didn’t do extra toppings or anything like that and when they got the call the correct pizza(s) went into one or more of several commercial conveyor pizza ovens and then to a motorcycle delivery rider waiting outside.
Even more impressive was their use of the English language. If you actually read the ‘guarantee’ especially the small print, it said they guaranteed to get a pizza to you not your pizza to you in the allocated time. They rarely had to give out free pizza apparently.
Last week we had the on-call where the staff got moaned at for not sending an engineer but fixing the problem,
This week someone sent to a site to meet the contract terms.
You do wonder about the people who draw up the contracts.
I used to work where we had a 4 hour fix contract with a supplier but if it was a hardware failure (i,e, someone let the magic smoke out) then that was fine but if it was a config issue then we didn’t need an engineer on site. Far more sensible
> I used to work where we had a 4 hour fix contract ...
Yeah, all of those are now 4 hour response, and best effort to fix.
We'll guarantee you get an initial response from T1 support via email within 4 hours. You may get parts by the end of day. Maybe tomorrow. Maybe nextweek depending where we have them.
I experienced exactly this sort of thing back in the day. Our side of the house ran Novell NetWare on Compaq Proliant hardware. NetWare never had any problems, and I made sure we ordered spare parts (hard drives and PSUs) whenever we ordered new servers. When I spec'd the servers, I included 4-hour response and wouldn't allow procurement to make any changes to what I ordered. So, we had the spares, and the experience, and were able to hot swap whenever a drive or PSU failed. If we couldn't handle it, we called the 800 number to get a tech out.
For some reason the suits - without talking to us of course - decided they needed to contract with local company to provide support. We had no input on the proposal or reviewing the bids. Being a government agency, low bid got the contract. The person assigned to respond was picked because they were closest. He was Microsoft certified but had no Novell training and knew nothing about hardware. The only thing he could do was show up - then either he would call the 800 number, or wait for someone else from his company to show up.
The company wanted to take all of our spares and move them to their location, which would have required someone going to that location to retrieve the part. I said HELL NO to that.
We tried it once - with predictable results. After that we just ignored it and fixed it ourselves, or called Compaq support ourselves. The suits did finally realize how stupid it was and at least didn't renew the contract, which caused the company president to start crying because they were a very small, new, company and really needed the contract to stay in business.
Our little software group were part of an oil-field supply company HQed in Aberdeen but we were in southern England
They hired a local (to Aberdeen) IT support outfit whose one visit to our site was to put an asset tag on everything, down to printer parallel port switches and then charge us £ /month per 'computer' - but would never send anyone to do anything.
caused the company president to start crying because they were a very small, new, company and really needed the contract to stay in business.
That's certainly a shame, but it's also a shame he either didn't realize they weren't providing a valuable service or take steps to remedy that... before the end of the contract. Either retrain the nearby employee, or hire a new one in the area with the skills actually needed.
We know this really. These contracts are set up by bean counters and legal beagles. They're all about costs, actions and penalties. The same applies to building maintenance, cleaning and so forth.
Actual connection to reality on the shop floor is superficial at best. Because no one actually asks the people who work there, or the hopefully soon to be tupe'd staff who were doing the job for the outgoing contractor what the actual job requirements need to be ( and this latter might not have been possible, because commercial confidentiality-built in ignorance is part of the process for contracted out functions- just one that the advocates for these systems never allude to).
Said managerial professionals are often quire proud of their ability to imagine, rather than possess, detailed understanding of what people actually do.
Quote
"Said managerial professionals are often quire proud of their ability to imagine, rather than possess, detailed understanding of what people actually do."
We had to go through a 'staff appraisal' review thing one place I attended.
That went quite well for a day as the 4 people selected to go first listed what they did verses the manglements idea of what they did.
Then the manglement decided to abandon the appraisals as it could lead to too many questions being asked of skills in use by the staff to do their jobs vs what they were paid to do...........
The rest of us were rather disappointed to miss out expanding manglements knowledge
Some years back my employers had the contract for a large institution with enough kit that there were half a dozen faults to resolve at any one time: it was taken as read that one member of the team would be on site pretty much at all times, giving more or less instant response and very little downtime, none of which was actually a measurable but was regularly cited by the client as a reason to renew. One day, though, the users decided they were unhappy with the kit itself, set up a little committee to Get Something Done and bent the head of acquisition's ear so hard that a tender was put out, resulting in what was effectively a beauty contest.
One of our competitors got the nod, at least in part due to the guaranteed 2-hour response to any callouts (which we were already delivering without it being in the SLA), and you can probably already guess how this worked out: a call would be placed, /someone/ would show up within the allotted 2 hours, sagely examine the device and determine it needed the ministrations of an actual technician. Who wouldn't be able to attend for another week or so. Cue more uproar from the users who'd cut their nose off to spite their face, but now had to live with it until the 3-year contract ran its course...
Or the "Hello Nurses". Blair's lot decreed that patients must be seen within some fixed time after arrival. So you'd sit and wait and after a while the Hello Nurse would come and take you elsewhere - to another queue where you waited ages to be seen by the actual doctor. Result: all patients were 'seen' within the time limit, and a (desperately needed elsewhere) nurse had to spend all day doing pointless makework.
We had a similar system introduced for out of hours doctor care.
This was just before NHS 0845 4647 around 2004/5.
There was a really simple and reliable database system, that was at the HQ, multiple bases and mobile terminals.
Worked really well , and was loved by the centre staff m
However it had poor report generation. So a new system was brought in.
Time to make a cup of tea.
So, the management decreed it was the best thing since the toast slice and wheel combined.
And it would be faster and better than it's replacement.
Ok. Great.
Yeah. For the first several weeks, we were using paper logs, faxing data all around and then had to rely on 1 landline emergency backup phone when it went belly up and somehow took out the phone system.
That was one Saturday morning, the busiest time for us, of course.
In the following weeks, many updates and tweeks were made, it became more stable, and it was all we could use.
So we made it work. Or so we thought.
One fairly hot bank holiday weekend, the server room AC failed, so everything went off or shutdown.
Even the phone system. But it was ok, that previous disaster had produced some backup mobiles. Yay !
Which then identified another problem.
Because our base terminals were all talking to the now switched off servers at HQ, no one could access the patient data, or clinical notes. So patients arrived, and the doctors were writing everything on paper, which as you can imagine doctor's writing to be, was a series of mostly wiggly lines that could only be read by Egyptologists.
So, a new backup paper system was used to log the patients at base and mobile appointments, with the corresponding case file number next to it.
Of course, when the system was back up, all those paper notes had to be inputted, so that took several days, luckily before the next weekend.
What had been learned helped, as we were hit with wide power blackouts a few months later, and we had learned enough so it wasnt a total disaster and people got seen.
If you'd like to see more examples of the above I'd recommend you take a look at The Tyranny of Metrics by Jerry Muller. El Reg readers will be familiar with many of the types of mad metrics he analyses and there are some examples showing how to do it properly too. I'm currently working in project management and leave my copy on the desk to annoy my EVM obsessed boss.
The whole target thing just does not work for things like health care, education and policing/emergency services.
Goodhart's law and all.
Plus the fact that you need to have slack in systems (perhaps education not so much because you usually know who is coming into the institution well in advance) to deal with transients and politicians always see this as 'waste'.
Going out to walk in the woods to ease my blood pressure (35+ years of this crap).
Even in education. It happens that a child goes rogue from time to time. But often staffing is too tight to manage the situation properly. Or there's only one paediatric first aid trained teaching assistant available at any one time and if she gets used to support an understaffed classroom the otherwise adequately staffed nursery has to close. Or there's a child protection issue and staff have to attend an emergency multi-agency meeting. Or a child in the school is meant to follow an individualised programme following a new diagnosis of one of the many things that schools are magically supposed to find time to manage.
I always love those kind of stories. Nincompoops who don't realize what they have, movers and shakers doing their majik to go someplace "better", and everyone ends up wailing and gnashing their teeth in frustration because the land wasn't actually greener on the other side.
Reminds me of a customer of mine who had a perfectly serviceable helpdesk application with monthly reporting and an Excel overview with pretty charts so manglement didn't have to think too much. Some bright spark decided that JIRA was better (probably because he'd read about it somewhere and wanted to look like he knew what he was talking about). Of course, the company went to JIRA.
Goodbye monthly reporting with pretty Excel charts and proper little targets, hello big JIRA mess without any reporting of any kind. It would seem one member of the Board was sufficiently miffed to actually state in a meeting that if they had known, they wouldn't have gone through with it.
Here's me thinking : yeah, well maybe you should have enquired about what you were leaving and what the replacement tool could do to fulfill the need.
But hey, what do I know ? I don't have all those fancy diplomas and golf club affiliations. I just work here, guv'.
Nincompoops who don't realise what they have, movers and shakers doing their majik to go some-place "better", and everyone ends up wailing and gnashing their teeth in frustration because the land wasn't actually greener on the other side.
Sounds just like Brexit
"Here's me thinking : yeah, well maybe you should have enquired about what you were leaving and what the replacement tool could do to fulfill the need."
Absolutely. Mid 80s, I was working for a small company, dealership for a particular niche-but-well-regarded accounts software. I'd not long helped install some nifty full tower 80386 machines at a client, a pair of which supported some 30+ serial terminals, when the client decided they'd had enough of us, and switched to the other local dealer. Fast forward a few years, I got a job at this other dealer too, and found myself installing a whole new system at this same client.
It seemed that in the intervening years, they'd got fed up of the new dealer, my new employer, and tendered out for a whole new system. I don't know who or what they chose, but it must have cost them a packet, as it involved PCs on each desk instead of dumb terminals, and thin ethernet everywhere. Then they tried to actually use it.. Apparently they had gone to the new company with a list of everything the existing software couldn't do, and were promised that the new software either could, or could be made to do all those things.
Of course, and I know you all spotted this coming, they forgot about listing all the things that the existing system /could/ do. So they start trying to use the new stuff, and it can't do most of what they were used to doing..
So they came back, cap in hand, to my new employer, apologising profusely, and we set everything back up as it had been, and were also able to address their list of Things It Doesn't Do. They didn't try to jump ship again, at least not in the time I was there. I don't know now - my now long since ex employer has switched over to selling SAP...
That's what I used to call it when I got paid to be on call or onsite when I knew that I was going to be paid for sitting around and doing nothing. The best one was a bank holiday weekend where I spent 24 hours in total on triple time watching someone else loading tapes in and out of a backup drive to transfer a system to some new servers, and they were worried that it might not start up again afterwards (it did). Another time was when I signed up to be on call to handle any problems between Christmas and New Year when they had stupidly planned to go live on a new system. I'd guessed that they would have second thoughts about doing this because everyone else would be on holiday, and in the end they panicked and delayed the date, leaving me getting paid to have a nicely extravagant break myself.
"between Christmas and New Year when they had stupidly planned to go live on a new system"
Stupidly? If there's downtime to make the cut-over it's a very sensible time. I've told here before of the client who, at the last minute, postponed that. It was December, 1999 and the new system was the Y2K-compatible version...
"Stupidly? If there's downtime to make the cut-over it's a very sensible time."
Probably not. To do a proper switch, you need plenty of staff there who can make the switch, quickly back out the switch if it goes wrong, and detect problems that would either be big enough to require rolling back or making emergency changes to the system. That's a lot of staff. You might also need the people who managed the switch if there's any part of this that would need approvals. Making all of those people work over a holiday wouldn't be popular when you could try using a weekend for it instead. This is all if all the people needed for a switch are in your company. If there's a chance you need to call someone else for support, a holiday is not a great time to rely on that.
There is probably a lot less downtime from the perspective of how many users are using the system than there is from the perspective of how many people are there actively working on the system. It's not guaranteed that failing the system during a holiday will be safer than failing it some other time, and if the fewer staff means that it takes longer to fix than it would at another time, it could make it worse. For operational reasons, there's a reason not to do it, and for a not having your technical staff angry that they had to work on a holiday and the management who decided this do not reason, there's another.
"It's not guaranteed that failing the system during a holiday will be safer than failing it some other time,"
It's more about who's watching. If the people that can cause the contract to be cancelled are all out on holiday, that's a good point to do the risky stuff. Paying people triple time to be on standby might be cheaper than every minute of downtime on a system.
"If the people that can cause the contract to be cancelled are all out on holiday, that's a good point to do the risky stuff."
That depends a lot on what those people are like. Are they the kind of people who will freak out if they see a problem, but if the problem's fixed by the time they show up, all is good? Then you would be right. Are they the kind of people who want to see what is happening and will be annoyed if you change things when they're not there to monitor? Are they the kind of people who value you more if you're present and look busy, so having people there to do the change while they can see is likely to give them confidence that you're productive and good at the job? In either of those cases, it's better to wait until they're present.
I'm referring more to the damage that's really done by a system being down. Managers yelling at you isn't always fatal. Customers furious about an unplanned system malfunction can be worse. If your goal is to optimize for causing the least damage, there are reasons not to do that on a holiday as I described in my last comment.
Well, do your Y2K update -before- December - this is a tip for future reference. ;-) However... if you have a robust backout plan for your upgrade, to go back to the old system... then, a long period while the business doesn't have to run, gives you time to do the upgrade, find out that it doesn't work, find out if you can get it to work after all, then finally give up after all and go back to the old setup.
My other Y2K tip is to check that your software gets February 29th 2000 correct - some of our out of support stuff was patched for 01/01/2000 but not for 01/03/2000 (British). And it still isn't, I think.
"The number of times I've had to just sit for hours and wait while a system is remaking..."
A drummer friend of mine had lots of time on tours in the van between gigs when it wasn't his turn to drive. He taught himself programming and used the travel time to write some really good apps for drummers (Polynome). Neil Peart was a voracious reader. I listen to audiobooks more than I sit down with a book these days as it lets me work in the garden and do chores with a nice distraction. I learned in college that if I could listen to the lectures (which I could copy on cassettes) over and over, it would hammer the material into my brain and I didn't have to try and make notes in class which I sucked at doing very well. I have loads of learning material since many colleges release lectures as podcasts for free. While the college may charge stacks of money to attend, what you are paying for is the piece of paper at the end, not so much the knowledge. I'm back into nuclear physics from courses at Stanford and MIT. I looked at taking a degree but both of those places are crazy money and there would be no return on the investment. Just learning the material is enough reward for me at my time of life.
Late '90's and I worked for a distributor of optical storage subsystems. Got a call from an account manager working for a systems integrator. They'd sold one of our largest library units to a major bank and could we spare a specialist to attend site at short notice? Their client had told them they needed to move this bit of kit within its server room and the SI didn't feel comfortable without backup. A suitable fee was negotiated for the out of hours operation and I headed into central London, armed with the appropriate spare SCSI cables, terminators and anything else I could think of that someone could lose/damage in transit.
Having got there and signed-in to the fairly secure area, I watched, bemused, as the units castor wheels were each unlocked, the machine rolled backwards a couple of feet so an item that had rolled under the box could be recovered. The jukebox was then ceremoniously restored to its original location and its wheels locked. A few system tests were run and then we all went out for a beer. Probably the easiest consultancy fee we ever earned, just a shame the SI didn't clarify the extent of the 'move' first!
The extent of the move didn't really matter. The systems integrator wanted/needed to have a scapegoat if anyone went wrong. And give the bank a bit of performance theatre to show them they'd picked the most responsible and reassuringly expensive systems integrator.
I know of a very dim sysadmin who deliberately copied a corrupted database over the live, production one. He avoided instant dismissal by using the Nuremberg defence, claiming he was only obeying orders from IBM tech support.
There might not be another side. Just hook it out.
Take an old wire coathanger, use pliers to roughly straighten out all but the hook. You now have a few feet of stiff wire with a hook on the end. Perfect for ferreting round in narrow gaps and hoiking out whatever rolled/fell there, or grabbing things which are just out of reach.
I keep an old length of 15mm PVC piping in the back of the car for just such problems. It's got just enough flexibility in it to bend through a removed floorboard into the floor void, and is long enough with the addition of the Mark One human arm to get across about 16ft of floor.
> Perhaps slide a small pole underneath and push whatever it was out the other side?
Reminds me of what was, probably, the first Personal Electronic Transactor 2000 in Australia. Despite our electronics engineer having a custom-made transformer produced to cope with the different voltage and frequency from the (United) State(s) of origin, it was never reliable. It was programmed in some dialect of BASIC, with an audio cassette for programme storage, but we soon reconciled to saving our code after pretty much every LOC change.
Came the chance to prop it open on a lab bench, with several digital probes and a couple of channels of CRO monitoring as well, it steadfastly refused to die. However, after a couple of days it finally stopped. And the cause? A signal on the 68000 microprocessor was grounded, halting the chip (one is reminded of the "DTACK Grounded" hacks on the same chip). Being a little bit young and green, I was all for desoldering the chip to look for the fault. However, a wiser technician decided to probe under the chip with a piece of plastic fibre.
Lo and behold, what emerged was a clipped off end of a pin from a transistor or voltage-regulator (we never established which) which had flown under the soldered-in CPU and came to rest just where, vibrations permitting, it was allowed it to ground the signal and halt the processor. The machine was much more reliable after that (apart from the fact that its IEE488 interface could not cope with delayed responses of more than a second, due to its clock counter overflowing...).
Your story reminds my of the legendary "gravity fed keyboard buffer" from my days as a PFY.
I was working in IT support for an electronics manufacturer, and was summoned to the factory floor to look at a PC with a malfunctioning keyboard. The PC was a white box 486 or early Pentium in a small tower form factor, with the cover removed for easy access to the expansion cards. I forget if the keyboard was PS/2 or AT, but replacing the keyboard didn't fix the issue.
I brought the PC back to my workbench and laid it down flat (recall that in this era, about 50% of PCs sat upright on/under a desk, and the other 50% sat flat with the monitor propped on top). It booted to DOS just fine. I found that if I typed something on the keyboard, nothing happened, but if I then tipped the enclosue up into the "tower" position, all the characters I typed would appear at the C:\ prompt.
I joked that the "keyboard pump had clearly gone bad" and that it lacked the strength to pump data up from the keyboard jack up to the video card. Rotating to tower position put the keyboard port above the video card, so the data could flow down under the effects of gravity.
The actual root cause was an approx 1/2 long (about 10mm for the rest of the world) lead that settled next to an IC on the motherboard. When the motherboard was flat, the lead shorted out a few pins. When the motherboard was vertical, the lead rolled aside and sat against the (insulated) body of the IC.
That plant floor had several areas where workers hand-trimmed compoment leads with side cutters, so the shrapnel was to be expected.
(Loads of setup and background for this story, I probably should have submitted it as its own on call entry)
"A signal on the 68000 microprocessor"
An excellent bit of fault finding. Especially under a CPU that didn't make into personal computers till a few years after :-)
The PET had an 8-bit 6502, not a 16-bit 68000 that later turned up in Amigas, STs and Megadrives :-)
I've been the victim of an even shadier contract. The equipment in question was key to the output of the department but so hideously expensive that a redundant machine wasn't a possibility. As I understand, the new intended maintenance contract (about 4x the price) was for resolution within 1 working day, instead of a week.
Unfortunately, the language wasn't reviewed and what was bought was a contract for 'commencement of repairs' within 1 working day. This meant that, if new parts were required, the engineer would dutifully arrive on his way home, pull the cover off and then leave it while his manager found the cheapest part supplier. This often resulted in downtimes greater than under the previous contract. The resolution was that some heads rolled, the money was found for a redundant machine and the old contract was reinstated.
Appeasement Engineer
"Because he's NEW and ALONE, he's what you call an appeasement engineer, the new guy they send so they respond within the 4 hour guaranteed response period. (Things are getting better and better) Your average appeasement engineer is about as clued-up on computers as the average computer "hacker" is about B.O, and their main job is to make sure the power plug is in and switched on, then call back to the office for "PARTS". The really keen ones will sometimes even take a cover off the equipment and pretend that they see this stuff all the time. I wonder what sort today's is...
Short version.
Printer down - Client demands action.
Two parts required for repair & within two business days (Well its now Monday as there was a weekend in the middle) I receive one of them - Client screaming.
Wednesday still waiting on part two, I am told to attend site in rural Alberta with part one (Do not fit part one, because part one will let out the magic smoke if fitted without part two) & do a pretense of basic fault finding because client apocalyptic. Thanks to a RTA, I take a longer more circular route to site, rather than divert down a stony range road in nice new personally owned vehicle & arrive at site mid afternoon.
Discover fault is in the wall power socket, verify by running, a extension cable into a nearby office not the printer as client does not believe evidence of DVM.
Leave site & start laughing the minute I get in the vehicle.
Rule number zero of troubleshooting (and design):
User behavior and ability to miss details are more unreliable than mechanical parts and, therefore, the first place you should look if there's a problem.
Before checking whether the switch isn't engaging right or the cable is not working, check that the switch has been set to on and the cable is connected to the right things and those things appear to be working. It's related to the first step of troubleshooting: verify that there is a problem and that you understand what the problem is, not what the report said.
This post has been deleted by its author
We had one recently.
I usually connect to the web server with Google Chrome. One day, the error message is that HTTPS failed because the date and time on my PC are wrong. I check. They aren't.
I try Microsoft Edge. It says that the web server's certificate has expired.
It turns out that the web server's certificate has expired.
Is this a thing that Google Chrome frequently does? Tell you that you're in the wrong time dimension? It's new to me...
It's a historical thing.
Back in the days when people in charge of IT actually had a clue, it was reasonable to assume that any problem with a certificate that appeared out-of-date was more likely to be on the client end (time and date incorrectly set) than the server end (certificate actually allowed to expire without anybody doing about it).
In the mid-1980s, my employer sent me to Austin, Texas. Our installation there was waiting on something--I don't remember what, but it was not something I had with me or could supply. I had a very pleasant day and a half in Austin, I got along well with the customer's staff, who understood all this as well as I did, and as I recall did nothing especially useful. I think that the salesman was happy to have me there, to show that the company cared.
Am I the only person who heard of this as "green rabbits"? It was described to me thusly: They're really really fast, but they're so green they can't do anything. Send someone quickly to the site, but they often don't even pretend to do anything except declare the SLA for a tech arriving onsite had been met.
Has taking no action whatsoever ever been the most sensible course of action in your job?
Many times, in a former life when I was still with one of the then major banks. My manager would approach me with: "can you do this or that?" To which my response was: sure! And sure enough I didn't let "this or that" bother me any longer. Most often, that was it. Done. But every now and then he would come back a few weeks later and enquire where I was with "this or that". To which I responded: not yet but I had planned to do so next week. And depending on his reaction I would sense whether I can forget about "this or that" for good or I actually should do something.
The background was that someone further up in the food chain thought of "this or that" but didn't deem it as important as to deal with it themselves. So they delegated it downstream. And all of the intermediaries couldn't be bothered themselves either and passed it on until it ended up on my lowly desk. And if it was that unimportant to end up on my desk, it really was not worth bothering at all, in most of the cases.
I remember a case in a hospital where the IT department were called one morning by someone who didn't even work there.
Medical consultant has IT problem. Tells his secretary.
Secretary tells another secretary.
That evening, 'other' secretary tells boyfriend.
Boyfriend tells mate while at pub.
Next day, said mate calls hospital IT.
IT dept backtracks chain, identifies consultant and sorts problem!
I like the title of 'Appeasement Engineer'. Very apposite.
One of my previous employers liked to use acronyms in their contracts.
And not include a glossary.
And Lo! and Behold!, the mighty acronym MTTR came to be.
And it came to pass that an Important Customer decided it meant Minimum Time To Repair, whereas our Service Management Team (a group of stress related diseases in mostly human form) were required to stick to the line that it meant Maximum Time To Respond*. After one of Important Customer's sites was disconnected from the network for several days, Deep and Earnest Discussions ensued, involving not only the Account Director, but members of the division's Senior Management Team, winkled out from whatever Golf/Country Club they were currently hiding in.
And thenceforth, MTTR was deemed to be Mean Time to Repair, and everybody went back to more pleasurable pursuits, leaving it to the Service Management Team to work out how to generate a mean, what counted as a measurable outage e.g. if a site had two connections, and one went down, but the service remained up, did that count? What level of packet loss counted as a service outage? Does repair mean temporary fix/workaround until a planned outage could effect a permanent repair, or not.
And the Account Director, in one of his unguarded moments when 'tired and emotional', never said that the ambiguity was deliberate to leave wriggle-room after the contract was signed.
NN
*A 'Response' could simply be an acknowledgement email.
This is not uncommon. In the early days of MSPs in London, typical "responses" were considered to be "logging the ticket" and "starting the ticket". We had multiple tiers of "response" that you could choose from when logging a ticket that had various different prices attached...basically they got higher the sooner you wanted an engineer on the job. Starting at 2 hour response going up to 10 days.
Of course, the managers saw this as a way to crowbar more money out the customers, but a couple of engineers (including myself) saw this as a way to make some serious side cash. See there was an interesting loop hole created by "over managing" the engineers...we had a team leader that would assign tickets to you based on your "customer affinity" basically, as an engineer you became generally attached to clients to keep some level of consistency going with the clients...to build a relationship. We could also raise our own tickets.
The great thing about this is that the greed of the directors leads to some pretty interesting opportunities.
Anyway, what I used to do for clients to help them save money, get the same engineer every time and to increase the wads in my own pockets was to log a 10 day ticket for the client, but go to the client site after my shift ended (most of our clients were Central London based)...so pretty much all of my clients had access to "same day" support for the price of a 5 day ticket. They still had to log a 10 day ticket on the company system, but the rate for this was so cheap, it was seen as a "booking fee"...it was like £20 an hour or something, and you'd log less than an hour against the ticket (when the time came to activate it and "work" on it). The same day call out fee was around £100-£150 an hour (yeah, I wasn't being paid that, I was earning less than £11k a year at that point, which was low even for the time) and the average ticket usually came out at 2-3 hours...so instead of paying the business up to £450 for a same day call, the client would pay the £20 "booking fee", and I charged a flat £75, no matter how long a ticket took...because I was charging a flat rate, I was motivated to get the problem solved ASAP...so it was usually less than 1 hour, and I could get in 3-4 site visits after my shift.
I was earning 4 times more on the side from the clients at the business than the business was paying me. They almost twigged when my tax code changed, because I was putting everything through self assessment to pay the tax. I explained it away as having to "do a second job to make ends meet" that was "unrelated to the MSP business".
At one point I was earning more money than the directors were! Eventually I told the directors I wanted to go part time for a while because I was "burned out"...but what I was actually doing was onboarding clients to my own business whose contracts had ended and had drifted 6 months.
Rumors say that Elon Musk hired a bunch of people to sit around on minimum wage at the NY "gigafactory" due to the possible fines for not having enough people on payroll. The state built and equipped the plant to the tune of around $1bn for a solar panel company that got bought out by solar city which in turn got bailed out by Tesla making Tesla the holder of the hot potato. The facility is currently valued at a few million with all of the installed manufacturing equipment for making solar panels having been scrapped with very little use. The contract stated that the company had to provide a certain number of jobs and I expect the state figured that they'd be well paying solar industry manufacturing jobs and not just minimum wage. If you like to sit around a read all day and are in the area, you might want to see if there are any openings.
It almost seems to be a given that organisations awarding big contracts have fewer skilled staff writing the contract and checking its details than do the companies who benefit. I assume that the concept of "cost savings" in outsourcing and- in this case- being economical while apparently from what's written here subsidising production and invstment are seen as more important than due diligence. iow "False economy"
"The Chinese royally played the hapless idiot Gov. Scott Walker."
The large corporations can run contract circles around any band of lifers feeding from the public trough. This is one of the reasons I don't like to see any government agency paying companies to locate to a particular area. The company's legal department will write the contracts with so many ways for them to get out of it that they can negotiate a few of them away and still be assured it's all in their favor. Those companies know where it makes the most sense to locate their next facility and I'll bet they tease places that don't have any chance of being unknowing shills to get the preferred locations to offer the most money possible. It's not like some firm such as Foxconn needs the money if it makes sense to build a new facility somewhere.
This is standard practice and not just in the IT industry.
Take, for example, a supplier that claimed to employ "hundreds of qualified engineers available anywhere in the UK hour's notice 24/7*"
The asterisk lead you to a footnote that the Scottish Highlands and any islands were not included in the one hour premise. Oh and that traffic or weather conditions might just impact arrival times.
Being as we were close to some major urban centres we assumed we'd get a "qualified engineer" well inside the hour to fix our issue. And blow me down, the engineer was on site well within the hour. I noticed immediately that his van and his polo shirt proclaimed that he worked for a totally different company. When he got into the plant room he looked somewhat startled by the equipment that confronted him. He got out his mobile to call in to his control only to find there was no signal. We were in a basement plant room, of course there was no signal. Did we have a landline phone in there? Well we had some PSTN lines for control so I dug out my butt handset and handed it over. He called his control and was talked through working on the kit which took ages because it appeared he'd never seen that kit before.
Eventually with the help of his control he isolated the fault to a failed part. Which would have to be ordered. Unsurprisingly since the engineer had no idea about this kit he didn't carry spares. And it would take a couple of hours to arrive. At this time the engineer left telling me another engineer would arrive to fit the replacement part.
The engineer duly arrived about an hour later, so about three hours from the original call out. This engineer was driving a van that proclaimed her to be employed by our supplier. She had loads of spares in the back.
Once the plant was fixed she fessed that they used several different contractors around the country in order to make it to site within the one hour SLA in a "remote hands" capacity. and "qualified" meant that these subcontractors were usually certified sparks and were insured to work on the kit.
It's a bit of a dodgy way of meeting the SLA, but had we known that we could have saved more time. I'm a qualified sparky so could have done their hands for them.
"I'm a qualified sparky so could have done their hands for them".
True, but do you want to become the punching bag in the middle if a job goes wrong or their staff are inept at giving instructions? Having someone involved in the job isn't under the service providers umbrella is a massive opportunity to escape blame.
Sometimes saving time can ultimately waste more time than you thought you would save.
I don't think using third party contractors is a dodgy way to meet an SLA...it's actually the responsible thing to do. It's also as old as the hills. Your local post office is probably owned and run by a third party. Your local McDonalds, Subway, Starbucks, Pub etc definitely is. As long as there is some sort of training, quality control and regular checks in place...it's a pretty reasonable thing to do.
In a previous job, we would receive complaints for non-attendance even if we fixed the problem over the phone with them.
In the end, some of those customers were 'let go' I was back in the area recently and quite a few of those businesses seemed not to have made it through Covid.
I worked for Oracle in the 90's as a DBA consultant. While waiting for all of my security paperwork to get processed and get a long term assignment, I was twiddling my thumbs in the corporate office. At the end of the day, almost Beer O'Clock, my project manager comes in and says "I need you to install a database in Norfolk tomorrow morning". Given that Norfolk was 250 miles away, I would have to leave that evening, and Beer O'Clock would have to wait until I got to my hotel.
Anyway when I arrived onsite the next morning, the Sun server was still in the box. It took them all day to set it up and install Solaris, while I twiddled my thumbs. The actual database installation took about 30 minutes, after which I drove home. So including the travel, 2 days of billable time for 30 minutes of actual work.
"Anyway when I arrived onsite the next morning, the Sun server was still in the box. It took them all day to set it up and install Solaris, while I twiddled my thumbs. The actual database installation took about 30 minutes, after which I drove home. So including the travel, 2 days of billable time for 30 minutes of actual work."
Did anybody else above you know the DB install would only take 30 minutes, with careful and thorough testing? It sounds like you should have been able to get at least another 1/2 day out of it replete with some very nice meals.
9 hours on a conference bridge with a customer, service manager, customer's equipment vendor, and a tech on site. I added nothing to the process, but customer insisted my tech support role was required.
Lots of double time while I watched the Six Nations rugby and drank beer in my comfy armchair.
(from the article)
"Whether the customer ever realized it was being fooled is lost to history."
Was the customer being fooled, though? I thought it was the customer initiating the callout, because:
"The cost of having Paul travel to the site and do nothing was tiny, when compared to the penalties the customer was owed under the contract if nobody appeared on time."
I assumed the customer was willing to take the risk of paying the tiny fee for a chance at the large penalties - a reasonably rational (especially since it appears there was a good chance Paul might not have made it in time) if less than ethical course of action. I don't understand how the story makes sense any other way. So how is the customer being fooled?
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
