Can your rival fix it as fast? turns out to be ten-million-dollar question for plucky support guy Have you got that Friday feeling? Well, you should, because it's just hours away from the weekend and we've got another great Reg reader story in this week's On Call. This time our tale of tech support triumph comes from reader "Ben", who in 1979 worked for a plug-compatible manufacturer that was in competition with one of the …
"you need to keep really, really close tabs - and lots of comments - on the source otherwise it'll trample all over you."
I remember taking a programming course years ago (Pascal). The professor was constantly reminding us to document out code. At the time my documentation was pretty brief as it was obvious what everything did.
Yeah....
That was great when you were working on it and everything was fresh in your mind. I remember looking at my old printouts the following year when cleaning up and had trouble making sense of anything without analyzing all of the code. Lesson learned.
Remember....you aren't documentation the code for yourself, you are documenting it for "future you" and those who follow.
Particularly: I remember looking at my old printouts the following year when cleaning up and had trouble making sense of anything without analyzing all of the code. Lesson learned.
Obligatory Codeless Code: Case 116, Trust No One. Covers this situation startlingly well.
http://thecodelesscode.com/case/116
In the 1980's our small team had the prestigious task to write (Pascal)software for an exhibition at the Rijksmuseum in Amsterdam. The idea was that visitors could point at a touch(!) screen and a corresponding incandescent light would lit a 3D scale model.
It was a chaos and no one could find the bug. Finally I discovered that an array was declared with length 5, but documented as length 4 AFAIR. Anyway the crux was that somewhere in the code was a reference to the last but one element of that array. That cured me from excessive documentation.
What I find tragical is that in stories like this, the expertise and work ethic of an arbitrary employee get again and again confused with a level of customer service a whole firm is believed to reliably deliver. Sure, "Ben" is a credit to his employer, but the thing is he's not bolted immovably into that job - half a year later, there might be someone else in his place, and deliver what his competition did, and "Ben" might be working over there with the same diligence - sticking with the outfit that delivered the better service in the past may make much sense, but offers basically no guarantee of a repeat performance, yet customers invariably seem to believe the opposite...
"the expertise and work ethic of an arbitrary employee get again and again confused with a level of customer service a whole firm is believed to reliably deliver."
OTOH the customer knows that (a) supplier has a Ben and (b) that may be typical of their expertise and (c) competitor doesn't have Ben and (d) their lack of expertise was demonstrated by their insistence that the fix wouldn't work. (d) might be the real killer there.
Without knowing how long Ben had been there, the only assumption they can make is that either the company has a great hiring practice, or a great work environment and are therefore able to recruit or retain someone of that calibre.
Given the evidence at hand, that's all they could conclude. Given the choice of a company like that, or one that has been shown to have incompetent employees, what route would you take?
Back when SGI still existed we gave them, IBM, HP, and Dell a benchmark of our own to test on their proposed multi-million dollar computer. HP and Dell didn't even bother to submit a figure but just pulled out of the RFP.
IBM sent back a figure with a dollar figure they'd cut off the price.
SGI returned a giant analysis of our code with multiple suggestions for optimisation. They won my vote.
Long ago, we were running a mix of VAX/VMS & RSX11 over DECNet, but the PDP11 kit was using 3rd party DECNet software. So one day we added some new nodes & started getting communication problems on the 11s. I passed the problem to the software vendors but they didn't seem to have a clue so I started looking myself.
The DECNet protocol was a bit of an eye opener - my first introduction to networking protocols - but I worked out what the various modules in this software did. I managed to get hold of a PDP11 disassembler & started looking at the module code. This was in the days before hex so the constants all printed out in octal. And guess what stood out - an octal 256 constant! WTF? So I changed it to octal 400 as it should have been, reassembled the code and it worked perfectly.
The constant dimensioned the node table (must have been DECNet phase III) so it overflowed when we added nodes above 173 (octal 255). I also discovered how the software bypassed the licence check for testing purposes, which was handy for quickly adding new nodes before we got a new licensed copy...
A certain large arabian country, yes that place, had installed a massive telephone system in the early 1980's for BILLIONS of $$$ (under their third & fourth, five year plans). It was installed by A random Telephone & Telegraph of a certain large world country, yes that place.
All was fine for about 6-months, then the ARTT/WeCo as we'll call them quietly asked for another billion for maintenance. This incurred the wrath of the king, who asked why when we've just given you a lot of oil money, do you then ask for a recurring payment in case it breaks? Rather than head-chopping, he just kicked them out, ALL of the operations & maintenance techs & engineers.
They helpfully took ALL of the spare parts and circuit diagrams and test protocols and anything that wasn't bolted down, back home to WeCo land, smirking.
As a few of the 10800 channel long distance Philips cable systems promptly stopped working, eaten by camels, bad local drivers, heat etc; the king quickly gave a few million $ to a nice Italian company and asked them to do the O&M. They did quite a good job, but it was a small problem that there were no spares or diagrams or anything to help. WeCo continued smirking.
I naively answered a job advert on page 25 of The Sun. "engineer wanted" £25K, as my training officer at Marconi had just said "all apprentoids must get a new job as there's no future for you here", and I was getting £7800 a year. Simples! It was an even better deal than I expected, representing a sextupling of my take-home pay, with nothing to spend it on.
I flew to the capital city, nice and warm at that time of year, and the O&M boss showed me my office. It had a warehouse attached, filled with broken bits of electronics, analogue FDM MUX, for those who remember that era!, channel banks, customer cards, PBX, coax repeaters, microwave line of sight radio terminals and everything else, under the sun. (seems the WeCo stuff didnt work that well at +40C ambient)
A sparsely populated country, about the size of western europe, and all the broken bits wre given to me, with no documentation. So I sat, thought, and traced & drew the circuits, redesigned the test fixtures, started digital cannibalism to get three cards out of four working, trained repair & cal technicians, and enormously helped the nice italian company to complete their O&M for millions, that the smirkers wanted billions for. I had lots of help from the six brits/canadians marooned there with me. It was a fun job.
The king still threw us out a couple of years later when a nice indian company explained globalisation to him and offered to repair things for free, or at least using near slave labour. I think WeCo had also stopped smirking by then (lawsuit), and delivered back all of the spares and documents, about five years too late and just as synchronous hierarchy was starting to take off.
bottles of wine were certainly not overtly available - but the local corner shops stocked grape-juice, yeast and sugar, in large quantities, for some reason, and I kept the king's phone going,
Ah, a location I recognise (for similar reasons) along with the sign-off paragraph. I was well regarded for my twice-processed grape juice and heard many a story of people flown home in a hurry after having being seen in public after a few glasses of modified 'near-beer'.
Could never get the hang of the modified Coffee machine to produce Sidiqi(?)
Ah yes. Remember it like it was yesterday. A brew could be fermented and drinkable in about a week because of the temperature.
Me? I stayed with the Pepsi. Didn't go there for the booze. Met a number a alcoholics who went there to get away from the family so they could drink away without getting nagged at!
I saved a contract for h.f. mobile transmitters while failed in operation.
The whole device was a clamshell pair with one half the 'exciter' and the other half the power output stage. The transmitter adjusted itself for the antenna impedance using a variable motorised output coil. But the microprocessor flagged errors about 50 per cent of microphone press to-talk.
Being a clamshell, there was no way to investigate until a cable with complex 'D' type connectors was assembled.
Then I could trace it with a 'scope and found that the mic PTT switch should unbalance the single side band that sent a short burst of RF to a diode and Capacitor/resistor resulting in a pulse to the processor to give it the go-ahead.
But sometimes the RF burst was too short and the resultant pulse too short to trigger the processor. Once the company head office in Texas gave us the go-ahead to fix it, we just increased the capacitor value.
We asked head-office if these radios were used elsewhere. Yes they said , the type of exciters were used and the type of PA stages were used elsewhere. We asked again and they admitted that our configuration was the first using those two units.
Didn't make any money out of this but the boss was pleased!
Hi, David Shaw,
The Title says it all succinctly and APTly.
Are All Arabian Countries, and All Other Countries too, Now Similarly being Denied Free Use of the Most Modern of IntelAIgent Communications in the Great Abiding Conundrum which Delivers Simple Words to Follow and Create Complex Worlds with. ........ which They Mentor and Monitor with Supplies of Future ProVision.... for Experimental Experiential Use/Dummy AI Beta Crash Testing.
And a SMARTR AIRealisation Facility for Greater IntelAIgent Gamers Play.
In AId Simulations of Future Live Operational Virtual Environments for Media Presentation to Earth/Ground 0Zero.
Or would you not be thinking, with any Free Choice Denying ITs Present Existence Requiring Further Gospel Truth Proving with Almighty Proof?
In AI Games All Can Play to Flourish and Traverse into One's Very Own LOVE Futures ... for there be Myriad Courses to Follow Shining Immaculate Lights upon the Ways Ahead.
Well one would sure like to fix such problems amfM .... its almost like the Fox being blamed for injuries sustained to the hounds, and certainly not an abvious solution to said when in all virtuous virtual realities it could have been easily prevented many moons ago.
However no point dwelling on such, and as you know only too well not through lack of trying and complying, perhaps by limiting resources therein lies the problem, and that itself AI’nt through FAir play as has been reasonably observed.
perhaps by limiting resources therein lies the problem, .... Cliff Thorburn
When resources are unlimited, CT, the second and third party problem to contain and first party opportunity to exploit is to throw the most novel of disruptive technologies and methodologies out there even as if pearls before swine ....... for prime uptake by the makers and clients of high endian bespoke jewelled operations ..... and not to waste time in these new fangled quantum entangling spaces with that which knows not what is preying on them and taking full advantage of catastrophic systemic vulnerabilities .... and in full flash fast cash crash mode, a fiat paper mega nightmare experience without traditional end.
Some can see and would recognise that as a Virtual Blunderbuss Type Mission which Supplies SMARTR IntelAIgent Services Freely to Friendly Servers with both Hostage and Hostile Assets to Satisfy ...... Remotely Command and Virtually Control Evermore Seamlessly with Media and Advanced IntelAIgent IT Programming Projects.
Dont let it be said amfM that it was never offered to meet with the PM or others, and as is recorded quite frequently and historically such olive branch could have and would have changed and altered the direction of travel most adequately, for it would have been an honest honour to do so, LIve Operational Virtual Environments Actually?.., and perhaps a chance for mutually beneficial outcomes for all ....
Dont let it be said amfM that it was never offered to meet with the PM or others, and as is recorded quite frequently and historically such olive branch could have and would have changed and altered the direction of travel most adequately, for it would have been an honest honour to do so, LIve Operational Virtual Environments Actually?.., and perhaps a chance for mutually beneficial outcomes for all .... ... Cliff Thorburn
Methinks that the Most Right Royal of Occasions, CT, all outcomes beneficial and mutually satisfying.
And Such Releases Almighty Energies even now Not Fully Fathomed in AIMaster Pilot ProgramMING Circles ........ Casting Almighty Round Tables for Future Terra Phormers to Beta AI Crash Test Virtual Reality Programs and Projects in XSStream Experimental Live Operational Virtual Environments.
Ich Dien Natuurlijk.
Things are Getting LOVEly and Lively, CT ....... with El Reg at the XRoads of Quantum Leap and Phenomenal Success, Phantom Fame and Fabulous Fortune or Rocky Roader Traveling Empty Final Trails ..... for the Future is where IT's All At Now and here we choose Enchanted Courses for Enchanting Sources shared clearly here for Heritage Systems ReProgramming for Virtual Booting of New Secret Service Services ... Universal AIdDevelopment Works.
It's that for old systems, or extinction. Embrace fait accompli or slowly perish. IT aint rocket science and difficult. .... to Believe in SomeThing Utterly Remarkable, Devilish and Heavenly especially whenever the Evidence for Defence is so widely shared with simple presentations of complex situations being mentored and monitored for Future Broader Band Casting in places everywhere like here, where Prime New Ideas are AIred to be Further Shared with Current RunSlow/NoGetUp&Go Programs simply ignoring Progress as She Sails Effortlessly By to Secure the Future with High Ground Advantage and Deep Underground Foundational Support.
As you can surely imagine, such has an Almighty Magnetism for the Insatiable and Highly Curious.
What Sparks and Drivers your Curiosity to the True Source of Anything and Everything ... and How Does One Manage the Shock of Discovering the Answer and Extensive Roles in Future Arrangements for Presenting the Journeys to Take/Make?
It is certainly character-building;-)
Are No Votes Practical Tacit Acceptance of the Facts/Words as Presented?
Where would you Like to Begin Again ...... to be as ReBorn into Other Alien Worlds , in which you May Seek Counselling and Guidance for when None is Made Ready Available.
IT's Mad, Mad, Mad, Mad Worlds out there where the fittest and most friendly of assets prosper incredibly.
What is for Sale in COSMIC Brothels? UnEarthly Pleasures in Heavenly Delights?
What's not to like when IT is So LOVEly ........ in that Very Particular and Peculiar Live Operational Virtual Environment?
Certainly more than enough to tempt any Saint try Sin with Vice for Mutual Climactic Satisfactions ..... :-) Perfect Orgasm :-) I Kid U Not.
NSFW or the None Grown Up. Sudden Revelations can be Immensely Disturbing and Explosive.
“I favour diplomacy through discussion not through tweets ...” - President Emmanuel Macron
Could the same be said for swift resolute resolution and remeydy in Live Operational Virtual Environments amfM?, for complete mutually satisfactory deNUKlearisation?, and almighty heavenly Sleigh bells ringing, and cliff top nearing from the snowy peaks of DOver?
One would certainly hope so, and it has to be said that as a mere passenger on this magical sleigh ride deciphering instrumental instructions delivered through SMARTer Global Operating Devices ...
Can one be truly accountable to blame for the destructive force unleashed by a truly hostile environment throwing directions off course in a truly TITANic manner?, when all opposing spread betting angry birds be flying in erratic formation? ...
In the late 1980s we had a VAX/VMS system driving a piece of custom hardware, for which the firm who supplied the kit wrote a device driver. We soon discovered that one uncommon but not impossible sequence of events crashed the system; the crash dumps weren't much use as the CPU evidently staggered on for a while before finally giving up, but the custom device driver seemed to blame.
The firm looked at the problem, but all they could suggest was that we avoided the sequence of operations which triggered the crash. Eventually after spending far too long staring at the code I found the reason.
The lowest-level language on VAX/VMS is called MACRO, and as the name implies it is rather more than an assembler. Unfortunately the manual didn't make clear which instructions were pure machine code and which were macros and therefore had side effects. Specifically the instruction to disable interrupts was a macro which pushed the previous interrupt priority level on the stack; the enable-interrupts instruction did the reverse. One code path through the driver altered the stack between a pair of these instructions causing the crash.
I went for an interview at Seqoia (they made large for the time UNIX systems), and was presented a test.
"There is a problem in the printing system that causes lpd to crash and corrupt the print queue. See if you can spot where the problem is likely to be be" they said., before leaving me at the console of one of their test systems, with the root password.
I found the problem in about 10 minutes. I then proceeded to spend the time until they checked back to come up with a patch, work out how their compilation system worked, and compiled it ready for deployment.
I did all of this, and what I would do to test it, and ended up twiddling my thumbs for some time until they decided to check back with me.
From their reaction, I don't think that they expected me to even find it, but I knew my way around both the System V and BSD UNIX source tree quite well. They made great noises about how I would fit into their support team, and how it would be really good if I could join them, and the local Managing Director wanted a chat with me, before admitting that they could not even match the package I was getting where I was working at the time (even though the job agent knew exactly what I would need).
So I left furious, as I would not have turned up for the interview if I had known the maximum package they were offering. I think that the agent was using me as a foot-in-the-door for other candidates.
I was not happy with the agent, even though I was on quite good terms.
I imagine everyone who grew up in the interface between greenbar reports and print-to-disc has at some point spent all morning getting more and more annoyed at a fix that won't, only to find they've been editing the bloody output from the compiler. Found a boss doing that, gently pointed out the error (and pointed out that the reason I spotted it was been there, done that) and got snarled at.
But the best was when I returned from a stint in a youngish start-up to an enterprise I had years of working for/with/in and was given the task of administering the product I had been supporting while on the bleeding edge.
One day I get an incandescently angry phone call from someone who has never thought much of me over the years and who is trying to administer the same product on a different site where they flew in the face of Sperry's advice and eschewed the software-building tool in favor of "pick what's best".
This has meant that a certain vital shared memory resource with absolute addressing involved (BDIs for the cognoscenti) is a jumbled hodgepodge and that now *any* software install must be a very manually intensive thing indeed instead of a few parameters and a COMUS script. Think IRQs in the bad old days, and raise it a few binary orders of magnitude if you don't know Sperry stuff.
Anyway, the lady is screaming at me because she has been looking at "my configuration file" and is demanding to know why I set all the values the way I did. This means of course that she is trying to sneakily take over my project - which unbeknownst to her is jake with me.
"I didn't fill in any of those fields" I say, baffled. I've no idea what you are talking about. I accepted the defaults in every single case when running the set-up utility."
"Oh yes you did! I'm looking right at the file as I speak. Why did you even set all those field values?" she howled.
The penny dropped. I dialed down my own reaction to her snottiness in order to get exactly the right tone. I was aiming for George Sanders at his evil lizard best. It probably came out more like Snidely Whiplash, but that's life.
"Which file are you looking at?" I asked innocently.
She told me.
"Ah, I see the problem you are having now. I'm afraid that *isn't* the configuration file. What you are looking at is the report you get when you run such-and-such a utility, which fills in all the missing info with the defaults that will be assumed by the software. I can understand your confusion now. Why on earth didn't you simply call me and ask me which file you needed to look at? You must have wasted your whole morning."
She hung up with ill-grace once I had told her where she should be looking for her sneaky recce, she in the sure and certain knowledge that the story would be passed to people she *did* respect.
Because I can be a miserable c*nt too if the circumstances call for it.
Or lack of same.
Been there, done that, shit happened, fixed it (usually).
But that was long ago, and in another country. And besides, the wench is dead.
I have no idea how you remember the blow by blow (or bit by byte) interaction 20 years or more down the line. I have problems when I climb the stairs deciding if I need the bathroom or was collecting output from the printer.
Back in 2007 we had major floods across the country in places like Hull etc. I worked for a very large insurance company on the IT relationship/project side.
The idea of having a mobile claims centre that could visit sites was often talked about and several groups had been given the task to sort it out but had just given up after 6 months of wanking around.
I walk into the office one morning and the Directors are all on a telelconf. I asked what was going on and they said that they were thinking of getting a 'Battle Bus' ready to handle the flood locations. "Ha they are dreaming! No poor sod will ever get that sorted!" said me. I think you can guess what happened next.
My director pops out of the meeting and tells me I have to sort it out ready to go for next Monday morning (it was Tuesday). All that had been sorted was they "had a bus".
Well I made up a list of all the stuff I though they would need (laptops/printers/scanners/3G cards/Phones/backup kit etc. etc.) then called round all the IT teams, calling in favours and begging for gear that was needed NOW (not 3-4 weeks delivery time).
I then organised additional support from our telephony and data suppliers, managed for a local branch office to be open over the weekend for staff to get acquainted. To have a local IT support guy on hand with the bus too to make sure it all worked.
So from a cold start at 10am Tuesday I had everything ready, checked off and agreed by 3pm Thursday afternoon. I even scrounged promotional toys and sweets for the kids. I think I had a phone stuck to my ear for two whole days.
I then called the local manager that was to supply the staff. I told him the bus was all ready and equipped and good to go for 9am Monday morning.
There was a long pause. "Oh...I didn't expect that IT would actually deliver anything!"
Yeah I couldn't blame him, our usual IT was pretty bad but I wasn't the regular IT having come from the claims/accounts/dogsbody side of the business before moving over to IT. So he had no staff lined up to man it. Well he got that organised, the bus went out on the Monday to Oxford or Hull and it was a huge success. It was out for several weeks helping folks with their problems. It got masses of good publicity and was praised for being the only insurance company to bother.
So yeah I felt pretty good about it. It was 90% pretty much all my own work and gumption.
You can imagine with such a great boost to the company I would have been in line for a promotion or a big performance bonus.
Nope. No bonus. No promotion. A few of the Directors did very well out of it I heard.
I got a £30 Thank You voucher. The rub was under the scheme the max was £250 but I wasn't worth that I guess. To this day I wish I had told them to stuff it up their arse. I left the company later that year. Oh well!
