In-house techies fixed faults before outsourced help even noticed they'd happened Welcome to another instalment of On Call, The Register's weekly reader-contributed column that shares your stories of weird and wonderful tech support jobs. This week, meet a reader we'll Regomize as "Callum" who sent us a story from his time working for a company in the north of England that ran an 8-CPU Sun server which used …
One was that the designated contracted on call support lived sufficiently far away that the slightest bit of bad weather – and there's plenty of that in the north – meant road conditions became so bad he could not safely arrive within an hour as required.
What happened if said engineer was at another customer's site?
A long time ago I lived in an area of no analog mobile phone coverage (and still no digital, although I can physically see a mast without using binoculars), but for one week in the month I had to have the out-of-hours support phone and be on-call for the organisation that did much of its warehousing and distribution work over-night.
I informed the person who was imposing this rule that there was no mobile signal and that I would never receive the call, and they just shrugged. So, I accepted the phone (no extra payment unless called into the office) and never had a call in the 2.5 years before I left.
Funny. I can't tell you how many times I had to take a call in The Follies in Atlanta, before deciding that it would be more convenient for me to start going to The Oasis because it was only 5 minutes from work.
How many times have you driven into work at 3am smelling like cigarette smoke, scotch, and cheap perfume.
> no reception for the pager.
That's rare.
Partner is sometimes on-call and their government organisation bought "On call" iPhones to be taken home
The iPhone was the one that uses the weird Apple cable so you needed to make sure you had the iApple iCharger for it
Because the phone was $$$$$ it came in a protective Otter box the size of 1990s laptop
The reception in this rural bit of the frozen north is fsck-all
The reception at the remote sites they are on-call to visit is double-plus fsck-all
I suggested they get old style analogue pagers
The "management solution" is to look into Spot satellite messaging beacons
Not IT but, back in the 80's I was on 24/7 call running an inspection contract. Before mobile phones were in vogue, so I had a pager. When that beeped I would find the nearest phone and call the company's 24 hour emergency desk. I'd be told which supplier had called for one of our inspectors (to release goods needing to be shipped asap - usually to be loaded onto a waiting boat or helicopter); I would then phone the appropriate on-call inspector with the necessary instructions. Calls rarely came when I was at home or near the suppler (on the rare occasion they did, I would go and take care of it myself); usually, the calls would be at the weekend when I'd be away with my family (my daughters were keen highland dancers - we lived in Scotland - and there were competitions almost every Saturday and Sunday). In the summer, those competitions were often Highland Games, often located in rural areas where finding a phone could be a challenge - many times I'd beg use in a local shop or pub. Notwithstanding, no matter how rural, the pager never failed to beep!
I also got to know quite a number of rural shopkeepers as bags of frozen peas were often needed to treat sprains (yes, I kept a freezer spray in my first aid pack, but only as the first line treatment - a bag of frozen peas wrapped in a towel and strapped to the injury was the next stage).
I had to carry The Pager periodically at one place. It was great for getting away from boring people in pubs. Visit the Gents. Use the payphone to call the magic number for free. Return to table and pick up glass. Pager goes off, and it's obviously not you pressing some button to make it happen. Sorry, have to go!
> and never had a call in the 2.5 years before I left.
You'd need more upvotes.
Question: Was it ever noticed that you never had a call, for whatever reason?
I.e.: Nobody called you vs. nobody reached you vs. nobody reached you and nobody noticed vs. nobody reached you and nobody noticed 'case nobody gave a sh?
This post has been deleted by its author
Once settled into a new job in 2007 I was added to the on call rota, weekends and evenings every few weeks. It was a good deal - the on call phone that never rang, £40 per month contribution to broadband (deemed vital to provide support, and 2007 prices were closer to £25/£30 per month), a few quid extra pay for giving up a Friday / Saturday night drink every few weeks, and the business we were supporting didn’t routinely work evenings and weekends.
Eventually a bean counter from higher up the food chain spotted this nice little earner and scaled it back somewhat. We could hardly complain, but did and were quickly knocked back. Busted…
I had something similar: Business decided that needed 24/7 support for ONE of their processes (FTP upload of data to their system)... thing was: the FTP upload was scheduled for 3pm weekdays. Never after hours.
So the money-for-jam oncall bonus was to cover the inconvenience of having to carry an extra phone and learning to say "sorry your password reset (WTF do you want to log in 4am on a Sunday morning?) isn't covered by the oncall agreement, I'll log a ticket and someone will contact you 8am Monday"
(It all fell over when a 'precious' business group had asked the desktop manager if support would be available over the weekend while they got they Big Special Report out - it was for a .govt.nz so no doubt they were meeting a ministerial-imposed deadline.... desktop manager "yup, fill your boots"... Cue a phone to me late Friday night/early Saturday.... cue my usual speech about logging a ticket... cue them saying "no, we were told there would be support"..... hmmm... "who said that".... "Billy-Bob Wankenstein Fucknuckle III"... oh (I say in my head).... out loud: "no-one told me about this" back and forth a few times, and evetually they said they would ring "my manager" (who was not BBWF3)... "ok".... 30 minutes later BBWF3 is ringing me, first saying I needed to support the Precious People.... "need?" .... I only NEED to do what I'm contracted to do.... in the end I think I helped them... might have been a printer issue. But come Monday I resigned effectively immediately from the oncall roster, and also emailed BBWF3 and their manager about how poorly they managed the whole thing - offering up my services without telling me, or at least going through MY manager.
Then months, years later - at least 3 times - they tried to start up 'proper' full cover oncall, and I would relay this story of how not to do it.... would generally kill the plans until next time.... fuck the BBWF3s of the world and the horse they came in)
I used to be on the on call list at work. Thankfully I was so far down it that I think it would have taken a major disaster for me to have been called.
I say thankfully because I can’t drive, and live far enough from the company that it’s not practical to walk, and difficult to cycle.
As such, me getting to work if something had failed over night, I would have had to get a cab.
Thankfully I was never called.
On one job the customer demanded that we supply out-of-hours cover, so we had an on-call rota, with appropriate allowance added on.
In practice, should something go awry outside contract hours, we did nothing until we saw a charge form faxed through.
All of a sudden the customer saw that it was very advantageous to leave it until morning, when the job would not affect their budget.
The other fun one is when some bright spark gets the brilliant cost-saving idea of flexibility and "cross-training", so that one person can work on multiple products or toolsets, because "there are always quiet times for each product and people are not utilised fully".
Then of course they are seen as an available resource and headcount by each of the support groups for said products, and so are assumed to be available at all times for call-out and work generally.
And Sod's Law dictates that nothing will happen for a while, then three calls will come in for urgent support for different products in different locations (always of course on a Friday afternoon around beer o'clock) and the real fun "discussions" begin about who has priority/seniority etc, and why no-one has yet realised that being on three different headcounts doesn't make you three people who can be in three places at once.
What usually happened where by dad was a CSE was that his boss would send someone from the next nearest office, and the call the customer and give them an ETA for their arrival. Sometimes my dad was that next nearest CSE and would have to drive a couple hundred miles at 2am. If he had to detour through the regional office to pick up specialty tools, the trip could be even longer.
What happened if said engineer was at another customer's site?
The perennial problem with such contracts, especially away from major urban areas.
Not mine but one from the days when my brother was a manager at an IT Support Provider who had little presence in the north of England and none in Scotland:
Sales - We've just won a major support contract!
Engineering - Oh yes, where is it?
Sales - Aberdeen!
Engineering - What's the response time?
Sales - A guaranteed two hours!
Engineering - You do realise our nearest support engineer is based about 4 hours away, and the next one nearer 6? We could barely promise next working day.
Too often engineering would get ignored and told to just get on with it but I believe in this case the contract was refused.
This reminds me of the notion of an "appeasement engineer" (someone with an engineering diploma so fresh the ink is only just dry), in what I believe is the very first BOFH episode (certainly the first I read).
I have used the term appeasement engineer many times.
I'm still sad I can't find the BOFH quiz where some of the answers were "I have everything I need in the car" which was an excuse for the engineer to go back to their car and drive away, quickly.
IIRC there was something about recognising an experienced printer repair person because they would have the scars from getting their tie caught in the rollers.
> recognising an experienced printer repair person
My uncle supported IBM Line Printers in the 1960s. They were LOUD, so were enclosed in sound reducing housings. Of course when things were going wrong it was the custom to open the enclosure and put his head inside to see (and hear) the trouble ASAP and get the operation back on-line fast. He said this caused his deafness and all his coworkers' too.
(For Reasons, ties were not required in that shop.)
We once had a new middle manager who, as is normal, wanted to make their mark. She demanded that I wear a tie.
I demanded that she sign a declaration that she had ordered that health and safety rules were to be ignored. We didn't just sit in an office; we visited places with very unforgiving rotating machinery.
She saw the light.
The first company I worked for made a variety of technical products, with increasingly complex control systems as the world progressed towards electronics. After completing training, I was sent to sites to get new machinery/controls up and running and handed over to the customer/end user. More than occasionally, I was called to 'have a look at' some other piece of equipment which was 'giving a bit of trouble' which was not in our supply and sometimes had been installed a long time ago. Often the problems were simple and my (excellent) training covered the general principles behind the processes involved. Being helpful was good customer-relations and considering that these customers were responsible for future orders we assisted where reasonable.
I drew the line at one site where the contractor asked me to set up a large number of automatic actuators; I showed him how it was done and left him to it..... whereupon he called in the manufacturer, at some cost.
Or, as was the case with a previous work hat, I was sent to one of our satellite sites (a company we'd acquired) and found the manual was still lying around for the PBX (A BT badged Meridien that BT sold a lot of at the time). The site manager had recently "left" and the office staff in particular were happy to be rid of the obnoxious ****. The status I acquired by being able to remove his name from the phone quite surprised me.
I got to see some paperwork about a support contract our company were looking to bid on. The radar sites were dotted all over at various airports. Some of the sites were in Orkney and Shetland. The intention was to service those from Aberdeen.
I told the boss to check the ferry timetables carefully. Assuming you got on the very next ferry after the call came in, it could be days before you reached the relevant island. Drive from the port to the radar site, get out of the car and see the ferry heading back out to sea. Flights would be restricted because of the fault with the radar, and the engineer would probably need a car load of parts anyway.
How many calls a week did you say?
Reminds me of the time I worked for a large blue company doing break/fix for all sorts of customers. One well known (sadly departed) supermarket helpdesk called me one night to demand an engineer meet the p1 SLA of 4 hour response time at their store in Lerwick. The rest of the team enjoyed the half hour it took to explain that "it was too late for the ferry", "well I'm not going to send him in a rowing boat" and "have you an atlas to hand? look at the back, see the wee box that Shetland is in? That means it's not just next door to Scotland, it's over 100 miles away. he'll get there when he gets there."
At a previous job, we had a series of routers - Cisco GSR 12012 - break down on us in pretty rapid succession. It was deemed this was due to a poorly designed backplane that would bend out of shape due to the warm air circulating around in the metal box, which would lead to poor or intermittent connectivity to the inserted line cards. Cisco accepted that the fault was in their construction, and when we asked to have the troublesome boxes replaced on our 4 hour Advance Replacement contract, they said that "Sorry, we only offer 7 business days replacement on the chassis". Which was of course met with severe skepticism from those that negotiated our support contracts and knew exactly how much we were paying per annum for 4-hour replacement of the chassis.
As a compromise (since they didn't have any of the big metal boxes in our nearby depots) they eventually renegotiated our support contract, waived a bunch of fees and send us a stack of backplanes to replace the faulty ones. A colleague of mine replaced one and came back exasperated to the office - "You know how many screws there are in one of those?! I counted 136 before I gave up!" - so Cisco eventually agreed to send an engineer to do the remaining replacements for us, on their dime.
Ah, the old mantras:
1) why use one screw when you can use 20.
2) at least one screw must be hidden away and positioned such that half the machine has to be dismantled to access it.
3) another screw must be in a position which requires the engineer to have an 8ft arm with at least 4 double-joints in it to reach.
4) as many different screw head types and sizes must be used as possible (with extra points for using Torx, or Allen bolts in spaces too small for an Allen key or driver),
5) non-standard sizes must be used, ideally positioned over gratings or other locations where they can fall and be easily lost.
A company I worked for had a designer who believe that every single screw in a machine must be the absolute optimum size and length. If an M3x6 screw would work that's what he'd insist on, even if it was the only one on the machine and an M4x10, of which there were several, would be just as good. There might be some validity in large volume manufacture where saving even a fraction on each machine has an impact but we were building custom 5-figure machines and having to keep so many different sizes in stock was actually costing us money.
Or the other peach, of using for example an M3x6 and an M3x10 in close proximity, but in such a way that if they were swapped the M3x10 would protrude just enough to catch on something moving within the tool and cause all sorts of interesting mechanical collisions and interactions with automation.
Or - yes looking at HP
They design a standard 3 1/2" size tape drive (back in the days when DAT was pretty advanced stuff) and put a flexible link between two boards "just" inside one of the mounting holes. So you have to rummage around in the box of "I never throw those away" screws for an extra short one for that mounting hole.
Most vehicle manufacturers seem to start with a heater box on the assembly line and then build the vehicle around said heater box, which has radiator fluid going through it and eventually clogs, leaks or both. Please don't ask me how I know, the nice man here the asylum said it was best to forget some things...
"using Torx"
Which nobody except the vendor's engineer was supposed to have. Our HP tape drive was installed* with a transit locking bar held in place with Torx. The engineer arrived to remove it only to find that I'd already done that with my cheap screwdriver set.
* i.e. fitteing into the rack.
That's the same logic as the management at a former employer of mine. Policy was that all quotes were written in Word, but sent to the customer in .pdf. normal enough, except that the reason was "so they can't edit the quote and give themselves better terms."
That was one of those moments where I could feel myself getting dumber by osmosis.
Have you met Boring Barry the Renault service tech at the local main dealer?
Every Sunday lunchtime, smashed out of his skull down in the pub (Where else), telling us again his two or three stories, invariably before he lost bladder control:
Triggering the airbag with a bulb & two test leads (ONCE), instead of using the correct (Non available) tool.
Training course, querying the placement of the oil filler cap & being told "You* are the last person we think of".
Erased from memory.
*The service tech.
Icon - He wore really thick lensed glasses.
Once bought a second hand Espace from a Renault dealer. "Belonged to the head of the service department" before we had it.
So a month or so later, when the clutch was becoming very, very difficult, we took it back for them to fix.
And fix it they did, and for another couple of months all was well until one day the clutch basically fell apart.
I took it to my long-standing "back street" bloke who took the thing apart and showed me an array of sheared, stripped and generally mismatched screws. Apparently the casing was held together with 17 bolts of a very particular type, tapered and exactly the right length. Whoever at the dealer had earlier "fixed" the thing had evidently spilled them on the floor and refitted the casing using 17 random bolts he'd recently swept up.
Back street bloke reckoned there were about five (if memory serves) of the original bolts used, 12 floor sweepings.
Needless to say we needed a whole new clutch and then camped out at the dealer with the oily, dirty broken clutch on the boss's desk until they agreed to refund us a reasonable proportion of its cost.
M.
> you've all worked on Renaults...
Over here it was first-run Ford Falcons. Bob McNamara engineered-out every part-penny. To get oil from the bottom of the engine to the top, it didn't have a drilled oil-passage. No, instead, oil was fed into a head-bolt hole and ONE under-sized head-bolt used in THAT hole to pass oil. The assembly workers didn't care and the dealer mechanics never saw that memo. Many-many Falcon engine rocker-arms quit in weeks from lack of oil. (Is amazing how little oil rockers can run on, in WWI aircraft open-rockers were common, tho competing with oil-in-fuel rotaries.)
A relatively minor council official who was deemed critical to dealing with the infrastructure after a nuclear attack was allocated a place in the regional bunker somewhere in the countryside 10+ miles from where he lived. As was usual for town dwellers at the time he didn't have any vehicular transport so enquired how he should get there if the balloon went up.
The answer? They gave him two bob* for the bus fare!
*For youngsters and non-UK people two shillings, 10p in decimal currency.
Definitely one case where ignorance might desirable if not actually bliss.
Four minutes of absolute terror v. "Oh what was that bright…"
At least in these enlightened times our lords and masters couldn't be arsed to give the polloi any warning.
"five-figure leased-line connection to the support vendor"
so minimum £10,000 ( pa presumably)
Thats amazing , times have changed eh ?
and thats just for the support vendor to ping it every 5 mins? not the main architecture ?
It doesent say when this is , but "sun server" gives an indication , thats a probably a years salary
The first Sun servers with a dual CPU card were the SunFire series, 12K, 15K, 20K, 25K that came out in the early 2000s. £10,000 in 2002 for a leased line isn't a years salary and not horrible for the cost of a lease line.
disclaimer: I worked for Sun in the consulting side in the early '00s.
I used to play with those! Passive backplane, as many CPU memory or io cards mixed any way you had slots for. If booted in the right mode you could ask the os to empty and turn off a card while the machine kept running for upgrades or replacements.
Real enterprise hardware.
And then everyone noticed it is cheaper and more reliable to combine two computers in a cluster instead of endangering that one machine with hotplugging CPU and / or memory. Storage, PSU and fans are the only real world Hotplug components left.
But even for a fan change everyone avoids doing it hot-plug since it still means you have to pull the server out of the rack. I.e. physical movement, including the cables attached, while running, open the hatch for the fans and doing the exchange... Additionally use the managemet tools to make sure to let the pre-failed component blink if it is not dead yet to make sure.
Some say that we all got cowards. But the reliability statistics and cost-when-something-goes-wrong statistic are clear. And experience of many many admins clearly warn to do it the most boring and safe way possible. Or, to link to Kelsey from 74 gear: "You are an airline pilot, zero balls needed. Your job is to fly the plan in the safest most boring way possible. War pilots, they need balls."
This is true, hence why we still have FT in the servers. Usually power supply, fans and storage, quite often all three hot swappable. Next FT is RAM, ECC (thanks AMD for giving us that with nearly every consumer CPU). Where the cost starts to jump are higher RAM FT techniques which combine several ECC Modules with a different logic to make whole RAM chips or RAM modules fail without breaking the machine like Chipkill and/or Hot-Spare RAM and/or RAID1 RAM. They all have in common to be 100% hardware level, thus the OS does not need to know about all that below. (OK, some implemented their Storage-RAID-WRITE logic in the driver, and the BIOS/UEFI only has enough to read and boot the OS, quite often even limited to RAID1 for the OS drive)
Technically even x86 still can do hot swap CPU, but then it switches to special OS capabilities to handle that and the price jumps more. Funnily Windows NT can do that since at least NT 4.0 server, and I know such machines existed somewhere 1996 to around 2000, but I never actually saw one like that. For Server 2003 I never even heard of any machine making use of it since the price literally explodes vs. clustering.
Dammit, now I have to keep my urge and hands to try "eject CPU" on my "Server 2025 on Ryzen 5950x with ECC since I could not stand Windows 11 any longer" machine. Might just throw a "Nope", a "OK, now you have one CPU core less" or a <random-color>SOD.
Heyrick,
Generally we are. The problem is that dealerships employ an array of clueless front-desk monkeys to form an impenetrable barrier between us & the customers, presumably because management hired some consultants who told them that the customers are all "nice" people in clean clothes who would consider mechanics to be untouchables from a lower order.
The monkeys have another important job, to "interpret" every fault reported by the customer in such a way as to make it meaningless to the poor idiot on the spanners, thusly guaranteeing multiple visits for the same fault.
The Sun (Sparc?) server was stated to be old so I imagine the dual cpu cards would not be at a premium so retaining a couple of tested refurbished cards at Callum's site might have been a cost effective and practical solution.
I recall Sun stuff when good, as it often was, was very good; when it wasn't, it was a right cow.
Trying to run Solaris 2[.0?] reliably on an early SparcStation 10 with two Sparc CPUs was one such complete cow. ca 1992·3?
I'm from the Northeast so I'll agree about the bad weather, but where is a company located such that weather can impact the ability of someone to get to it? Unless you're up in the hills or on the Coast of Inaccessibility (Workington, Maryport etc.) you are going to be near an urban centre. Maybe this is a case of one emergency help line covering the whole region, in which case, why have the contract at all?
Most likely the engineer was the wrong side of the Pennines or Moors.
Manchester to Sheffield or vice-versa is often basically impossible, despite being nominally an hour away. The Snake is not to be trifled with!
A lot of managers only look at the distance as the crow flies, and completely ignore the actual route.
Eg NYC decided to send a lot of kids to the "nearest" school as the crow flies, despite the fact that it's an hour drive along the valley, over the nearest bridge, then back. Having driven past three other schools on the way...
Or a mechanic friend of mine based in Cardiff who used occasionally to take calls from insurers, "we have a client broken down in Weston Super Mare. Our system says you are ten miles away, can you get there in half an hour?" It would be impossible today but this was in the days before the Second Severn Crossing was opened.
M.
US based test and repair techs will get most failures fixed within a few hours at most, with most of the time spent waiting on a tech to physically arrive. EU based techs tend to be a bit slower, but that seems to be related to travel issues. Many techs use the tubes and walk where US techs mostly drive. Outsourced call centers to India and Asia, with the exception of Japan, will take a week just to actually see that a circuit is down, much less find the problem.
Or not.
The very large company I worked for in the mid-90s had a large engineering department, of which I was a member. At the time we had our own internal IT support team until some arsehole bean counter decided to outsource our IT support to IBM.
Our own team sat in an office in the adjoining block and would often appear within minutes of putting the phone down to them. The IBM contract was negotiated on the basis of the response time being proportional to the number of machines affected. Thus, more than 25 machines (we had about 100 in the section) and the response time was less than an hour, 5-10 machines it was next working day, but for a single machine it was 5 working days.
My machine went down and on ringing the support line I was told that someone would be with me "early next week". Now at the time I was a company SME and my charge out rate to the business was £65 per hour. My PC was also one of only 4 in the company which had a specific software package installed and I was one of only 6 people authorised & trained to use said package. Therefore using a spare PC wasn't an option.
My line manager was a quite senior person in the company with the title Professor. I pottered off to his office and told him the good news at which point he went ballistic (not with me, I hasten to add). The issue was then upped to the head of engineering (who sat in an office just down the corrisor). The engineering department at the time was just over 500 people. This was then (I was informed) bounced up to director level and just over an hour later I had an IBM support bod at my desk. I should explain that I probably could have fixed the machine myself as it was a driver error, but our machines were locked down as part of the IBM contract.
IBM did not get to renew the contract and we went back to having our own in-house team.
In the early 00's I ran a team replacing Nortel phone systems with Cisco VoIP phones. Since the sites were hundreds of miles from each other and required reliable service I had local servers at each location. After this was rolled out, the Big Boss in their infinite wisdom decided that everything would be outsourced to EDS.
EDS set up a call center where any trouble tickets would go into a digital black hole - they were entered, noted and then disappeared. Folks recalled that I had a hand in those phone systems, and soon I was getting calls for help. EDS hadn't bothered to remove my access, and I found myself fixing issues as they came up. After six months or so, EDS got their act together and started checking on all those tickets, only to find that they had been mysteriously dealt with.
I later found out that the outsourcing contract was a pay per fix deal, where EDS would get $40 for every password reset or name change and a nice chunk of change for a server reboot. By stepping in and keeping things running, I had ensured that EDS didn't get a dime. Their accountants were the ones to sound the alarm, since this lucrative account hadn't made the company any money.
Alas, the tale had a bad ending. I got reprimanded for not using proper channels, my accounts that I used to connect to those servers were removed (I had others) and EDS was officially in charge once more. I suspect there was money being passed somewhere, but wasn't stupid enough to learn more. My workload went back to normal, aside from the occasional request from a boss's administrative assistant. I took care of those!
Our in house server room had temperature monitors on the rack servers. I rigged these to feed into an Icinga dashboard with some parameters to flag up anomalies.
after a year or so, a red light appeared on the dash. Inlet temperature was showing a slow climb (which was a separate alarm to the threshold one).
Aircon maintenance were called to check. And lo!!! They found a failed/failing valve (or something). The engineer was impressed as changing it early saved a lot of the unit.
Of course I am still looking for a job - but that's where all that experience is a drawback.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2025
