I would drive 100 miles and I would drive 100 more just to be the man that drove 200 miles to... hit the enter key The best laid plans of mice or Register readers can founder on the rocks of iffy connectors and wobbly cables, as this week's entrant in the Who, Me? archives discovered. Regomised as "Chris", our hero's tale takes place long before the seemingly infinite (and infinitely expensive) cloud storage of today. Even before NASes and …
Not clear from the story what the old server was needed for after the new one was put in place, but assuming it was needed then replacing a duff keyboard seems like a better bet than changing the BIOS to ignore errors... well, that would be my thinking. Oh, and next time he would have known to ask someone to check the KVM for errors.
As a side note, in a previous job we had a server which had a BIOS error on reboot which was annoying when working remotely. One solution would have been a remote-access KVM switch, but they were more money than the boss was willing to spend. The server was eventually retired.
As a side note, in a previous job we had a server which had a BIOS error on reboot which was annoying when working remotely. One solution would have been a remote-access KVM switch, but they were more money than the boss was willing to spend. The server was eventually retired.
I had one of those, standard practice was to *always* leave the KVM switched to that server just in case.
We eventually managed to do a P-to-V onto vmWare and retire the hardware.
You can have errors reported without having the keyboard error enabled. Not sure why you would ever want this option enabled on a server. It essentially states you have no keyboard "press enter" (or F1) to proceed and then waits for interaction. It's not so much the keyboard as a poor choice of BIOS option configuration.
The classic -
"Keyboard missing, press F1 to continue"
type error message!
Had to set up some old PC's as information displays, alas there was no BIOS setting to skip all errors on that PC. I didn't want to leave a PS2 keyboard plugged in, so in the end I took apart the keyboard, removed the PCB bit, put it inside the case and fed the cable through one bent blanking plate on the back of the PC into the PS2 socket. PC now boots up without a keyboard!
Or the even better "UNDEFINED FAULT - CONTACT SERVICE" that I got from an important piece of hardware a couple weeks back.
What was even better is the hardware worked just fine, and the root cause was another piece of hardware not doing it's job properly, so the first piece of hardware panicked and decided "I'm broken"
Reminds me of the "IMPOSSIBLE ERROR" message I got from the wonderfully named D02BAE routine from the NAG Fortran Library Mark 14 on the Cray J932 way back when. I was running multiple instances of this ODE solver in parallel, and it turned out library had not been compiled for parallel execution properly (brilliant, on a 32 CPU shared-memory parallel machine). This meant only one instance of the the named common block used by the routine was created, so the 10 instances I was using were happily overwriting each others data, causing such confusion that an IMPOSSIBLE ERROR was generated.
Not a hardware fault, but it took some hunting down
I'm a paranoid sod. I tend to stay on hand until the project-de-jour has been fully installed, tested, and is currently in operation. Because I just know that Murphy has one or two in his pocket for the moment I turn my back.
Even with that level of paranoia, he still gets me occasionally.
I was visiting a friend once. He had to install a new VPN router at a customer site, so we toddled off together.
It was about a 3 hour drive to the customer. He then spent ages trying to set up the router, in German - his mother tongue. I was meanwhile given a guided tour of the logistics company.
When I came back, he was still slogging away on the router. I was kicking my heels, so he got me to install a couple of new PCs, whilst I was waiting - they were pre-installed, just needed plugging in and setting up the user accounts, default printers etc.
That finished, I went back and my friend was now cussing and swearing over the router.
He was trying everything, but the router's address wouldn't work. Eventually, he asked me to look at it. The dialog window was too small! I could just see a pixel of black at the edge of the dialog box. I managed to select the text out of it (the wonders of web browsers) and paste it into Notepad.
Turns out, the dialogs were sized for English text, but not properly tested with foreign languages and German has the odd long word... Leistungsschutzrecht, for example.
Anyway, it wasn't asking the router's address it was asking the the router's address network which was off screen! Quickly replacing the routers .1 address with .0 and the router connected with the HQ and the VPN was up and running!
If only the manufacturer had tested their translations, before chucking out the firmware!
Still, Windows Vista was even better. Microsoft decided that Redmond could just as easily do the German translations as locals in country... So we ended up with the classic network neighbourhood dialog translation of "adjust the attitude of your neighbours". Unfortunately, it never did do what it said on the screen.
"Those security people were just doing their job. Imagine hosting servers for multiple companies on a system and allowing anyone with admin access for one company's server unmonitored access to all the servers..."
never had that issue at the ARK DC I used to have to go to.
other customers racks where properly locked in the cold isle but most where unlocked in the hot isle.
I only found out when I went looking for an unused power cable to power my laptop psu while doing some work and saw one connected to a monitor, I thought I would borrow it but the screen was showing device names not part of our naming convention so I stopped and closed the doors to that rack.
cold isles where access controlled but hot isles where open to all.
and potentially spoiling a good headline and tag, but the Proclaimers walked 500 miles, and were prepared to walk 500 more! A 200 mile drive is chicken feed!
(This as somebody who lives north of Hadrian's Wall and hates the aforesaid singing duo - so not springing to their defence, just being an annoying pedant.)
Go military and call them "Klicks", and the rhythm will work again.
However, seeing that the main goal with Brexit now appears to have been the ability to return to the Imperial system when going on a binge, why not replace "walking miles" with "drinking pints"?
I believe that this would make the song work well in a Scottish sense as well.
I've had a 390 mile round trip from Uxbridge to Newton Abbot and back in a day, just to press return on a server. Mainly because this server was locked in a secure room, and I wasn't allowed to tell anyone on site the code for the door (the two people who did know were off sick or on holiday in the USA), and my colleagues in Chester worked out that I was closest to Newton Abbot, but only by a few miles.
Back in the early 70s I had a one day round trip from Wellington to Auckland just to convince a George 3 sysadmin that his system config was wrong, explain why, and fix said config. Annoying, because I'd already sent them what was substantially the same fix and had it ignored. Total working time onsite: under an hour, plus a discussion about why it was really silly (and slow) to run tape sorts using simulated mag tapes in the filestore when a perfectly good disk-based sort was a one of the standard utility programs.
On the other hand, it did get me a a couple of rides in one of the, then shiny new, B737s which had replaced Viscounts on that route. so it wasn't all bad.
I once flew from the UK to Australia to conduct a two-day audit of a potential contractor - say 20,000 miles. The contract was for a job in the UK, and they had a UK office, but TPTB insisted the audit was conducted at their head office. I left home around noon on a Saturday and arrived back home about noon the following Friday. Totally knackered, needless to say.
However, I did see the Sydney Harbour Bridge from the road (my taxi from the airport missed a turn and had to cross over and back) and river (part of the audit required a visit to a site a few miles upriver, the quickest way being to take a river taxi). My biggest regret was that my return flight was just a retrace of the outward one, meaning I couldn't claim having travelled around the world in those six days :(
Did that trip in the noughties - Wellington to Auckland, hired car at Auckland, not paid for, luckily had a credit card. Kangaroo petrol (not driven a manual for over 6 months).
Turned up at the office, had to wait for network engineer, had lunch looking over the bay. Did my thing and flew back. Saw my house (we lived on the flight path - J'ville) and was home by 7pm.
These days you can actually attach a robot finger to the keyboard, next to the key you want to press and then control the finger remotely through a wireless handheld phone.
It's amazing how far we have come when it comes to technology. It used to be unthinkable that you would be able to press enter (or any key for that matter) without actually being physically in the room, now you can be pressing keys from the comfort of your armchair while being 1000 miles away from the keyboard.
Or knocks it behind the equipment when dusting, where on activation it shorts out the 48V @ 300A DC, thus giving the entire quad rack the finger.
I saw the end result of a cleaner knocking a metal coffee cup down the back while the system was running ... No video, alas. It actually managed to melt a good portion of the cup as the so-called quick-acting fuse was blowing. The arcing was quite impressive. Halon didn't dump, thankfully.
No word on the state of the cleaner's pants ...
"A Dark Traveling" specifically has an electric security golem on loan from another universe (read the book) who is passed off by the child characters as not speaking English. Which is true.
"Futurama" has Bender, an evil robot built to bend things, whose closet space is explored in "I, Roommate".
I was involved in software performance, and had to deal with a motor manufacturer in the US. They had a performance problem in production (but not in test) which was slowing down the production line. It looked like a simple configuration problem, but they could not stop and restart the server because it would take the production line down.
They needed "on-site support". My management team said not until you apply the fix. Because the CEO of this corporation plays golf with someone senior in my corporation, my boss came in and said "can you go to the US ... today(Friday)? a plane goes in 4 hours. So I went, arriving late Friday evening, checked into the hotel, and looked for the instructions to go on site etc. and found an email saying "We managed to restart the server and the fix has fixed the problems. If you haven't left yet you do not need to come to visit".
When I went on site on Monday they team were very apologetic, and said it was Chinese Whispers up the executive chain. They were told to demand on site support, even though they knew it would not help. It was a case of the executive shouting "SOMETHING MUST BE DONE!". This is something we could do. Go do it. Next!
I stayed a week and did a review of their system. In my report I said they were a very lucky team. They looked very pleased, until I said there was so much wrong with their system - they were very lucky it had not broken in many places.
I've done similar things. 3 times basically. (semiconductor equipment support).
First time I arrived on site and the stuff I was sent in to do because there was no way local team could do it without support had already been completed by local team without my support. Spend nearly 2 weeks mostly as an inbetween and interpreter to work out some work instruction issues and "incompatibilities" between OSHA and EU/Dutch work safety rules. (If anybody can tell me what OSHA rules apply for HV insulation testing (megger test) I'd be much obliged. Still haven't worked that out 8 years later).
Second time around I knew going in I'd mostly be the "be there locally to get shouted at and run interference so the home team can actually focus on getting answers" guy. Worked out that way, but was a very tense series of calls and meetings with the customer while there trying to skirt the borders of ITAR regulations, company "crown jewels" and providing customer with enough information to keep them happy. Never been so mentally exhausted after a 90 minute meeting in my life.
Third time was an: arrive on site, find local team left a set of transport lockings in place we had explicitly asked them to check, remove those, then spend another week mostly waiting around for them to get things up and running.
I was consulting for HP managed services a couple years post 9/11 and they had an old DEC office building in Reading, UK. One of their primarily US-based clients was rapidly consuming disk space in the UK/EU area and due to run out at any moment, so they had procured a pair of new arrays and a new tape backup system which finally arrived.
Since I'd worked with that client's systems before and knew their special configuration I was called up on Sunday and asked to be on a plane tomorrow instead of flying to the site I usually worked from. Airport (central US during winter) was fogged for a couple days but I eventually made the trip on Wednesday which meant I couldn't get everything done by COB Friday - and I wouldn't be able to access the site over the weekend so I couldn't leave until the following Tuesday.
The big problem was the fog delay meant that I didn't get back in time for my girlfriend's birthday on Saturday, which she was pretty miffed about. Probably mostly because I got to spend a weekend gallivanting around London while she sat at home. I would have flown her out for the weekend but I checked fares and it would have been $1000+ for the last minute travel. I didn't mind my $1700 fare when I was expensing it but it is a different matter when it is out of your pocket and for only a weekend lol
Once had to drive from N Yorkshire to Edinburgh to turn a server on.
The beast in question was a low end IBM Netfinity with no remote management card. When given power, they would spin everything up and light all the LEDs for a couple of seconds then go off, waiting for you to press the On button.
Needless to say, we got a call describing exactly this. I suspect somebody had unplugged it to do some vacuuming. After asking in as many ways as possible "Have you pressed the On button on top of the server", I eventually got the inevitable "Do you think I am stupid?" question. I naturally had to bite my lip hard to prevent answering.
So jump in the car, head up the A1, then A697 (best of the many routes I had tried to that point) to walk in and press the power button. Fortunately, I had managed to fit in another job so it wasn't a complete waste of time. Plus I played tourist on the way back and came back on minor roads through the borders and Kielder forest. Nice leisurely drive home.
"Could you unplug the server from power, plug it back in and turn it on again? Sometime the contacts need that. Stick for a Minute after you plugged it in and turned it on, sometimes the the box does not behave."
Been there, done it that way. Whether it is the "On" button or the "Of course it is plugged in" etc... I used the "faulty contacts" excuse a lot to force them to actually do that they say they did.
I actually have something where that is a real issue. That being the wires going to the seat belt tensioner on my first Gen Citroën Berlingo. Whenever the airbag light doesn't go off a few seconds after switching on the ignition, it's time to reach under the seat, find the plug and unplug and plug it back in a few time, and it'll be OK for a year or two again.
Unfortunately, they could master the unplugging and plugging back in bit, which is when the server would briefly spring to life, apparently doing a self test or something. Lights would flash, drives and fans would spin, then it would go off waiting for the On button to be pressed.
It was just impossible to get them to press the button to switch it on again.
No matter what I said. "Its the white button on the top right front corner. Please just press it!"
Similarly, I drove from West Yorkshire to Edinburgh one day to commission a phone system. (Early 90s I think).
We'd been subcontracted by a local Electrical contractor, who normally did industrial stuff and had no idea about PBXs etc. All the equipment had been sent up in advance, with instructions to the sparkies to mount the exchange in the plant room, hook up power, and run cable the phones, but no actual fitting.
My job was to second fix and commission. So connect everything up, test and sign off.
Bos reckoned it needed about 6 hours or so to complete the job (just me). Set of around 04:30, aiming to get there for around 09:00.
Turned out the senior sparkie got bored one day the previous week, so he'd looked at the instruction manuals that came with the gear, and decided to 'have a go' himself. Oh oh, I thought!
Took a couple of phones sockets off to inspect, and had a look through the exchange wiring. Turned out to be one of the neatest jobs I'd ever seen! Couldn't fault it all all.
Switched on, spent about 30 mins programming the system to the requirements, then a walk test of all the phones, make sure everything was fine. Signed off, and was out of the door. Don't think I'd been there even 90 mins total!
Called the office, no jobs that far up North, "Do what you want, see you tomorrow".
So spent a nice rest of morning wandering around Edinburgh, including going round the castle, and finished with having a decent pub lunch. Headed off home about 13:30, got home by around 18:00 or so.
Sometimes you do have to stand in front of the box to make sense of what you're seeing.
The SQL Server box was performing fine, but there was one processor that was permanently maxed out. When we got the local people to log on, the logs went back to being sensible. Left alone for a while and a single processor hit the ceiling again.
Eventually after much pulling of logs and scratching of heads my colleague went to see it. He came back after a long day on the road with a sadder and wiser look on his face, telling tales of the Pipes screensaver, complete with the teapot options (for this was NT4). He had a nice day out though and brought back biscuits.
Being a Compaq server, a remote lights out card would have been a good precaution for the servers in the satellite sites....... but then these things have a habit of seeming like a great idea after the event!
Oh yes, and don't assume anything. Just sit there and watch the job complete - especially if there's any overtime going :D
Compaq servers had so many quirks that they brought with them to HP, that ILO was a must unless you had your office right next to the server room.
Proliant 1600 with a triple PSU instead of the single?
Will halt on boot if one of the PSUs doesn't have power or is non-working for other reasons.
AAAARGH!
At least my insistence on ILO saved me from an hour-long journey to push a key.
Had something similar, except I was the one that caused the trip this time.
We have a major European customer with several fabs in various countries, and back in the day they also used to have a couple of fabs in the US as well. As we were installing a new tool there (in Phoenix) which we had several of over this side of the pond, it was deemed worthwhile to send a hardware engineer and a process engineer (me being the latter) across to help out with the install and start-up.
All went surprisingly well, and the FE's got the tool up and running a week ahead of the scheduled date when the American process engineer was supposed to arrive. So being bored, I tinkered around with it and ended up getting everything set up and qualified the day before that date. So he basically arrived, met me and the customer, took us all out to lunch and then the a couple of days later headed out again. And that was (apparently) only due to the earliest he could rebook his flight to, although I have the feeling he grabbed a company-funded mini-break when he had the chance.
Even got reports (both from the customer and internally) that it was the smoothest and easiest start-up they ever had, which of course went down well with my boss too. So then I too took the same opportunity and had a nice few days of sight-seeing too (Grand Canyon and various other bits) and headed home for Christmas.
Many years ago, I had to remotely support a DOS/Windows laptop. The laptop was in the USA and I was in Australia, so the only way to do this was via fax.
The person with the laptop was about as computer literate as your great grandfather, so typing in commands was going to be a *real* stretch for them:
Me: type in "some command <enter>"
USA: I've typed that in, both with and without the quotation marks, but it just sits there doing nothing!
It took about 4 rounds of faxes to find out he actually typed in '<enter>' and not the 'Enter' key.
I was just glad there were no 'Press any key to continue' dialogs.
A long time ago a client's manufacturer said my circuit didn't work. I told them it did, because the samples I'd sent them worked and so did all the others I'd made.
A 12 hour flight to Hong Kong plus a 3 hour bus ride to Shenzhen ensued. In the meeting as soon as I got there I asked them to show me the thing that didn't work and realised they'd ignored the layout I'd sent and redesigned it for some unknown reason. Since it relied on photodiodes which they'd not out under the lenses in the right place, it wasn't going to detect anything. So I put that aside and told them to produce it like the sample. Job done, but it took the best part of a week.
There were a few eye-openers a long the way...
The karaoke bar that was really a pick-up venue for the local ladies of negotiable virtue (I didn't but the client's bloke certainly did), the cyclists wobbling along against the traffic the wrong way up a dual lcarriageway with half a ton of wood and a child on board, a crew of builders opposite the hotel making a new tower block when another crew were taking one down right next door, a load of blokes with diggers levelling off an entire hill so they could put a new factory on it, impromptu paddy fields on unused ground between factories.
A lot of the workforce had small flats on site, but it was only to dissuade them from swapping jobs to the opposition at lunchtime. It was so bad that they only got half their wages before Chinese New Year and the rest when they returned, but loads of then still defected.
A different world.
I sent customer in India, some code to solve a problem. They compiled it, and ran it, and it didnt work.
Strange thought I... so I did more testing. Lots of to-ing and fro-ing.
Eventually I asked them to send me the listing of the program they ran. About 20% was my program - they had altered it, and added lots of stuff, and it was the stuff they added which caused the problems.
I learned a lesson - always check what they are running, and avoid sending source code - send executables instead.
This whole story is basically a version of when we turn away from one machine (patient) assuming it can get on with progressing some kind of repair based progress bar only to come back after the estimated time to see that it popped up a "are you sure" or other stupid question.
The absolute WORST example ive seen recently is the "log out" button on banking websites.
Whats the greater risk?
a) The user has pressed log out and that wasnt what they wanted to do , neccessitating an "are you sure?"
b) The customer presses log out and fucks off , leaving the machine unattended in the cyber cafe for the hovering shoulder surfer to pounce on a subbornly NOT logged out Machine!
it beggers belief.
i cant imaginr how much money is being stolen this way.
I'm looking at YOU Santander!
Frankly anyone doing their internet banking at a cyber cafe is putting themselves at significant risk even if their bank does log their session out immediately. From what I've heard the incidence of keyloggers etc (known to the operators or not) at those sorts of venues is rather high.
"Cyber cafes stopped being a thing ~20 years ago."
In your jurisdiction, perhaps.
I know it's hard to believe, but the world is much larger and far more varied than the view you get out of YerDearOldMum's basement windows. Perhaps travel a bit? Further than Hull would help.
“Travel is fatal to prejudice, bigotry, and narrow-mindedness, and many of our people need it sorely on these accounts. Broad, wholesome, charitable views of men and things cannot be acquired by vegetating in one little corner of the earth all one's lifetime.” —Mark Twain, Innocents Abroad
My Dad had me read it when I was pre-teen, just before we started traveling as a family between home base (Palo Alto) and Blighty (Yorkshire) on a regular basis. It has stuck with me for what should be obvious reasons, although I'm afraid many of the people reading this are entirely too insular to understand why.
Because "the WiFi isn't working" and the customer wants it fixed, but doesn't want to pay for a hotel for the night. They'd tried troubleshooting over the phone for a few days (nobody had thought to do a remote session), and it needed fixing now.
WiFi wasn't broken, but the consumer ISP had limited the DHCP scope to five IPs and I couldn't convince them to increase the scope as that was a cost option (more expensive plan), nor did asking for the router password work either. So, I spent the afternoon figuring out how to buy an access point in central Amsterdam (I speak no Dutch, nor did my host), but once a consumer grade AP had been purchased and configured to serve WiFi (and DHCP, passing the traffic on to the ISP supplied router, but only consuming a single IP), all was well and I was driven back to the airport at breakneck speeds to catch my flight back with five minutes to spare.
But I did a stint as an on-site hardware support rep once upon a time at a university campus. Got a ticket about a computer making a beeping noise in the library. I head down to find someone had set a book where it was pressing down on the space bar so when they booted the computer up in the morning, the BIOS complained about a stuck key. The person who submitted the ticket was at least appropriately grateful and embarrassed when they realized how it was both their fault and a ridiculously simple solution.
I once flew 250 miles (each way) to flip a single DIP switch.
We had written some test driver software that used a serial port for communications with the system under test. The software was successfully sending test command to the target system, but would not read the responses.
So I loaded up my briefcase with null modems, a breakout box, screwdrivers, and any other RS-232 diagnostic equipment I could think of, and hopped on a plane. Got to the customer's site, saw what was happening, and started to check configuration.
This was back in the days when serial ports were configured by DIP switches, and the computer had two serial ports... BOTH configured as COM1. Once I flipped the switch to set one of them to COM2, the software magically started working again.
We had a high-profile customer making a transition from their own internal world-wide network to a more commercial T-carrier based system. The company I worked for supplied the necessary gear to interface between $TELCO and their own equipment. This wasn't pre-Internet, but it was before the general use of the Internet for routing internal traffic around the world, so all their WAN links were supplied by one $TELCO or another. Call it mid 1980s.
Our customer service got a phone call from the customer allowing as to how one of their offices in Sydney, Australia refused to see the rest of the world. Customer service called me (the primary TAC Engineer for the customer), and I eyeballed it. Digging into the network, I could see that the loopback switch in the Sydney office was thrown, it would need to be flipped back to connect their LAN to their WAN. I informed customer service, and figured that was the end of it. Until about two hours later when my Boss wandered in and asked what I knew about Sydney being down. I blinked three or four times to reboot (kernel hacking again, probably) and told him I located the cause and informed our guys as to the fix, and then got on with putting out fires elsewhere.
He replied that apparently the dude in charge of the Sydney office didn't like the answer, had called his Boss, who called his Boss, who called the director of the Australian branch, who called the owner of our company, who was informed by our CS guys that I was responsible, and so now my Boss had been put in charge of fixing it. He wasn't happy. So using my TAC access, I showed him the "fault". He expressed disbelief. And called the owner down to my office. The owner (the engineer who founded the company, and a real tech, not just a suit) also expressed disbelief. I believe his actual words were "What the fuck are those useless fucks doing?" ...
ANYway, he called the director of the Australian branch (just a suit, apparently), who got all shouty and demanded an immediate fix, now, or we'd lose the entire contract if he had anything to say about it. Our owner tried to calm him down, but the dude wasn't having any of it ... so to make a long(er) story short(er), he promised to "put his best man on it" ... and I got sent to Sydney on the next flight. Out of San Francisco. First Class. At very short notice. To flip a switch. With invoice in hand, to be presented personally to the Director in Oz. I honestly thought he was going to take a swing at me when he read it ... it was a tick over $20,000 ... in mid-1980s dollars. Broken down in glorious detail. To flip a switch.
But wait ... it gets better! When I was at the airport heading home (having been in Australia for maybe 2 hours total), I got called to the proverbial White Courtesy Phone[0]. Seems a different branch of the very same company had a similar problem, this time in a satellite office outside Boca Raton, Florida. I called my Boss and asked something like "WTF‽‽‽". He tiredly allowed as to how he personally had checked, and indeed it was the exact same issue. Our owner had asked if I wouldn't mind doing the hono(u)rs ... a completely different Director had called and threatened him in a similar manner to the first. So instead of flying East and home six hours later, I had to take an immediate West-bound flight, changing planes in Jakarta and London, to Florida. Arriving somewhat cranky & disheveled in Boca, I was rather pleased with the similar result (one switch, and out!), except I didn't feel physically threatened after the Director read the invoice. He went white and slumped in his chair. I excused myself.
Back to the airport, and home to California. Still First Class. Four+ days on the road, literally once around the world, no hotel rooms, not a single proper meal, showering in airports, just to flip two switches. Such was the life of a field engineer. Tell that to kids these days ...
.
[0] This was in the days before the ubiquitous cell phone, and I had left my DynaTAC at home ... it not only wouldn't have worked in Oz, it probably would have been confiscated at the airport ...
1961, an RAF Thor missile site in Yorkshire (https://en.wikipedia.org/wiki/Project_Emily) had a USAF supplied, RAF manned IBM data transceiver (https://www.computerhistory.org/collections/catalog/102645476) used to receive/send 80 column punched cards as part of the USAF high-tech spare part supply chain for Thor missils. One day the data transceiver would punch received cards but failed to print the data at the top of the card. The on-call IBM engineer drove from Alconbury to Yorkshire, walked into the room and flipped the "print" switch from "off" to "on" then drove back to Alconbury
I was scheduled to fly from London to Milan to help setup a new office when we had some issues with a Sun Server in our Frankfurt office not responding.
After spending several hours on the phone to the desktop support person in the office including getting them to power cycle the server and not getting anywhere, we decided that I would fly to Frankfurt, deal with that issue and then head onto Milan.
Got to Frankfurt and into the office to find that the keyboard for the Sun was unplugged - if anyone remembers those early Sun boxes, they did not like to come up without a keyboard.
Plugged the keyboard in, rebooted the server and back to the airport - total time in the office about 15 mins.
Did a 300 mile round trip from Mid-Wales to Milton Keynes and back to plug a set of speakers into the correct port of a MD's PC - in the days before they started colour coding audio ports on motherboards/soundcards... All because the MD was having a 'bad day' and my Boss asking if she would be prepared to check cables, the 4H response time for a critical issue was stated - just so she could listen to some pre-recorded webinar.
20 years ago…a colleague planned an Exchange database defrag process (as I recall it was), from the command line, first time the process had been performed and was deemd a Saturday onsite OT task. Diligent research had been done, which stated the process could take a few hours to complete. The command was duly typed which produced an acceptable response, and after a little while he stepped out to do some shopping. Came back a couple of hours later, to a blink cursor with the prompt ‘Press Enter to continue…’
Had fun (not!) recently trying to upgrade the IOS on a poxy little Cisco router. First it failed to recognise my USB stick, a known problem with these, normally just needing a reboot but not today!
(Cisco suggest using another USB stick... but I used the same stick on dozens before AND AFTER!)
I finally went through the upgrade and the endless 'are you sure? y/n', 'are you really sure? y/n', 'would you like to change your mind? y/n/ffs' and decided on a bite of lunch while it was rebooting.
I came back an hour later to find it stuck at a pointless 'accept EULA y/n' prompt! (pointless because the licenses were already installed)
Back in the day, working on a super secret project, to send a secure fax to the other party in another country, took six weeks. Yes, weeks. Embassy to embassy. Also, if you sent multiple faxes every few days, they may not arrive in the order they were sent, which caused much repeated questions and confused answers. So we ended up implementing a manual HDLC protocol with two numbers at the top of each fax: The number of this fax and the number of the last one received. Once I flew over for a meeting: It took 36 hours to get there, had a very groggy 2 hour meeting and then travelled 36 hours back. Phun times…
<unnamed> were building a global CDN for <other unnamned>. We assembled and tested racks of compaqs, then had them crated and shipped from the UK to site. Each one had a simple diagram of the "which plug goes where" type.
Rack arrives in New York. Customer's local hands email "it's plumbed in and running". Cue a ping. Nothing. We tried everything.
Customer (at the next desk) said "must be your hardware, you'd better fix it".
The response was "OK, we'll send someone. But if it's your fault you pay".
Newest recruit (six weeks in) got a ticket to New York. He landed, got a cab to the site, found the rack. Looked at the back, moved an RJ45 one hole sideways, called us. "Yep, it's all working now!"
Cab back to the airport, flight back home. He was on the ground about 3 hours.
Customer received an invoice--.
OK, long time ago, they all are?
Frantic, Sunday, phone call from the south coast, "It goes live on Monday morning and it keeps crashing"
We have tried everyone else and we have tried everyone else, there is no one who can help.
It was a PABX with twin, fully redundant, hot swap, processors, they kept swapping before they could get any useful data configured. All they told me was it kept crashing.
(I am guessing you know my Sunday fix already)
They must have been desperate, I was the technical trainer and, at that date, unproven to the field engineers. Can't do it become a trainer, can’t train, become a manager.
So, as a non overtime grade I drove from Luton down to the south coast where I was witness to all those Kiss Me Quick hats wearing holiday makers, me tools in hand.
I got to site, where the errant box was, no phones on that site as there were no externa lines connected yet, and a 20 minute drive to the main site that had service and possible support. You could feel their pain.
I asked them to show me the problem, sure enough it crashed and the hot standby took over but as they had been unable, in the time it was running, to save the database, it started empty.
I watched this a couple of times, turned off the standby processor and there it sat; unhappy, sulking a bit but up and running.
Silence, and joy; then off they went programming and saving until they were happy it would do the job. Shed loads of “WTF” alarms but stable.
Proper fix came late Monday.
Me, no glory from the mangers who said I should not have gone, true but unkind. The engineers, totally my best ever buddies. Really helped me in my job as they knew I could actually “Do It”.
Not being buried in the situation as they felt they were, I was able to see the wood for the trees.
Sometimes you just get lucky.
I certainly know those kinds of daytrips.
Most actual one was to drive around 350 kilometers (one way) to northern germany to adjust a limit switch by about 3 millimeters.
Prelude is like "Customer sent a video, look at it.The emergency switch has kicked in, they must use the maintenance override to move the machine back."
- (After looking at the video)"No they just have to adjust the one and only movement imiting switch a little bit in that direction. BTW, that machine has no emergency switch safe the e-stop button on the user panel."
- "Impossible, you have programmed that thing so you must go there. That switch is an absolute end switch"
(Spoiler: of course a simple initiator is not any kind of "absolute end switch", "E-Stop" or anything remotely security related. We are talking off the shelf cheap 6 euro one channel switches, not even NAMUR sensorics (which the built in cheap PLC could not status check anyway)... But OK, i am dumb and that manager with his extensive doctorate in "i know how to turn on my coffee machine" is a specialist...)
End result: going there, adjusting that limit switch by the aforementioned 3 mm, confirming issue gone,going back. Time on site including CoViD rituals and a small excursion to the porcelain throne under half an hour.
Same with other excursions. No matter what you say, no matter wether you are the guy that build and/or programmed the machine, it does not even matter if you were deeply involved in a certain machine since its prototype days and have more than a decade experience with it, some incompetent but sadly "managing" individual will send you out, not listening at all to you, not believing you, sometimes even paying external personnel to verify a diagnosis that you made prior to driving on site.
And do not expect them to EVER learn or admit that they maybe were a bit wrong, no matter how long your uninterrupted list of "i told you so" incidents may grow...
I learned. I calmly explain to them why the actual excursion is not necessary.
Once.
If they insist, thats a nice day on the road far out of reach of those (use insult of your choice).
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
