Client tells techie: You're not leaving the country until this printer is working Welcome yet again to On Call, the reader-contributed column in which The Register immortalizes readers' stories of escaping tech support traumas. This week, meet a reader we'll Regomize as "Tom" who told us about the time he was despatched from his UK home to an African nation, where his client operated both a mining company …
Not quite at the same level, but in the 90's I was doing work for a company that was in the process of building a satellite communications network (it never saw the light of day - they ran out of money long before they were ready to launch their first satellite). My responsibility was with a part of the ground network system. One week I (plus my boss in that company and few other people) flew out to Japan to visit the ground segment prime contractor; it was a sort of multi-function visit, with my job being to witness and sign-off factory acceptance tests for the part of the ground segment that I was responsible for. This was scheduled as a two-day job, with me returning to the UK at the end of it. The first day started OK and the morning tests proceeded fine. After a break for lunch, testing resumed and things went distinctly runny - a test aimed at proving some critical functionality failed completely and pretty spectacularly . Nothing I could do; I just stepped back and watched quietly while the prime contractor's minions rushed about trying to work out what was wrong, and management bodies panicked.
I reported this to my boss in the evening, only to be told that I would have to stay in Japan as long as it took; we had to sign things off to keep to the schedule, and if that inconvenienced me then so be it. I have to admit, I was not a happy bunny at that prospect!
The next morning, I rolled in to he prime contractor's sight to be greeted with smiling minions and management - they had found the problem at 2:00am (it turned out to be a small error with big consequence - they had not purged an old configuration before starting the test, which meant that the equipment was in a very odd state). They thanked me for me patience and understanding the previous day (nice to be appreciated) and we cracked on with the testing. We managed to get it finished in time for me to get to the airport and catch my scheduled flight home, with me filling in my paperwork on en route and faxing it to them the next working day.
Was a consulting gig in Dearborn in 2003, where I almost was stranded in the huge east coast blackout. I typically took the Thursday 4:30 flight out of Detroit each week, and I showed up at the airport without incident, at my gate ready for normal boarding to begin which typically happened around 3:45. Saw a notice that the flight was delayed by 30 minutes. Not a big deal, that sort of thing happened all the time but since it was a direct flight home and terminal flight on that route for the day it didn't matter to anyone's schedule so none of us were concerned. I went up to a little cafe above the gate to grab a bagel to pass the time. Then the power went out. OK a little weird since its an airport, but it was mid August so wasn't something I saw as too much of a red flag nor did others in the vicinity.
A couple of us went down to the gate to inquire if the power outage would prevent departure. I figured the plane is sitting right there and surely they have backup power for ATC so they could probably make it work. Sorry, she says, "if the plane had pushed off from the gate then it would be able to leave, but we can't release one at the gate until the power returns". After about an hour she said she didn't think they'd fly because even if the power comes back schedules will be pretty snarled and smaller flights like ours are likely to be the first to be canceled. Oh well, I figure I'll have to go rent a car to go back to my corporate apartment.
In the main airport near the rental car location it was pure chaos. They had to do everything by hand, didn't know what cars they had, etc. but they were pretty sure there wouldn't be enough for the people already in line, let alone those of us just coming up. Cell service was impossible too, so there was no one to call. I came upon a few other people in similar circumstances who were talking about getting a limo to take them back to Chicago, which they knew still had power. I figured that would work for me, I could rent a car from there to get the rest of the way. It took some doing, and cost us I believe $200 each for four people, but we set out at around 6:30 for O'Hare (where the other people's cars were) and it was eerie drive for the first 45 minutes not seeing any lights anywhere.
Finally we saw civilization, got to O'Hare, I rented a car, drove back home then another 30 miles beyond it to the airport where MY car was parked, and finally got home around 1:30am my time (2:30 am Detroit time) I could have saved an hour by waiting to get my car the next day, but I just wanted to get the whole thing over with. I really dodged a bullet finding that limo, otherwise I guess I would have been hanging out at the airport until power returned the following day! If it wasn't for that brief delay I would have been on the runway somewhere when the power went out, and had a much easier trip home!
Wasn’t previously aware of (am right-pondian, didn’t remember) this blackout, which has its own Wikipedia entry - https://en.m.wikipedia.org/wiki/Northeast_blackout_of_2003 according to which:
The blackout's proximate cause was a software bug in the alarm system at the control room of FirstEnergy, which rendered operators unaware of the need to redistribute load after overloaded transmission lines drooped into foliage.
Thanks for the story!
Coincidences...
1. The anniversary of that outage (August 14) was just a couple days ago.
2. I was in grad school (not inside the affected area) and about to take the high-level undergraduate Power Systems course at the time, followed by the Advanced Power Systems, hoping to get a utility job.
3. I didn't get hired by a utility, but did get a job in Metro Detroit -- moved there in March 2005 with no plans to leave.
4. My wife -- obviously before we ever met; she grew up the area -- was working as an EMT through that blackout and has quite the horror story about their ordeal, particularly with regards to the potential dangers of nearby heavy industries (the Marathon refinery and Zug Island steel mill). The worst-case scenario did not occur, but could have, making an argument that reliable electric power saves lives!
The *proximate* causes of the blackout, as detailed in the official report on it were:
1) The power company had fired anyone with any real knowledge of managing power infrastructure and replaced them with cheaper computer-literate staff.
2) Computers had frozen on "nothing bad happening" screens because too many real-time problems were being received and no-one realized that.
3) because of point 2, no-one believed the field reports being phoned in of shorting lines.
4) Said lines were shorting to ground because they were heavily overloaded and the proper tree-trimming had not taken place (see point 1). Sagging lines arced to tree limbs causing line breakage and fires.
5) The overloads were happening because the power company had 6 "resistive load" type power generating stations and only one "reactive" load station (with the required I/V phase corrective hardware), which was down for maintenance during the heaviest reactive load season (summer, a/c etc) because, well point 1. The current in the transmission wires was, as a result, way higher than spec.
6) When the computer staff finally got a clue, they didn't get a proper clue and rebooted the computers, which helpfully re-instated the message queues that jammed them again. Time was wasted because this was seen as a computer problem, not a power distribution disaster in the making. See point 1.
7) The people in charge of mitigating the situation (more experienced staff would have started rolling "brown outs" to decrease loads, but see point 1) had an actual, honest to Cray playbook. It was set out in "Failover Class" order, a class one being that one power station was down, a class two that two stations were down and so forth. The staff were working the book from the class 0 situation, failing to recognize that because the "reactive" power station was down they were already in a class 1 failover. The mitigation script they were following (blindly) was the wrong script.
When the power went out, the grid momentarily reversed "sense" before throwing open all the breakers everywhere all at once, which caused a false assumption the problem was originating in Canada when in fact the problem was entirely localized in Ohio.
All in all a self-fulfilling prophecy of cheap-out economics, and a wonderful fiasco if you weren't unfortunate enough to be caught in it in, say, Manhattan on a hot August night.
That blackout... I was doing a projector installation in a northern Ontario town a 10 hour drive from home (Toronto). Where my wife was in charge of our 1 year old girl. I had finished the setup, and had to run a test show (35mm film projector with a 4kW xenon lamp). I pushed the lamp on button... and all the lights went out. I found the building power was out, then that it was the entire street, then that a lot of the eastern US and Canada was out. At least the car radio picked up some news.
One gas station was open in town with a generator but the drive home was at least two fillups. No way to know how long the outage would last or if fuel was available en route.
Wife really enjoyed the dark sky in Toronto, it's impossible to see any but the brightest stars with all the street lighting. I wish I had been there for the blackout, it was apparently a fun time with free ice cream and overall a friendly experience with everybody out
One frustration was that an industrial property visible from town had its own power station on the river and was still lit up and running but there was no way to pass power to the town.
The original failure was due to a cascade of transmission lines overloading - load from one overloaded line kept being passed to another line which shut down on overload and its load passed on ... and so on. The problem after that, with the grid basically disconnected everywhere, was that a generating station needs grid power for all the controls, pumps, fans, and all in order to start, and needs a live grid to sync up to, and finally connect to. Plus the intervening switchgear operates on power from its grid connections - unfortunately all without power. I don't know how they got one GS running but then each switchyard and GS had to be connected in stages with workers manually closing switchgear to get the electric switching system going. This had never happened before.
Quebec was the only area unaffected in the overall blackout area - their semi isolated backbone distribution is via high voltage DC, with inverter systems feeding local distribution and at borders. The border connections just shut down when the outside load went crazy and they kept their lights on.
Something similar took out a big chunk of Europe.
https://en.wikipedia.org/wiki/2006_European_blackout
My mother's first comment? Hey, Rick! What did you do?
Gee... Thanks. ;)
During that blackout I was several hundred miles north of you - backpacking the mountains of Michigan's western upper peninsula with my brand-new bride. No cell signal at that place and time of course.
Got back to some sort of civilization and all the locals could talk about was how badly the trolls (lower Michigan - "under the bridge") had messed up the power!
By the time I was able to contact my boss, all was well. At least for us.
Many years ago while working for a small MSP we had the support contract for semi-sub drilling rig operator.
I was put through my BOSIET offshore survival course, and sent off on many adventures aroung the North Sea, North Atlantic and the rest of the world where sea/ocean oilfields were to be found.
One suche instance after an hours helicopter flight in a westerly direction from Shetland I was on a very very large semi sub in the very wild waters of the north atlantic. I finnished my task of updateing their mail server on board, and replacing the fileserver. in 2 out of the 4 days allocated.so had 2 days to kill on board before my flight slot back came round. I was allocated the so called "Company Man" accomodation, Which is the largest most lxurious cabin on board. (luxurious being relative) the bed as still narrower and sorter than asingle bed. and im around 2m tall.) On day 2 the wind picked up to the point that waves were breaking over the top of the accomodation module multiple 10s meteres above the surface of the ocean, and fhelicopter flights were canceled.
So flights canceld for 12 days, when flights started again then obviously the crew who had done nearly 5 weeks on instead of 3 weeks and all those who had been longer than me went first. so in total my 4 day trip to the north atlantic took 18 days. Got paid daily offshore rates for all 18 plus 3 (more if I was up at midnight) really good meals per day, snacks juice etc. Plus the oppertunity to sort out every staff mmebrs personal laptops to make sure were working great.
Not as many nights (well, only 1, in fact), but I had to hop from Montrose to Arbroath platforms for a day to inspect some pumps, no kit with me at all. Chopper got diverted and I had to stay overnight on Arbroath, which meant in fact on the jack-up drilling rig over it. An old Russian thing ISTR. The one good thing I do remember was that, as someone also 2.00 m tall, the bunk I had was not only "long enough" but had space past the end of the mattress! Similarly with a couple of platforms in the Norwegian sector.
I very nearly didn't manage to get out of the US on one occasion, but due to weather.
I'd been in some meetings with colleagues in Kalamazoo*, Michigan, and was booked back to Chicago on an early afternoon flight to connect with the red eye to the UK. As I had to vacate my hotel room I went to the airport early to see if I could get an earlier flight, thinking there'd be a lot more to occupy me at O'Hare than at a tiny one terminal place. When I made the enquiry I was told, "No problem" and was informed the weather was closing in so it was fortunate I was there as the later flight would almost certainly be cancelled.
The landing at O'Hare was, to say the least, hairy as the plane was swinging some distance either side of the flight line on approach but all credit to the pilot as he touched down smoothly right in the middle of the runway. It was clear that getting away back across the pond was by no means a certainty - there were calls for passengers to check in ASAP then a very early gate call as they tried to round everyone up. Even having boarded we sat for 20 minutes or so while they de-iced and I believe it was about the last plane out before the airport closed for the night.
It wouldn't have been a total disaster if I had got stuck but even one extra night in a hotel that had no restaurant, and not many options close by, would have been a pain.
*No, I didn't know a girl there (for the oldies on the forum)
I've been trapped at O'Hare by weather.
One time, I was flying back home from a business trip, to the Lansing area through O'Hare, and my wife was flying back the same day from a conference on the other side of the country, but also through O'Hare. We both got to the airport as they were cancelling all outbound traffic due to a snowstorm.
I called the company travel agent, and they found me a room at the Sofitel. I looked around for my wife, but this was before we had mobile phones and finding someone in O'Hare is not trivial, and I didn't know whether she'd made it in, or if she had and had made it out; I didn't have her flight details. (We both flew so often in those days that we'd gotten rather lax about sharing itineraries.)
So I took the shuttle to the Sofitel and spent the night in a very nice room on the 20-somethingth floor. Had a lovely view of the Chicago skyline at night with snow pouring down on it. No luggage, but I had complimentary toiletries so at least I could brush my teeth and whatnot, and room service supplied my dinner.
Later that night, when my wife called me at the hotel — I'd left a message on our home answering machine — I found out she had caught the last flight out of O'Hare to Grand Rapids, where she'd rented a car and driven home through a major snowstorm. Fortunately she grew up in Indiana and regarded driving in heavy snow as a mild inconvenience. So she had left the airport only a few minutes after I'd gotten there anyway.
Still, it's one of the few times I can remember when it would have actually been nice to have mobile phones; we could have shared that room. Though I wonder what the company would have thought when they saw the receipt was for two adults. "So, you say you just happened to meet up with your wife when you were stranded away from home? Yes, of course we believe you."
Once, I got stuck on a Friday afternoon in Munich. A lot of snow almost shut down the airport; I was very lucky that I was able to catch a jump seat on probably the last plane to make it to my destination that day. I was working for the IT arm of an airline at the time, meaning I had the security clearance to fly on the spare seat in the cockpit. I learned a lot about airport procedures that day, plus quite some vocabulary that my fairly prim mum would had disapproved of.
A senior colleague had to visit a very large bank in Europe to resolve a performance problem. We recognised this as a user configuration problem - however they wanted us to change our product because it would be a big change to their environment to fix it.
To get into the bank you had to hand over your passport to get a badge for their building. My colleague had his passport on him, but said he didnt, would a library card with his photo on it be OK? He went to the meeting with the senior people in the bank who told him you will not get your passport back until the problem is resolved!
He worked a long week, and managed to persuade them it was their problem. They wanted him to work the weekend, but he said he had a personal commitment he >had< to go to (10 year wedding anniversary) and he had some very expensive tickets for the opera.
He went back on Tuesday, and they realised that if they wanted the problem fixed in the next 6 months, - they would have to change their configuration. He stayed the rest of the week - and came home - exhausted.
"He went to the meeting with the senior people in the bank who told him you will not get your passport back until the problem is resolved!"
Anywhere in "civilised" europe such a statement would crumble in the face of the local police getting involved
This post has been deleted by its author
Well, this is the fun of On Call... making it a guessing game (sometimes an obvious one). ;-)
The fact a vintage 707 was involved might be a clue too... it's not a Southern African nation (none of the nations south of Angola/Zambia still had vintage 707s flying in the last two decades, not even basket case Zimbabwe). I call DRC (formerly known as Zaire)...
;-)
I had a contract to write some custom reporting S/W for the vendor of a process control system to be installed on-site. My client as in Crawley but the site was near Naples. The S/W was completed ahead of time and the installation was quick as well. However, running the reports kept resulting in system crashes and files containing what appeared to be segments of memory appearing in /lost+found. I couldn't get away until they had a clear run. Not only was I running short of Lira, on the Thursday I got a call from an agency to visit a prospective client on Monday with a view to starting a new contract on Tuesday. Fortunately we had a clear run on the Friday and I got out PDQ except that storms at Gatwick, where my car was, meant an unscheduled stop to hang about in Paris CDG and a diversion to LHR.
I heard afterwards that it was a faulty memory module; presumably the extra code tipped the system into using enough extra memory to reach that module - either that or they'd had previous problems they were keeping quiet about.
My dad was a KC-135 crewchief in the '60s. They had issues with the dust and sand of North Africa (Wheelus AB outside of Tripoli).
I was a KC-135a crew chief in the '80s. Only time I few on my plane, (Washington state to Alaska) we had an inflight emergency where half the hydraulic fluid went away. Co-pilot hit the crossover valve and made the remaining hyraulic fluid go away. Ended up circling Eielson AFB, Alaska many times while dumping fuel (take that you nasty mosquitos!) and then we hand cranked the landing gear down. Only problem; could not get the gear to lock in place by hand. Pilot started flipping the plane back and forth until both sets locked. We landed to a mess of fire trucks, waiting to put out the expected *BOOM*. Insepction found a metal hydraulic line was not clamped in place properly, after the last phase inspection. On shorter flights, it did not cause any issues but the extended flight north caused the line to chafe against a metal bulkhead and finally fail. It's my understanding that back in Washington state, the phase mechanic who failed to replace the clamp ended up beaten up one night. Strange.
So yeah, I wouldn't want to fly in a '50s era plane but the USAF still has a bunch in service: https://www.flightglobal.com/fixed-wing/us-congress-halts-kc-135-replacement-until-usaf-produces-stealth-tanker-acquisition-plan/156264.article
The only reason B52s are flying is because they're flying (Circular logic, I know)
Their original mission is long since expired, other large bomber replacements (NOT B1/B2) were vetoed because there was no mission to give them and Rapid Dragon means that if they were all grounded on Tuesday the mission would continue by the end of the week on other aircraft, probably for lower costs
There are only 2 reasons they continue to fly. Propaganda and Congressional pork
Boeing 707, 747, never liked the 727) and the earlier 737 models were quite OK. Problem is that they assumed that they cash in on those 2 models (707 and 727 are out of action/production for a really long time) as long as they could, and when they had to stop producing the 747 due to waning demands, the 737 alone just couldn't provide for shareholder satisfaction. 757 was kind of a bust, 767 and 777 were mediocre at best, and the 787 another bust. And then they tried to mess with the 737(max) in order to try and keep the same type certificate rather than develop a competitive model in a market dominated by Airbus and they even would be hard pressed to compete with Bombardier and Embraer. That's when the proverbial soft brown matter hit that fast rotating appliance.
If it's Boeing, I ain't going! (even recent astronauts are chiming in on that one now) >:)
I had to travel to South Africa to work on a customer system. It should have been just a couple of days, but ended up being a long week of long nights. I had to use the local office's cash float to go and buy some spare clothes. Part way through the customer asked my for my plane ticket (this was so long ago it was one of those old, grey and red flimsy things, nothing onlne) so they could "check my flight" for me, unthinkingly I handed it over. At the end of the gig I did get my ticket back. At the airport I handed over the ticket at checkin and they asked me what address I was going to back to in the UK. I queried this and they said that it was part of my ticket. The customer had been so happy with my work that they had upgraded me to Virgin first class, which comes with a limo service at the other end.
An old boss of mine had been doing some long-term contracting work for the police service (that bit will become important later) in a country which shall remain nameless but let's say that their criminal justice system isn't all that it could be.
He took a weekend home, and on his return was promptly arrested at immigration and serious criminal charges were thrown his way. Despite protestations of innocence, he found himself being processed through the system, even ending up in a jail cell with some very dubious company. By sheer stroke of luck he happened to be in a police facility when one of the senior officers came through and saw him, and came over to enquire as to why one of his top contractors was on the wrong side of the fence.
It turned out to have been a case of mistaken identify, but thanks to the way that country's judicial system works, that would probably not have done much to help him. Were it not for the timely encounter with his boss, who was able to get everything sorted out and secure the poor guy's freedom, he could well have ended up serving many years in a very nasty prison.
A friend went to [Redacted] with a team to set up their first electronic stock market for which his company provided all the software and hardware. When he got there, all servers and assorted networking gear were already waiting to be cleared by customs for several weeks, and for 3 more weeks he was there daily, every day a different problem arising - he and the customs' officials seemed to be unable to reach any king of understanding on how (much) to clear it.
Until one day an armed party burst into customs and took the crates by force. My friend only knew it wasn't a robbery when he got a call from a colleague telling the gear had arrived at the future stock exchange's facilities, brought by an army general very close to power... someone eventually got fed up with all the waiting.
A week later, after a successful installation, he was then able to finally return home.
A few years back, I was on a (non-IT) mission to visit some ports in northern African countries. On immigration, the queues were long and slow moving. When it was our turn, the guy stared at my passport and the plethora of visas therein and told me to wait. And wait. And wait...
Then my phone rang. With an apologetic smile, I took the call. The voice at the other end asked me where I was, as they were waiting to pick us up. I explained the situation, they asked to please hand the phone to the immigration officer - whose demeanor very suddenly changed. Within seconds, we had our passports stamped and were ushered through where the secretary of state for transport had been (less and less) patiently waiting for us. The drive to the ministry escorted by police on motorcycles with flashing lights felt a bit surreal.
I'm reminded of a visit I made to the Rodeo, a town near Groningen in the Netherlands. I flew from Aberdeen to Amsterdam Schiphol and took a train to Groningen on the Wednesday - a fairly short journey with hand luggage as I was due to fly back to Aberdeen on the Friday. It was a week before Christmas and it snowed. Aberdeen was hit on the Thursday, the Netherlands on the Friday. The taxi from Rosen to Groningen rail station was hairy, and I only just got there in time for the train back to Schiphol. I was booked on the 9pm flight that evening (I habitually booked the late one, in case of delays to my work, and, on most visits there, would be finished in time to switch to an afternoon flight). I arrived around noon, to an airport with queues everywhere. My frequent flyer pass might have meant just a couple hours queuing, but I decided to settle into the lounge until my booked flight was due - seeing as it was one of the few flights on the boards that hadn't been cancelled.
About an hour before boarding its departure time was changed to "cancelled". No flights were leaving, and only a handful of long haul inbounds were landing. By then, camp beds were being brought into the airport for some of the transfer passengers. Time to book a hotel - but all the local ones were now block booked by carriers. I had a colleague living about 30 miles from the airport; a call to him and there was a room awaiting me in a hotel near him. The 30 mile journey was usually straightforward by a direct bus, but not tonight. Even taxi drivers were calling time on taking fares, but I found one who was prepared to take me for just triple the normal fare.
The following morning (Saturday) and I was told I had to turn up at the airport and queue to book a new flight - no phone or online bookings were being taken. I was also told it was unlikely I'd get a flight over that weekend as there was already a backlog. Online and I booked a Eurostar seat from Brussels for the Sunday and set out to get to Brussels. Ironically, the ongoing rail disruption actually meant I was in Brussels by Saturday lunchtime. Big queues there and no chance of moving my own ticket up, so another hotel (but, being a weekend, the Ibis opposite the station had plenty of rooms).
Sunday, and things were beginning to steady there, and I got my train back to Blighty - that last one as the snow had now hit SE England. A B&B near St Pancras saw me through to Monday. Queues at Kings Cross and no chance of a ticket north - but I got myself a berth on that evening's sleeper back to Aberdeen. I picked my car up from the short-term car park Tuesday lunchtime - the charge for the 6 days made my eyes water, but I got home that afternoon.
Did I mention I'd travelled with just hand luggage? That meant, clothing-wise, I had become quite adept at washing clothes in the shower and using hair-dryers.
It was an interesting trip - and the client paid my "slightly" bigger expense claim!
2010 was it?
I was stuck in Schipol 16 hrs on the back end of it, nothing much flying, aircraft anywhere in Europe and legs off schedule,(crews no doubt time exceeded tooo etc) eventually. BA come through with some aircraft there's a 757 going to Heathrow, that'l do. Full load and off we go, I don't think it got above FL10 all the way back followed the Dutch/Belgian coast to Ostende then across the oggin and a direct approach up the Thames estuary.
I've still got the Miffy mug purchased on a whim during those looog hours of boredom. It's much coveted in the office...
This post has been deleted by its author
[edit: changed it before it's picked up by search engines]
In the late 80s I was working for a company that did quality control, and we developed a quality control product for a certain car manufacturer (quite revolutionary, actually, as it removed large swaths of Just In Time quality assurance issues).
After we demonstrated it we left them some test gear, some of which was my personal equipment. When I got back to them after a week to collect it I was told they would not return it until we handed over the finished product - I was effectively blackmailed with my personal equipment.
As we were only a tiny company we had no choice but to comply, but from that moment on I have not touched anything by that company.
Now for the fun bit: I am dealing with regulatory issues of car companies.
Guess why I'm not cutting anyone much slack?
I was on the other side of such a situation. I was the customer.
Major OS upgrade of a HP-UX server. As the clueful customer who had sysadmin background, I asked for three days to do the OS upgrade. HP professional support told me I was ridiculous, it'd be done in 4 hours, maximum. We convened everyone, including the lead "sysadmin" (who actually had no sysadmin background, but I was helpfully training because they were a good problem solver and learned well) to do the upgrade -- only to find out we had a known bad version and we had to abort before we began. The next chance we had was the next week Thursday at noon...
The sysadmin was a professional organist and was playing at a wedding that Saturday, even if they didn't have a job the following Monday. We get together, confirm we have the good OS disks (I think these were DVDs, but I could be off, it was 2-4 disks total). Computer hangs on boot after upgrade. No recovery.
Professional services looks at it, shrugs and says "Wow, we've never seen that before. Good luck with that, good bye!"
We pull out our HP Ignite image we carefully took right before we started, against Professional Service's advice, rebuild the system from that, verify it works right and start figuring out how to recover.
Friday afternoon, the sysadmin flies home on their flight, we haven't figured out how to solve the problem. The sysadmin's boss called another person and said "Here is a one-way ticket to $CITY. You are there to represent the sysadmin team." The person had no sysadmin background, didn't know a lick of HP-UX (or even any close enough Unix to be useful), but they were there and not allowed to leave until the problem was solved.
Meanwhile the application vendor's tech rep and myself (in theory just an application admin) are doing all the technical work of figuring out what could be the problem, what to try next. I'm briefing VP level leadership every four hours on our progress, or lack thereof, so everyone knows exactly how bad the situation is.
Saturday night, 60 hours into our upgrade effort, working 20 hour days, we found the source of the problem -- from the first line tech support at HP. None of the second or third liners knew the problem, but this first line tech support person rattled the solution off immediately, and it fit perfectly the symptoms. We kicked that off and 12 hours later, we had the OS upgraded, the application updated to use the new OS and everything working fine. I'm told the poor sysadmin with the one-way ticket actually did get a ticket home before Tuesday.
Total time start to finish: 72 hours. Exactly the three days I said I wanted for the change.
It was, if I recall correctly, a Christmas Eve several decades ago. I was heading out from Maryland to a destination in South Florida to support the opening of the winter meet of a hippodrome. I was booked on the last flight of the evening. I had with me 9 boxes of plastic admission passes -- like credit cards, but different -- for the meet, which I had picked up at the manufacturer and was transporting as checked luggage. Uncharacteristic of me, I arrived at the airport hours early. The plan was that my SO would drop me and my stuff at the curb, go and park the Microbus while I checked in with the airline, and we would enjoy a relaxed casual meal at one of the airport's restaurants, since I would be away for the actual holiday.
When I got to the check-in counter, the monitor was showing that my intended flight was canceled. The agent confirmed this, "but we can get you on the earlier flight, if you hurry!" This was several decades ago, so no TSA. Also no mobile phone, so no way to contact my SO. They threw my boxes and luggage on the baggage belt, didn't take the time to process the excess-baggage fees. I said "Do me a favor. In a few minutes, a young lady will come here looking to meet me. Would you please explain to her what happened?" With that, I sprinted toward the gate.
The relationship survived.
the time he was despatched from his UK home to an African nation, where his client operated both a mining company and the national airline.
An unstable African nation (pick one) had a certain government agency "forget" to pay their internet bill. This made the ISP (a multinational) rather upset, so they sent out an expendable *cough* junior tech who "looked the part" to do support on site. And also, to get the customer to pay its' (huge) overdue bill.
When I say he looked the part, the criteria was essentially "Joe is black, he's less likely to get shot, so send him". That may or may not have been true, but "less" did not mean "zero", a fact not lost on Joe.
Joe cleaned up a lot of the bad processes and discovered that the nonpaying customer was not only paying them (or not paying them) for internet access, but also VOIP. So, when they refused to pay for the third time, he pulled the plug on the agency, disconnecting all of their phones.
Unknown to Joe, the agency was the parent agency of an "off the books" security agency. He was told this 5 minutes after he pulled the plug, by terrified local co-workers who realized what Joe had done, as they were fleeing the building before the pissed off Black Ops soldiers arrived.
Sure enough, ten minutes later, about 20 Toyota Technicals arrived at the ISP building, and it was soon surrounded by a hundred armed guys with fatigues, AK-47s, and black sunglasses, as the immaculately dressed leader calmly entered the building with a dozen of his crack troops.
They went into the head office, where Joe was sitting behind the desk. The leader lit a cigar, placed his AK-47 on the desk, and calmly said "I am here to ask why you have disconnected all of my troops' telephones".
Oddly enough, the meeting was strangely uneventful, and even cordial, as Joe explained how much was owed, and how, if he couldn't get some of the money owed back, the head office would simply pull the plug and cut off the entire country.
He was successfully able to get about 30% of the bill paid, and also not get shot in the process. That may not register in the company's financials, but Joe certainly appreciated it.
When the money transfer was confirmed, he turned the phones back on, and the soldiers left. So did Joe, the next day, to non-African pastures.
When he returned, he wasn't exactly covered in glory by his bosses. His direct manager wasn't impressed with the 30% payment he had negotiated, but begrudgingly told him that "you earned your ticket home", apparently thinking that was an option the company could choose to ignore rather than a requirement.
Reading the writing on the wall, Joe quit six weeks later. While the execs were not impressed with the 30% payment he had negotiated, it was 30% more than anything else they got in the future.
Was travelling around the USA many years ago and was given the instructions: catch a South-West plane to Manchester; get the bus to Concord; take a taxi to the Best Western; and we'll pick you up in the morning. Flight was fine and I exited the terminal building, virtually into a bus marked for 'Concord'. Got on board and thought "This is too easy". Got off at Concord, into the only taxi and asked to be taken to the Best Western. Taxi driver was a bit perplexed and said: "We don't have a Best Western here". Turns out I was in Concord, New Hampshire and not Concord, Massachusetts! By this time, it was -20C, 10PM, snowing and I had no accommodation. Managed to get a room in a hotel, hired a car the next day and drove down to the correct Concord. I was not the first person who'd made the same mistake.
I was sent to the Coolangatta, QLD for work (year 2007). After the job at Coolangatta, I was meant to fly to Cairns, QLD (via Sydney, NSW) to do another job.
Coolangatta's Gold Coast Airport (OOL) shares the same "radar" as Brisbane Airport (BNE) to the north. And every local know that whenever there is a gray cloud over OOL, all flights are canceled. Well, on the afternoon I was to depart to Cairns, it poured.
The airport and airline staff has been-there-done-that attitude. They all have a role to play. And while waiting for my checkin, I can hear each flight status going from on-time, delayed and then canceled. Police were called in as routine precaution.
When it was my turn, the staff told me that my departure was delayed. When I explained that I was flying out to Cairns because of my job, the police nearby moved closer. However, one of staff at the airport recognized me and he stepped between me and the police, "`tis OK, mate. He's one of us." The police stepped back and smiled back.
The only other flight to Cairns that evening, the airline check-in staff said, was flying out of Brisbane. And the flight leaves in three hours!
Without even calling to ask permission from my boss, I hailed a cab for a two hour (AU$160) cab ride from Coolangatta to Brisbane Airport. While I was on the cab, I got hold of Qantas reservation. Because I was on a corporate travel account, QF reservation immediately re-booked my flight.
I got to BNE just in time and only to be told that the same weather front has delayed my flight by another 90 minutes.
When the flight left, the captain not only went on the PA and apologized for the delay but he instructed the staff to "open the bar"!
Those were the days.
I could have almost written that. I have worked in Congo Brazzaville, Kenya, Nigeria and most recently Sierra Leone. I actually adore Africa but there are times when you end up in situations like this. I have been arrested because my escorts refused to pay a bribe of 40Naira (which at the time there was 150Naira to 1GBP). I once ordered chicken for dinner and after 3 hours of waiting asked them jokingly if they were having difficulty catching it, only for the waiter to admit "yes". I travelled on an internal flight in Nigeria where a forklift had run into the wing on a plane so they had filled it with car body filler. On the way to Kenya I was told that one of the fuel tanks on the plane was leaking so it would have to stop off in Italy to add some fuel.
The one thing I will say about Africa though is that its countries are free; certainly when compared to the UK.
I used to work for Emirates, and got summoned to Dubai to complete the work on a systems integration.
Was due to fly home on the Friday. Thursday morning it wasn't finished, and they said I couldn't go until it was done and that they'd cancel my ticket.
They were joking. I think. Luckily it was done by the evening...
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
