I was called out on Jan 1 to do a software fix.
I was paid 1K.
Took the best part of the morning.
There's a reason why some in IT remember the days of Y2K fondly. To quote a lyric from an erstwhile pop combo of the 80s, it really could be "Money for Nothing" for a lucky few. Welcome to The Register's reader recollections of the era. "Ed" - his name for the purposes of this tale - had joined the exodus from the permie world …
Exactly, there was a lot of hype too but at its core things would have gone wrong if we hadn't done something.
As a BTW, I was working at what sounds like the same company as Ed but left a few months before Y2K. It was quite a good place to work.
The place I joined after had me in to cover Y2K even though they insisted on powering down the servers from 10 until 1am. So, for several hours we just sat around and played Quake while eating Pizza. My boss had at the time suggested I set up an exact mirror of PROD so we could use that if anything went wrong....... until I pointed out what the word "mirror" might imply..... :-)
As for "project fear" I always had the impression that was a better name for UKIP's strategy.....
(so no IT angle)
Anyway, the evening of 31st December I was working on enhanced pay as an agency at a nursing home near Camberley.
My job, since I am a guy, was to carry an ungodly amount of bottled water, tinned tomatoes, and toothpaste (!) from the back of a minibus and up into the loft. When I pointed out to the activities girl (who was dealing with all of this for about a fiver an hour) that it might be hard to do anything useful with the tomatoes if there's no power, she actually peed. Like, right there.
She scurried off, I thought to go change, but she came back half an hour later with boxes of tea, UHT milk, and several large bottles of butane gas. I had no idea what the gas was going to connect to, but since the poor girl was on the verge of a heart attack, I said nothing.
I don't know what the hell she'd been listening to, but she basically passed out in the recreation room at about eleven PM, absolutely convinced that the world was going to end.
And I wouldn't be at all surprised if all that stuff wasn't still up on the loft under a twenty year (my god, has it been that long!?) layer of dust.
>> at a nursing home near Camberley.
> Ah, yes, I remember Admiral... ;-)
Oh God, yes. I worked there after the Logica/CMG merger.
Lots of ex-service types, all stone deaf from standing too near to the guns. Never mind the year 2000, some of them clearly had problems coping with 1900.
The only women were secretaries.
And the office was empty by 5pm every night.
Such an easy life for them, having money thown at them by their mates in the MoD.
it might be hard to do anything useful with the tomatoes if there's no power
Chop up some of the furniture, bonfire out in the back, tomatoes poured into a big pot, and off you go..?
My initial thought was how to open the cans without an electric opener. Like all good scouts, I have a manual opener with me at all times but perhaps this little item was overlooked in the mosh?
At previous employer, on call after 10pm on any night incurred a minimum 1.5 hour callout at weekend rate (double time). The best ones were the calls at 22:05 which involved the ops person saying "we've had alert XXX out of server YYY", me responding "yeah, that can wait until morning", "Ok, thanks" Ka-ching!
I implemented a four hour minimum for on-site visits in (roughly) 1990, a couple years after I went solo. Double on weekends/holidays. A few clients balked at the new rate ... I simply told 'em "Don't call me unless you actually need me".
A new issue arose. Convincing 'em to pay 4 hours (or more) for a one minute visit. The old TV repairman's maxim applied, "I'm not charging you for thumping your telly with a screwdriver. I'm charging you for knowing where and how hard to thump your telly, and for showing up to do it". The explanation seems to have worked ... although about a year and a half ago, a child CEO wondered why I'd need to thump a telly.
Worked in mainframe testing for a bit after graduating from University. The terminals at first were random logic built from ICs on 3-4 boards. To save costs they did not use gold plated edge connectors.
So when they started acting up. The first level "fix" was to give the case a fist thump on top above the cards. That would work most times. But some of the terminals did develop a noticeable dent in the steel top.
I was perfectly happy having nobody to charge call-out rates to (I did spend the next few weeks wrangling things for a client whose beancounters insisted on on using their old, definitely non-Y2K compliant box because they wouldn't risk(!!!) a cut-over to the fully tested new one until they'd completed their year end).
In consequence we saw in the new year, as ever, at my cousin's house, with a view for miles - including what must have about a hundred simultaneous firework displays, near and far. I still can't understand why my daughter and her future husband just stayed indoors in a pub down in the village.
"I still can't understand why my daughter and her future husband just stayed indoors in a pub down in the village."
Cast you mind back to when you first became engaged ... Would you have preferred to have partied down the pub with your mates, or would staying in with your parental unit(s), partying like it was 1999 have been attractive?
Around Y2K the USA got paranoid about people invading from ships. Seemed improbable as there were lots of quicker and more dangerous ways - like the Canadian and Mexican borders but they were paying. Well actually the shipping support company I worked for, sounded a bit like Inchcape cars, was.
Had three months to get it all ok - me and me oppo (Hello Pete) and we bloody well did it. Not helped that the US Coastguard who were administrating it were prioritising US firms first and we came well down the list. Anyway, it were there. Singing and dancing.
And (probably Y2K+1) I got a phone call from the state side scene manager. At 3.45 ish on New Years day. "Panic. Panic. Panic. Oh and definitely Panic". Understandable as demurrage charges could run to tens or hundreds of thousands of dollars a day. (This was just stupid, taking on a risk of $100K plus based on a computer (bog standard PC) running in Essex for a deal worth 50-500$)
And just me and my oppo (Hello Pete) running tier 1-3 support as well as developing the damn thing. I had tried to get it offloaded (no team available), had pointed out we were hired as developers (well doesn't that include support? You wrote it) and told me oppo (Hello Pete) to go off for Christmas and the New Year, and get stotious and I would handle the brown stuff hitting the fan.
Being very soft hearted and actually not blaming the American lass at all, she had been eminently suitable to talk to on the phone, liked Mark Twain, and after we had explained that consistency in her requirements was perhaps the deal breaker (Only three months, shedloads of work, …) I explained I was pissed, very pissed and would deal with it some time later today - only to be told "I don't care how angry you are - we need to fix this!)
After the 2 people separated by a common language misunderstanding I managed to point out that
* the office was closed
* I wasn't a key holder
* I didn't have remote access
* I had been whinging (repeatedly pointing out these issues) for the last 2 months
And when I looked at it the next day it was a simple Y2k+1 issue.
That solved all the first three points above.
Not even a conspiracy. Just an unfortunate set of circumstances.
(This a re-post from another of the Y2K articles, but it is more apposite here. It is worth hunting out the other posting for the attached Y2KY story.)
In 1999 I was a prod support Solaris admin for a Very Large Bank in a Very Small Country. I was on-call on 1/1/00 (1/1/2000 if you prefer) and scheduled to go in to the office for a few hours early in the morning.
I took a phone call at about 02:00 that said things were running very slowly and not to go in at the scheduled time as my bit was going to be late. When I did arrive I found that the pace had picked up and someone else had performed my single task. Everything completed correctly and the world's banking system remained safe.
My (entirely idle) shift was at triple time, which was nice, but the fact that the enhanced call-out fee for spending a few seconds on the phone equated to £30,000/hour is what I usually tell people about that day.
When working in a large multi national IT company, we had tested all of our products, and had a team on site over the new year. Besides the odd customer application problem there were no major problem - except for the coffee machines - they refused to vend. This was cured by someone one phoning home, and having their coffee machine brought in ( illegally because the power cord had not been checked by the on site electrician). Of course having had lots of caffeinated coffee's people could not sleep.
I flew to the US on Jan 1st to provide on site support for one of our worried customers. The biggest problem there was caused by people firing their guns (with bullets) into the air at midnight. Of course what goes up - must come down, and there was a spate of head wounds in the local hospital!
I have some fond memories of the Y2K non-event.
I was asked to be on call to support a customer's systems over the New Year. Normally they didn't have any out of hours support. I was offered 250 quid just to be on call (ten times my usual rate) plus double time + time off in lieu if I did get a call.
I wasn't surprised that I didn't get any calls as that customer was closed for the New Year and none of their systems were being used.
It was two months later that they did get an issue (during office hours) where some of their code didn't recognise 2000 as a leap year and refused to process any transactions dated 29/02/2000.
Also, out of curiosity, I had salvaged one of the old servers that was replaced for Y2K reasons and took it home to see what would happen on 01/01/2000. It kept running, but I found odd dates in some log files, such as the to be expected 01/01/1900, but also 01/01/19100 and 01/01/A0.
For about two years before Y2k I had the thankless task of going through every item in the asset list of the BBC World Service to identify those which contained any sort of computer/microcontroller (i.e. quite a lot). I then had to negotiate with the makers thereof to guarantee that there was no date-sensitive code in there - this was studio stuff, the computers were someone else's headache. I don't think I found anything, but there were a couple of HP UX servers that I also got sidelined onto...
Turned out their OS couldn't handle Y2k. The machines couldn't handle the updated OS that *could* do Y2k. And yes, there was an awful lot of date sensitive stuff in them. So muggins here got pressganged into replacing them, as they were considered critical... only discovered a couple of months before the event, only two of them in the country available, and because of that I even had to transport them on different trucks on different days, just in case...
Spent 31/12/99 on the beach in Rio, with three million Cariocas!
1999 I was at Southampton Uni, I'd sometimes babysit for a neighbour who was a contract COBOL developer. I still recall with awe how cushy he had it - a 2 year contract, getting paid over £1000 a day (ludicrous money for the time), and all he seemed to do was go to the office at 9am, print out several hundred pages of code (on a dot-matrix), then head home to "review it"... Must say, all I ever saw him do with it was dump it on the table.
All of my customers had basically no problems, including one with a very old unix micro (with serial terminals) that was obsolete in the 90s, but still in use in 1999 and after. Only issue I have seen was with Novell netware 3.11, that began showing years as "19100" and so on. That system kept working until 19103, when it was replaced by a Linux-based server. And still the application software on it went on naming some files (that had a date-based name) with "19100" and so on. It's still in use today, in 19119.
You mean 2038, I suppose. Yes, it will probably crash because it is, of course, based on technology from 1995, so I doubt it uses 64 bit for date management. But I hope it will be dead and buried in 2038. Or maybe I will be dead in 2038. Well, I have another 18 years to ignore the problem.
I was a fulltime, 24x7x365 desktop engineering employee in '99 (basically perpetually on call as one of the two guys who built and maintained our standard workstation images). My youngest had just been born the first week in December, but I returned for New Years' week to join the "all hands on deck" festivities with the rest of our few hundred internal IT staff. Confidence was high because the inside crew had already completed fixing the problems with our contractors' Y2K remediations. The contractors had actually done an amazing job going through reams of mostly COBOL, but the home grown code was a minefield for anyone who hadn't spent 20 years caring for and feeding it like our senior devs. We got through it with only a couple of hiccups: an overseas system that didn't get updated because... reasons, and a desktop app that a certain big hardware maker pushed a last minute (OK, last _hour_) update for that contained a regression. Earned my (modest) bonus that morning for fixing that one with a manual (as in over-the-wire to the hidden system drive shares) rollback to the previous version that went out to thousands of machines.
From a post I put up in 2018 - at this point, I was serving my notice ahead of a move to a new employer, partly because this lot refused to take things like Y2K seriously... Icon, because I wasn't officially on-call that night, and had already consumed several of...
I didn't get paid an extra bean. They even chained up the door to the computer centre just in case the heating control system set fire to everything, that's how confident they were. (I did point out that this was the same type of heating control system that covered student halls, which were still occupied by a few overseas students over Christmas)
Just after midnight, I quickly checked the most critical systems, said "Meh, my stuff's fine, everything else can go whistle" and went back to partying like it wasn't 1999 any more.
Some sites turned EVERYTHING off, just in case. Some of them then discovered that leaving stuff powered down but still plugged in won't necessarily save it from a nearby lightning strike. Just when you think you've covered all possibilities in your disaster recovery plan, Mother Nature says "Hold my beer".
an erstwhile pop combo of the 80s
That "pop combo" was the fifth most-successful UK rock band of all time, and the song in question comes from the first album to sell a million copies on CD. Though you did give them an "erstwhile", which (at least in one sense) is something of an acknowledgement.
(They're also really "of the '70s, '80s, and '90s", but I'll let that slide.)
Working for support services of a , at that time, famous Unix systems company. Although we warned customer not to shutdown server and disk systems , there was one customer who exactly did that. And guess once, when the Sun set that day, the system didn't want to boot anymore and customer didn't had a 24/7 support contract. So he called the local support center ( staffed with engineers around the clock, all night no work, backup mobile phones of all local phone companies, nice food, playing Quake over the company network , extra bonus paid ...). Yes , sure he would pay the premium price for an intervention on January 1st. Support Engineer came on site, gave the disk a slight slap and the system started booting without any problems. Guess that must have been the best paid slap in IT history.
There was a large team on duty that night in the Slough support centre including 20 temp workers from India.
Everyone's ears were tuned in at midnight for any Y2K calls. We had one. A scrum round the callee to see what the bug was. 10 mins investigation. The customers product license had expired... at exactly midnight.
Biting the hand that feeds IT © 1998–2021