'Twas the night before Y2K and a grinch stole the IT department's overtime payout Not just the week but The Register's working year has drawn to an end, and that means it's time to wrap things up with the last of our slow-news-week festive editions of On-Call, the column we compile after kind readers send in their stories of support jobs that went south. Today: tales of tech support over the holiday period …
The company where I was working offered a one-off payment of £750 for anyone willing to sacrifice their 'new millennium' celebrations in order to be on call over the midnight of Y2k. This was all agreed early in 1999, even though we'd clearly demonstrated that there was no issue with our PCs or servers, by the simple process of setting the clocks forward to 31st Dec on a range of test systems and watching them rollover.
Then in mid-99 they decided to eliminate the European head office and manage all European ops direct from the New Jersey head office (try to imagine how well that worked out). But those who left still got their £750 on top of a quite generous severance package. Which made for a very happy new year.
This post has been deleted by its author
I was running a station for a broadcaster one Christmas and was going down to my parents for the festive season. There wasn't just me there was engineering cover on call as well but there were non technical things that only I was going to fix. Also as it was Christmas there was a a cover presenter for the late shift who was new to the station. There isn't much of a signal on any mobile network at my parents house so I gave them my parents land line number. I said that if it was a real emergency and only if it was they should call it. I pointed out that my dad would not be pleased to be woken up after midnight. Oddly the only call I ever got was to say "everything is fine" and "Happy Christmas". Apparently my notes were so good that it was hard to go wrong.
A mate who was about a month into his new job at a new company was doing the Christmas period departmental cover. He spotted a serious problem on the 31st December at 17:55 and attempted to escalate it but was unable to reach anyone. In desperation he resorted to the disaster recovery plan looking for phone numbers. He tried to call the Head of Technology but somehow read the wrong phone number. He had in fact dialled the Chief Executive who was already out for the evening and had no idea what the problem that was being explained to him meant. He did however have the personal number of the Head of Technology and passed that on. Fearing a bollocking from the CEO he sent an apology email and received one back saying that he'd done the right thing. Oh and thank god it hadn't been any later as the CEO might not have been of sound mind and body much later.
... I always volunteered to work the final week or ten days of the calendar year. To me, those days are the start of NEXT year. I celebrate Solstice, and have since I were a nipper. Seems more important to me than a made-up birthday party trying to take the place of the start of winter ... or the last day of a calendar year that doesn't match up to where the Earth really is in it's orbit, for that matter.
No, I'm not Wiccan, or any other religion ... I just like the fact that the days are getting longer now. Starting early spring veggies soon!
If you mean knotweed, goats when it first shows up (to eat what's above ground), followed by hogs to root out what's underground. You have roughly two seasons to implement this. If it's older than a couple years, either dig and burn, or good old fashioned glyphosate applied every time/place there is an outbreak. There are no other options.
You can guess how I know this ...
To you and yours, all the best for a wonderful holiday season, and a healthy and prosperous 2018"
I was hoping for a nice quiet Christmas, last-minute dash to John Lewis on Sunday, family, friends lunch on Christmas Day etc. But no. at 3pm yesterday I get last second screaming from a project manager, "biggest client ... blah, blah ... SLA level 1 support needed .. offshore SLA support not available from Mumbai .. panic panic .. can you cover 24/7 until Thursday?".
As my better half said, "just charge them £1,000 per call until they get the message"
Happy Blooming Christmas to all (except my employer and project managers everywhere)
In this country I think people have an aversion to talking about money generally, we just hope rewards we are promised are big rather than have the awkward conversation to clarify just how big. We don't like talking about the salaries are on or sometimes believe we deserve more than we are on, even if we do we seldom do anything about it.
Everyone else thinks IT is a cushy job and we are paid a fortune for little. We all know that's not true, we also know we are the first to be screwed over financially or with our jobs as opposed to everyone else in the company.
I think the attitude towards IT and IT workers in general needs to change to be more positive and recognise the skills we have and how much the world would be in the shit (and in the dark ages) without us. That would make for the best Christmas present we could all get one year.
"Everyone else thinks IT is a cushy job and we are paid a fortune for little. We all know that's not true, we also know we are the first to be screwed over financially or with our jobs as opposed to everyone else in the company."
Having previously been in the scientific side of the Civil Service I can tell you that that's not confined to IT. There seems to be a strange dichotomy in the managerial mind: if they can't understand what's being done it must be very little and yet they realise they couldn't do it themselves.
I've been fortunate to have had one of those managers. I was on a two week secondment to our US team and wondered why everyone there was so happy and motivated, compared to the grumpy crew I worked with in the UK. For instance - our manager refused to hire someone we had personally recommended, for their technical skills, because she didn't like the socks he wore to the interview. She also had no clue as to what we actually did, in stark contrast with the US manager; who I once spotted walking through the support centre and, seeing that one of the incoming call queues had a long wait time, he pulled up a chair, put on a headset and dealt with the backlog. First day on my return to the UK I made a request to move permanently to the US, so glad that it got granted.
Now be fair.
When you have half a dozen users all screaming that their computers aren't finding the shared documents at 4:30 on a Friday, you can't see anything wrong when you log in yourself and and the network expert has gone home early to nurse a sick relative you need to have the right kind of socks. Otherwise who knows what might happen.
Happy New year.
,,,they realise they couldn't do it themselves...
20+ years ago, my then employer provided IT support for a number of small businesses. I came across a couple where the management was proud that it did not understand IT. Oddly, those particular businesses were not there a couple of years later when I decided to do a bit of research. Engineering businesses and people who actually produce things do go bust from time to time but those ones were accountants!
Here in America would would sue. there is a slight chance of a work place shooting for that much money.
I grew up with family in the trades. I can tell you this if you shorted union trades men that much money the feds would be involved. Not because the have that much pull or the feds want to help out.
No it's cause your ass went missing and the FBI suspects foul play. At a minimum some union thugs would show up.
My current IT department has put up walls over the past few years to push staff to use our support system and not call us, literally everything to prevent interaction with staff is done. It must be done via the support system, if that's not possible it must be done via e-mail and if that's not possible there is an EMERGENCY ONLY line they can use and if it's not an emergency they get a telling off.
We've gone from being fairly visible to invisible in just 3 years, we're no longer allowed to attend "silly project team meetings" which means we're now unaware of what's coming as IT management aren't attending either and we're being left to run around at the last minute. We're landed with software/hardware that we've to support which in some cases we have no idea how to support!
So absolutely we need people to see us in a different, more positive light but that's got to start with our IT management enabling us to be seen more often and in a more positive light. If the only time you see someone is when things have gone to **** then you'll quickly start dreading them turning up!
OH God! Silo working. Over the years (decades) I've butted my head against both sides of deliberately created silo walls so many f*cking times.(Both in the IT and non-IT aspects of my work). It's either management preventing us from contacting other departments directly, or them being not allowed to contact us. This is never in the interests of efficiency - and of course is a brilliant way to prevent efficient working. It's purely about poor management trying to make sure that their incompetence isn't found out. i.e. if one part of their empire is being screwed over, or the service they provide to other departments is being curtailed even in the teeth of sound logic they don't want any one else to know until it's too late. And then they will play one team off against the other. For example front line staff being told that the reprographics team will only be able to accept a limited range of the work they've always done because we've been overwhelming them, while the reprographics team are told that they are having cuts made because front line staff are no longer sending them certain kinds of work. Of course it does get found out eventually. But either they just ride out the storm ( and of course by then it's become old news so no one takes much notice ) or these so-called managers have moved on.
I feel your pain, we were expected to find out everything for ourselves, training was non-existent - my boss said you should read about new products and how to support them in our lunch hour.... and if I ever hear 'Agile' again in my life, I'll stick that Surface laptop down their damn throats - bitter, moi?
Firing the staff and telling them you are getting £50 is a dick move
Except that this is a tried and tested method by administration and HR staff everywhere. Case study:
JP Morgan UK have x number of support technicians with the exact required skill set for Infrastructure Support - Cloud Automation Engineering. Come end of Q3 an admin wonk (not an HR one I might add) decides to offshore these jobs and place all the UK technicians on 3 months notice, and because they all work in the sensitive areas of the bank they are placed on gardening leave: hence there is now no support at all. During this time JPM try to recruit a team in India (understanding of LOB technology drivers, cloud structure, LDAP, SAML, OAUTH2, OpenID Connect etc.) but to no avail.
The admin moron is now summoned to senior management to explain in person why there is now no support at all for DevOps.
Best option for JPM is to take a deep breath and admit the mistake, pay the support team a very large bonus and hope that they will come back to work, but this is JPMorgan and they don't do things like that.
All the support team have their redundancy period accelerated to 1 week and receive 6 months pay plus their expected annual bonus. They are then signed back to JPMorgan on a minimum 3 year contract, at contractor rates of around £1,000 per day, i.e. more than 4 times their existing salary.
OPM and all that ...
Hiding a kill switch in case of no payment reminds me of a coding contract that was starting to wobble and it was decided to make sure we would get paid by removing access to the source code that was being developed on the customers machine (it was some Sperry/Unisys machine that we didn't have access to in-house). Encryption options were few and far between in the late 1980s so we simply used compress to obfuscate the data. We did this every evening before leaving work in the knowledge that the backups ran overnight and would only contain the "encrypted" data.
Sure enough the customer refused to pay the next instalment, and said they would continue on their own with what we had done so far. We said, go ahead. Despite engaging the top support people to "de-crypt" the files, they never figured out that they were just compressed, and had to cough up the rest of the contracted money to get the source code back.
In the early days of "The International C Obfuscation Contest," one of the Honorable Mentions reduced all *.c and *.h files to gibberish; compilable gibberish, but gibberish to any human.
All variables were converted to one-to-three letter/number meaningless tags. All white space was reduced to one space (CR/LF + spaces or tabs would become a single space).
File names were altered in the same way as variables, with the .MAK files updated so they would still work.
Imagine a 5,000 file application "converted" with this kind of "obfuscation." It's not compressed. It's not even encrypted. It's just so inhuman that each and every file would require entire rework to be maintainable or modifiable.
The only step that would have made it better was another entry in later years that would take all text files and "format" them so every printed page would look like the silhouette of an airplane.
An old guy had worked in IT for the same big company since leaving university. He was doing an exceptional job - for which he was well underpaid by market rates. His new boss decided the guy was worth a decent pay increase and it all went through.
Shortly afterwards the guy decided to take a slightly early retirement for family health reasons - but gave the company 12 months notice of that event to ensure a smooth transition. That period also meant the pay rise would create a permanent increase in his "final salary" pension.
Then the pay rise didn't appear. His boss was puzzled and started to investigate. HR informed him that the pay rise had been cancelled by reason of the company policy not to give such things once it is known someone is leaving.
The guy was really unhappy. His family's health situation had deteriorated further. He was now considering going at the Christmas instead - which was still some months away. However he felt a moral obligation to his boss and the team to stay for the originally indicated 12 months.
His boss's boss apparently decided the guy was trying to screw the company - and made a "generous" offer of a future £1k bonus if he stayed on until the date he had originally planned. The guy was so upset that he finalised his notice to leave at Christmas. People who knew the principled guy had told the boss's boss that was what would happen in those circumstances.
Never believe what you are told. Always remember the Pied Piper and "1,000 guilders? Here, take 50".
Always get the offer in writing signed by someone outside the immediate project. Someone a lot closer to the purse strings than the immediate project management who will promise anything to keep the project going.
I remember one notable project where the staff were persuaded to work long hours including evenings and weekends without booking the time because there would be a big bonus once the project took off. Personal (home) relationships were strained and some relationships broke up. The project was then canned and there was no sign of a bonus; turns out verbal promises made by people who don't own the purse strings aren't worth the paper they aren't written on.
Nearly all the project members were young and still a bit idealistic. The more experienced said a few gentle words of warning then steered well clear.
I always kept a spreadsheet of hours worked after my first long hours project where I was persuaded to take £1,000 bonus instead of time off in lieu. Then I worked out my hourly rate for all the extra hours.
""Always get the offer in writing signed by someone outside the immediate project."
This, above all."
This will cause a serious syntax error situation to develop between three parties, the people who was promised a raise, the people who made the promise, and manglement.
"This will cause a serious syntax error situation to develop between three parties, the people who was promised a raise, the people who made the promise, and manglement."
To which the answer is "see you in court" which, in this case, would be an employment tribunal. The company can sort it out internally.
Actually, "Always get the offer in writing signed by someone [in management]" is all you really need.
If they are a project manager or HR manager doesn't matter. In the states at least that will be enough to get paid, and if not it will be fuel for a great lawsuit. Juries particularly don't like businesses that shaft their workforce so where the compensatory damages may be modest, the punitive damages can be quite the retirement account if invested appropriately.
I know one guy who ended up with a $2.2M payout. $200K / $2M. After taxes and such he still has 7 digits in the bank, invested such that he earns ~$50K/year just from that, and the principal still sees some dividend reinvestment.
I always kept a spreadsheet of hours worked after my first long hours project where I was persuaded to take £1,000 bonus instead of time off in lieu.
Always a good practice for two reasons. Obviously good records - kept off site! - are invaluable if you need to dispute payment, especially after you've been tossed out the door.
Just as important though is that keeping careful daily records of hours worked and tasks completed really reinforces the reality of how much you work. Usually people grossly underestimate such things. Once you can sit down and see that you're only being paid for two thirds of hours worked you can consider whether it's time to move on.
Just as important though is that keeping careful daily records of hours worked and tasks completed really reinforces the reality of how much you work. Usually people grossly underestimate such things. Once you can sit down and see that you're only being paid for two thirds of hours worked you can consider whether it's time to move on.
Yup, and when you work for yourself this can be pretty depressing. I am often left wondering how in the hell I worked so many hours but can only bill for so few.
You have to take this approach, documenting hours worked and work done, with just about every job, especially where you have autonomy. I worked as porter and maintenance (mostly superficial grounds-keeping) for an apartment complex. I would make stops in the main office to get a drink and spend a couple of minutes catching my breath and talking to the sales staff. I kept getting into trouble for spending too much time in the office and not enough time working. I started keeping a log of every single thing I did, including residential contact which no one else did, and turned it in to my supervisor every day. The lists I turned in were generally rather long. Even so, it never vindicated me in management's eyes so I left. Got screwed out of my Christmas bonus because my last day was at the start of the Christmas holiday.
Later in life I worked hourly for an ISP and would get harassed about rolling in around 11am after having worked until 3 or 4am either on a project we started at 6pm the previous night or coming at around midnight to deal with a critical issue. At some point the VP of the company who complained about all of our working habits instituted, or rather tried to institute, a time-clock method and I warned him that if he did so he would find he owes us a shit-ton of over-time. Finally got tired of the crap and marched over to accounting/HR (both departments in the same person) and demanded to be put on 40 hour salary, at which time I was told I could not be authorized for over-time. Fine by me. The Friday of the first week I had to spend an over-nighter I came in at 9am and started packing up at 10am. The VP "just happened" to be wandering by and had a word about it, and I told him I had already hit my 40 hours and since I was not authorized for over-time I had to leave, and I had my supervisor's blessing. Red faced, he had nothing to say about it.
In my time I have had really good managers, then I have had managers like these. By far my best managers were in fast food and retail. Go figure.
Sadly, my boss now is a total jack-ass and I have no escape from him.
I experienced the opposite time keeping problem as a team leader. The company owner would always be around late in the evening and would chat to those still working to solve the latest issues. These team members always got the best pay rises and bonuses. This group of engineers would turn up before midday if I was lucking and created the problems they stayed late to fix. No amount of pleading on behalf of the guys who arrived early, produced good reliable work and left at a reasonable time managed to get them recognized.
Yep, always get a written confirmation from HR or the senior boss and reply to it confirming that you accept the terms (emails are sufficient to be legally binding). I once agreed, as a contractor, to be on call for the holiday period on the basis of full rate for being on call at home and double for actally going in if I had to, travel time billable also. The project was then postponed once they realised that this was a stupid time of year toi schedule such wotk, as any screw ups might have ruined half the company's staff holidaies, but I'd made no xmas plans as a result of the agreement, so still insisted I got paid for not doing anything over the period (which they did, after a weak attempt to wriggle out of it).
If however, this was done to me as a permanent employee, I would have told tham that I am leaving right now, and if they don't phone me before 5pm offering the full amount promised, I'd never be back again. The story in the article shows why you should always have preparations in place to walk, even if it the job is secure and / or enjoyable. For me, people who meekly accept this sort of treatment just encourage higher ups to behave like this towards others. Of course you have to be pretty damn good and valuable to a company to have the confidence to do this.
Echoing the fine comments made by many others - Hope everyone has a really damn good winter solstice piss up and feast, and best wishes for 2018.
Always get the offer in writing signed by someone outside the immediate project. Someone a lot closer to the purse strings than the immediate project management who will promise anything to keep the project going.
My first boss was terrible at over-promising and under-delivering. Something critical comes up on a Sunday, he'll call up and ask if you can fix it and he'll "take care of you".
I never worked out a good way to say, "Absolutely, I can fix the problem once you get specific about what "take care of me" means - are you going to run me a bubble bath or give me a £1k bonus?"
I'm keeping my fingers crossed for this year. Last year was a disaster. At about 3pm on NYE, some alerts were raised by our ISP. Trying to get in to have a look I found a number of machines strangely non-reponsive (including our main monitoring server). Thinking the worst, I had a look at the UPS logs which showed the output had gone down for a few seconds. I managed to reboot a number of machines via remote PDUs and get to a more-or less working state.
15 minutes later *everything* went dark, so I was off to the DC. When I got there, I was greeted by silence for the racks and a 160kVA UPS festooned with red lights. One phase of input was gone and the UPS was in shutdown. Managed to get hold of an electrician and on-call UPS engineer. The sparky arrived first and found a blown 300A fuse in the UPS feed. We searched in vain for spares but managed to come up with a 200A in the same size which would do at a pinch. I went back up to the DC and via the radio asked the sparky to switch the breaker back on. I was confronted by a 6 foot fountain of sparks leaping from the front of one of the redundant UPS rack units and a very loud bang indeed. If I'd been standing in front of it it would not have been pretty.
Not too long after the UPS guy arrived with the smell of smoke still heavy in the air. The scorched unit was opened up, revealing a main board covered in soot and the input wires from the rectifiers melted back by over half an inch from where they had been soldered into the board, blobs of molten metal scattered around. The UPS chap although rather surprised checked all the contacts in the frame, which had luckily survived and set off back to base to get a replacement unit.
At about 5am he returned, new unit in hand. We had to replace all 3 phase fuses and then where was a very tense moment as the breaker was thrown again. Luckily power was finally restored. Thankfully due to the way the days fell we had two more days to recover everything. I called in the rest of the team and managed to get 3 people to help me sequencing the power-on (about 120 physical machines and a few hundred VMs). I left exhausted by 7pm (but still was connected at home) and by 11pm on the 1st we had all the servers up, with the application guys in Melbourne finishing up on the holiday Monday.
Ruined New Years for a good few of us that year. And we only got 1.5 TOIL/OT from it - but at least the "right" people remembered what we did and thanked us just a few days ago. Fingers crossed for this year.
Postscript: An IGBT had cracked open in the UPS module, had never been seen before by the engineers. One 300A fuse and 4 more 200A blown...
Ouch, and I thought blowing up a couple of normal PC tower sized UPS's was bad.
(Cue panicked shut down of everything, web servers, database servers network switches et al. Though we intentionally had a number of them paired to machines for redundancy but we'd had them a while so some of them weren't upto the task anymore. In A previous job though)
We'd like to keep you around, regardless of On-Call.
And if you hear somebody dismiss Y2K as nothing, remind them it turned out that way because of the how the tech community prepared for it. Hell, I even kissed my wife on the New Year standing next to a running 300kw genset.
Merry Christmas to all, and have a happy and prosperous New Year!
And if you hear somebody dismiss Y2K as nothing, remind them it turned out that way because of the how the tech community prepared for it.
You might imagine that the CEO of Birmingham Chamber of Commerce might have a bit of a clue about (a) IT and (b) the business implications of erasing 45 years of trade legislation.
Said CEO, Paul Faulkner, was a speaker at a recent event about brexit at Birmingham City University. He breezily dismissed concerns about whether planes could fly on 30 March 2019. "It's scaremongering," he said, "just like Y2K, nothing will happen."
In the audience time I took the opportunity to point out that "nothing had happened at Y2K" because thousands, if not millions, of programmers had put in the overtime to fix it. David Davis, on the other hand, had just admitted that he hadn't done his homework and didn't have a clue what the "impact" of brexit might be on 57 industries. I don't know what the legal basis is for flying a plane across the Channel, but so far as I can gather, it will be illegal when we pull the plug on 45 years of legislation.
"He breezily dismissed concerns about whether planes could fly on 30 March 2019. "It's scaremongering," he said, "just like Y2K, nothing will happen.""
If it's up to the British government, then indeed nothing will happen - no takeoffs, no landings.
My solution: Any EU plane landing in the UK without permission pays a fine of £1. Any UK plane landing in the EU without permission pays a fine of €1.
This post has been deleted by its author
I’m not exactly full of Christmas cheer after reading the stories and comments.
Where are all the stories that ended happily for the poor IT guys, or at least a good revenge / “company left themselves in the shit, forever regretting screwing their IT guys” ending.
Makes me less sad to have left my IT career though I suppose.
Many years ago I worked for a major financial institution. Christmas day fell on a Saturday so we were looking forward to our reward for working (I was part of the IT operations staff). Our conditions stipulated double time plus a day in lieu for working the public holiday. It turns out that this particular year the Government of the day decided not to gazette the Saturday as a public holiday.
There was a hue and cry from the public when they realised that Christmas day was not a gazetted public holiday because, effectively, this meant that people could be expected to work Christmas day as a normal Saturday (retail staff in particular could have been impacted). We protested but the HR department bought out the agreement we worked under and stuck to their guns. We decided to do some independent research and found that Easter Saturday was gazetted as a public holiday for the finance sector (and always had been). We had never received the appropriate payment for Easter Saturday so greatly enjoyed enforcing the rules and made the HR department calculate the back entitlements for all impacted staff (more than 5 years worth of Easter Saturday entitlement for most of the 40 staff on the shift roster).
It ended up costing much more than it would have had they just given us what we expected for Christmas day.
Our 'Y2K co-ordinator' tried to book holiday over the period. Now, I volunteered to cover one portion of the on call, because I was on anti-biotics after being bitten by a feral cat, and having a swollen hand, and not being able to drink (because of the anti-biotics, not the inflamed fingers). I had a word with my boss, who had a word with theirs. They worked.
Nothing bad happened anyway, because you know, we'd spent pretty much the previous year making sure it wouldn't.
I became an independent contractor for these very reasons.
I invoice every week. I invoice by the hour (plus expenses). My terms are "Due Upon Receipt." If the money isn't in the account in seven days, I stop working. I may lose a week or two, but I'm never hanging on for months' of pay; overtime or not.
If the prospect objects to my terms or says their bureaucracy can't handle payment turnaround like that, I know I'm dealing with a very high risk company. If I don't get the gig, I consider myself lucky.
The global company I work on behalf of uses Net 90 as their standard terms. In other words, all vendors get paid 3 months after submitting an invoice. Many won't accept this (it would put small companies out of business!), so we sometimes end up paying one vendor to subcontract another's work - at a markup, of course...
AC for obvious reasons.
As IT numpties are vastly overpaid and generally overstaffed the concept of them getting massive bonuses is an anathema to those who simply do the job and meet their commitments through personal integrity. That the bonuses are would be paid from the taxes of those who have proper jobs and personal integrity is doubly galling... and you want sympathy? Try getting out of the virtual world and into the real one...
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2025
