I got paid for xmas overtime.
My employer at the time shall remain nameless to keep the laughter from crushing their itty bitty egos again.
Our data center was located in a co-location facility along with multiple other companies.
It shared power and aircon and water, etc, all on multiple redundant supplies, that way if one failed it *hopefully* wouldn't take down the entire center.
(I bet you can tell where THIS train wreck is going, huh?)
Xmas eve rolls around and we're all told to enjoy the holiday with our families.
I was the low man on the IT totem pole and given the emergency pager.
Had I known then what I know now I never would have accepted the damned thing.
I've just sat down to xmas supper with my family when the bloody pager goes off.
I excuse myself, call in to ask WTF, and get told that our data center has gone offline.
"Get there ten minutes ago and get us back online an hour ago!" my boss yells at me down the line.
I wisely refrained from mentioning that time travel wasn't in my job duties and I'd charge extra if they were.
I hang up, give my folks the bad news, and head in to work.
Where I find out that the aircon had caught fire, the firemen were on site putting it out with chemicals (something about multimillion dollar computers not liking water for some odd reason) and told me I'd have to wait to enter the colo.
Fine by me, I'm getting paid by the hour and holiday emergency rates at that.
It took them nearly two hours before they declared the site safe to enter, by which point the pager has gone off half a zillion times.
Each time it does I have to trek over to a local phone booth (remember those? Cell phones weren't a thing yet ya young whippersnappers!) to call in and explain why I hadn't fixed it yet.
Every time I patiently explain that the building had caught fire, blah blah blah, and I'd get on it just as soon as they deemed the building safe.
Boss is screaming we're losing thousands of monies every minute and I need to get us back up NOW!
Repeat that bit about the fire department & cheerily hang up.
Anyway, I get in to find the colo *packed* with all the OTHER techs from all the other companies whom have sent out their own troubleshooters to the scene.
Turns out we were ALL getting holiday emergency OT rates for standing around watching guys playing with their hoses.
*Snerk*
There isn't enough room for all of us to be in the room at the same time, so the senior techs go first while us lowly PFY's have to wait in the hall.
Fine by me, I'm getting paid by the hour.
About an hour and a half, maybe two hours later it's finally my turn to enter.
The water isn't the problem; the power isn't the problem; the lack of any functioning aircon however is a BIG problem.
And not one I can DO a damned thing about since I'm not part of Building Maintenence and *they* are the only ones allowed to futz around with internal building subsystems.
They have been called out to try & get the aircon back up and running, but meanwhile I'm stuck there deciding which thumb to sit and spin on in my boredom.
All the other techs have verified that the racks have power, the UPS' kicked in long enough to ensure proper shut down, and the physical hardware is peachy.
I am fairly (nearly 99%) confident that our equipment will be in a similar state, needing only a reboot and a bit of hand holding to make sure it comes back up in a fit state.
But I can't DO that until the building guys arrive... and I'm getting paid by the hour.
Pager buzzes, find a working phone, call in, explain AGAIN about the situation, remind the boss that I've only had access to the computers for the last few minutes, and NO I'm not able to restart any of it until the building folks get that aircon up & running.
"No aircon in a room full of heat generating servers makes for a fatally hot room. NOBODY is allowed to run their machines for any longer than it takes to do a POST check of the basic hardware. We are NOT allowed to leave them on as they will generate too much heat. Heat that the room can't get rid of without a functioning aircon unit."
Boss snarls fine, tells me to stay there and get it running ASAFP.
(Chorus)Fine by me, I'm getting paid by the hour.(/Chorus)
The building guys show up about an hour later, hit the roof to examine the wreckage, and deliver the "good news / bad news" to us techies waiting in the colo lounge.
The original aircon unit is a total loss and can't be repaired in any reasonable amount of time.
The good news is that they had a replacement already installed waiting only for a thorough inspection before they turned it on and made it take over.
"We'll have it back up in about an hour. Is that alright with you?"
Hell yes. Why? Everybody sing it with me... (Chorus).
Once the building guys had the new one online and told us techs we could bring our computers online, the senior techs got to restart theirs first while us PFY's had to wait.
(Chorus).
I finally get my turn, bring the servers back up, hold their hands until they're humming right along, and turn their network connections to live.
I'm sitting there waiting to make sure it's not going to fall over when the pager goes off.
I find a landline, call in, and my boss tells me "It's working now. Go clock out before I fire you for incompetence!"
I got signed statements from the other techs about how long I had to wait before I could even start *and why*. I got signed statements from the firemen as to why it took so long to clear the building before they would even let me in. I got signed statements from the building guys why it took so long to resume aircon. And I took it all home with me since it was less than an hour before my normal shift started.
I went home, changed, showered, grabbed some fast food for breakfast on my way in, and slapped said paper statements on the copier as soon as I made it to the Xerox machine.
Copies went into my desk, the H.R. manager's inbox, and more handed to my boss.
He asked what they were for.
"To make sure that I get paid for all the OT I've spent, the fact that you told me to clock out less than an hour before THIS shift started, and that if you DO try to fire for ''incompetence'' I'll have something to defend myself with in court. Oh, and by the way, I've legally showed up for my current shift after NOT getting eight full hours since my last one. That makes *ALL* of the time I spent at the colo as OT to start with, the holiday emergency rates will apply *on top*. You owe me for 4 hours at a minimum and I'm going home to sleep now. See you tomorrow!"
I've never slept so good in my life.
I show up to work the next day and get met at the door by H.R. who is NOT happy to see me.
Evidently the Legal department had been brought in on it and explained in NO uncertain terms was ANY form of retaliation to be done against me left I "sue us into a shithole so deep we'd never see daylight again."
They had to get Upper Management Permission to cut me the cheque to pay me for my work that day/night/day, and Uppper Management was currently chewing the boss a new ass through which to shit from.
Budget? He was now seriously over his.
I left a few months later when one of the techs from the scene called me and asked f I wanted a job. His company was hiring and he liked what he'd seen of my skills.
I said yes, put in my two weeks notice, and damn-near glided out the door on Mercury-winged shoes of joy.
(Singing merrily)
Tis the season to be jolly,
I got paid... by the hour!
Tis the season to go shopping,
Because I got paid... by the hour!
Now I'll don my gay apparell,
Ha ha ha, ha ha ha, HA! HA! HA!
I put a down payment on a new car,
Because I got paid... By. The. Hour!
(/Singing)