Power outages, UPS's and overheating
My team had just finished upgrading a remote (as in >200 miles away) site. We'd pulled fiber and cat5e, changed out their analog phone system for VoIP and installed two shiny new server racks, complete with a very nice UPS. A week later, there were screams of rage from them and demands for my still beating heart as a sacrifice.
The site had had a long term power outage. After an hour on battery, the phone system along with the rest of the servers had gone down. This was unacceptable! Heads will roll! As the head of the team, I was called on the carpet.
Brought into the Star Chamber, I mustered my feeble defenses. I had designed the UPS setup to last an hour, and it had done so. I had not been given the budget to do more, and (heresy!) did not believe that it made sense to provide more battery backup. You see, there was a large emergency diesel generator that was supposed to kick on automatically when grid power went down, and it had not done so. Also, my servers had shut down from overtemperature, not power, after running an hour with no cooling in the tiny closet deemed appropriate to install them.
Well, that wasn't good enough for these great men. They demanded that I reprovision the system for 4 hours - nay, 8! When I pointed out that the system would still shut down in an hour, I was told to put that air conditioning system on the UPS as well. My laughter at this point was not appreciated. I was told to go and get a quote for this, to provide to the board tomorrow.
They weren't happy with the quote. I helpfully stated that there would need to be some building modifications in order to host this new battery room, and that they'd need to talk to someone qualified to provide that information. As an addendum, I noted the price of a replacement diesel generator, and that it was 25% of the cost of the battery extravaganza.
The incident was not spoken of again. I understand that the generator was overhauled and tested regularly afterwards. My rise in that organization came to a halt. Thou shalt not laugh at thine superiors, nor shalt thee make them look like fools, and I had violated those commandments. It was just that I've never had anyone demand a battery backup for a 5 ton A/C unit before (or since).
Biting the hand that feeds IT
