I used to work in an underground secure control room. Basically a load of people in a room with all of the required servers and other equipment to keep the company running.
The entire bunker was protected by UPSes and backup generators that got tested properly. Tests consisted of a supervisor pulling the main breaker and running on UPS for 10-15 minutes before switching to generators for an hour or so. This test was conducted every single week, with extended tests to empty the tanks every 6 months. The sort of testing regime most people here would love to be able to do.
Then there was an actual power failure. The UPS and Generators worked perfectly and all systems continued running... Except for the air conditioning units, which not only regulated the temperature but also supplied the meat bags with their oxygen. Apparently they lasted about 10 minutes before the shift supervisor told everyone to activate their respective DR procedures and evacuate to the car park.
Biting the hand that feeds IT
