Re: DR Testing Failure
I'll wager the components of a recovery plan were all documented and tested, including physical test runs of the gensets. I doubt they did a full "turn off the mains power" test, but if they had, they'd have been in the same position (sitting in a dark data centre, thinking "shit!").
The other possibility is that they have turned off the mains in tests, and everything went perfectly. That's a known problem with standby power - it only works most of the time. And on that subset of times when it doesn't work, you usually need it and everybody notices.
A question for the DR professionals: What is the ACTUAL failure rate of a completely successful, fully automatic handover from interrupted mains to on-site generators? My guess is nobody does it often enough to know.