The joys of generator tests and failover
One financial institution I worked at many years ago decided to do a generator test late in the evening midweek - I was on shift that evening.
The failvoer goes Mains -> Battery -> Generator
On this test it went something like Mains -> Battery -> Generator (fails to start, everything goes black) -> Back to battery (everything comes on) -> back to generator (everything goes black) -> back to battery everything comes on -> batteries give out -> everything goes black and stays black.
No emergency lighting in the DC where the bridge was it turns out and no outside lighting to shine in thru the windows as there were none.
The hilarity, the mainframe batch runs that were in the progress at the time, the chaos - fun times.
So issues with the generators and lots of issues with failed batteries, lots of root cause analysis, lots of beefing up batteries, failover procedures etc...
There was fun at one bank where the generator fuel tanks spung a leak and flooded the basement canteen to a depth of 3 feet, they never did manage to paint over the diesel tide mark enough and their was a odour of diesel for the rest of the time I was there which never quite left.
Biting the hand that feeds IT
