Almost time to hit the big red button
Only once in my 27 years of IT have I had that 'do I power off the ops room or not' decision to make.
One afternoon in sunny Surrey our cleaner spotted something shining through the air con vents of the raised flooring and asked me as the IBM Mainframe Operator on duty to take a look.
Looking down I realised there was a pond of water there instead of the usual bone dry concrete, with the air con still happily blowing away above it.
Rather than lift floor tiles to see how far up the sloped room the water had got - and I was thinking of those four-bars that were stashed under the AIX / Token Ring cabinets... - I did consider powering down the Ops room entirely via the big red button. However as I stood there I realised nothing had stopped working, nobody was dead, nothing was on fire etc., so I legged it up the corridor to pull my boss out of the monthly IT meeting. I'd never seen his eyebrows raised as high as when he looked through the same vent. When we did lift the tiles shortly afterward we had less than a foot to go before the four-bars would have been flooded. The IBM engineer said it was an impressive amount of water for a pin-hole sized water leak...