I was working for a major UK High Street bank who had an online share dealing offering for their customers. I was one of the lucky team tasked with keeping this up and running. We replaced some aging hardware with shiny new HP PA-RISC N Class servers and I set up MC Serviceguard to pair them as a HA cluster with lots of resiliency. All good for the first couple of months.
Then the day came we got an alter that one of the N classes had gone down and this was quickly followed by the second one. The shouting from the sharedealing arm of the bank got very loud very quickly and I set off out of the office block to the data halls to see what was going on. As I approached the racks, there were barriers round several floor tiles that were up and there were bundles of cables hanging out of the holes. The head of the data centre management team popped out of one of the holes holding more cables and he looked at me, at first with puzzlement and then with increasing horror. He was removing redundant cables from the racks where the old servers had been racked but had traced the wrong cables back to the core switches and pulled all the new live system cabling.