Predicting failure modes
...is incredibly difficult to achieve. There will always be a combination of events that can occur that will screw up the calculations. That one in a million possibility is the chink in the armour which then causes an unconstrained domino effect because it was down at the bottom-end of the risk assessment.
I used to work for an organisation that had two of everything to ensure failure of one system meant that the other could be brought on-line in case of failure, thus minimising downtime. One day the switching mechanism between the two systems failed.
At the end of the day you have to balance the cost of call-out teams to cope with one-in-a-million failures against day-to-day operations. Most companies would prefer the asterisk disclaimer at the end of their Up-Time Promise.
Biting the hand that feeds IT
