Reply to post: An entertaining and educational tale...

Electrician cuts wrong wire and downs 25,000 square foot data centre

Disko
Pint

An entertaining and educational tale...

I like to assume a 1 in a 100 chance of any operation NOT failing terribly.

Then the only thing that can be done to avert certain disaster, is "everything".

Treat every single element and action that is part of the project as if it were a sleeping dragon, only waiting for a minute change in the environment to begin spewing lightning and sulphuric acid into every connector it can find, to unleash catastrophic failure upon your entire operation.

Take inventory of the many many ways disaster can strike; known or unknown, worst case and best case scenarios. Management can deal with the numbers: we now have 13.768 scenarios for catastropic failure (a made up number, but realistic for a datacenter IMO.), and we only have 768 ways to stop it. Now management can break their pretty heads over the 13.000 things that can go wrong for which there is no solution, and the impossibility of preventing every possible doom-scenario, and realize the inevitable: we probably need to invest a lot more to make this work: more money, more time, more preparation, more people, and more effort on the part of management to ensure business continuity, even in the face of certain disaster. And yes, we are out of here at 5 and won't be back until wednesday.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon