Failure built in.
Anyone who has participated in the scoping and design of any high availability solution - be it storage, server, network, or other technology - will know the sad evolution.
- At first a simple design based on trusted - though possibly leading edge - technologies that gently carves away single points of failure and eases that nagging gnawing of the gut that plagues most system managers who have not yet traded duty for apathy in self defense.
- Then the sales pitch to convince management that the investment is worthwhile highlighting the benefits of increased uptime, easier management, improved fault tolerance, and availability.
- Then the intra-management chinese whispers that morph that into "always up, easy to manage, wont fail, always available". The critical application migration begins and suddenly the solution is the "The Solution".
- Next, or perhaps simultaneously, come the changes in culture. The changes in procedures and processes that at first "take advantage of" and then eventually "rely on" The Solution. The old techniques and processes that used to give application developers and system managers some level of comfort as the data flowed through the system are slowly phased out as unnecessarily inefficient under The Solution. Why duplicate that piece of data at rest when it is safely stored across so many spindles and in so many processing units?
- And then, for a time, comes peace - when The Solution does what it was designed to do and does it so well that more and more critical apps find their way there. Only now they are designed for the solution. There is no vestige of the old ways - the old habits of good programming or diligent system management. Applications keep data on the fly and tape units stand idle except for the occasional, mandated, "disaster" backup. The old distrust of the weak and vulnerable computer has become a faith in The Solution. Even when experienced devs and system managers voice their discomfort at that old gnawing sensation again, there is too much invested in The Solution now. Besides, it is working perfectly.
- Finally comes The Failure. The scenario that slipped through testing. The risk that was of so low a likelihood or of so small a consequence - at least in the original stages of planning - that it was completely disregarded. Regardless, there are no fall-backs because the fall back was The Solution and The Solution has failed.
OK, so they mostly dont turn out that way - certainly while sane minds are in charge. But these solutions are becoming more and more "all encompassing". More self contained. More commoditised. They are ceasing to be technical solutions that require analysis, understanding, and critical review, and instead are becoming commodity blackbox services that often have only a promise, a contract, and a fee.
The Solution will always look good. Being able to buy into The Solution without a crippling investment in knowledge, time and technology will always look appealing. You can even get contractual guarantees of The Solution's good'ness. It is just that when The Solution goes wrong, you lose an awful lot. Maybe everything. And being able to point the finger may be less satisfying than you think.