When disaster strikes, proper preparation prevents poor performance As Benjamin Franklin famously said: "An ounce of prevention is worth a pound of cure," and that's especially true when it comes to disaster recovery. Most companies with a decent-sized IT department will have an incident response plan, but that in itself is nowhere near enough. Such plans have to be constantly updated and …
Business disaster prevention:
Prevention costs money. Continuous backups cost money. Redundant systems cost money. Competent employees cost money. Training cost money.
And we don't expect a disaster this quarter.
Joking aside, the only effective strategy is to practice regularly. Every X days or weeks you shut down a server or service and restart it from backups. You move work from one site or cloud to another, etc.
Only then you can be confident the backups work and people know how to respond quickly and effectively.
But when this finally works, you cannot simply fire employees every few years as the replacements have to be trained again. So, now the board fires you and hires someone who doesn't waste the stockholder's money.
Remember from Buncefield and other disasters that the off site backup or the hot/cold standby systems should be an appreciable distance from your main business (which of course adds a large amount of suspense).
Too many disasters have been affected by large power outages, dependencies which turned out later on to have a single point of failure, or appreciate natural disasters.
I think some of work's data centres are around thirty miles from one another as the crow flies, which isn't perfect, but it's likely to need such a large disaster that the only thing most of us would care about would be staying alive to worry about greater resilience.
We know that if you haven't run your process through the DR system recently with your current staff and your current SOPs, you haven't got a DR system.
Management thinks if you have an idea about how you might do it and you created a runbook for it several years ago then the box it ticked.
I recall in a previous job, and this brings in the bit about "outside help", manglement had someone in from outside 9without actually telling IT !) - I don't recall if this was from the insurance company or the businesses US owners ...
So manglement then throws a list at IT of "failures" and an instruction to sort them. One of them was "no lockout on failed login attempts", which we were told to enforce even though the OS we were using at the time didn't support it, nor could we lock out "dynamic" TTY lines used by Telnet clients (yes, long time ago) - only when the fixes caused "lots fo problems" were we allowed to revert. Somoething we could have pointed out to the clueless auditors (who had zero clue about our systems and just used generic checklists) if we'd been given the opportunity. But I digress ...
After one such visit, our director just looked at us and instructed "write disaster recovery plans for the IT". Well I had some ideas, but fortunately got onto a business continuity course which was very illuminating - if you are in any way involved in business continuity (BC) or disaster recovery (DR) then I recommend getting on a good course. The main thing I learned was that you can't treat the IT as an independent entity - you need requirements from the business. Without this, you could implement something that won't support the business, or spend more than you need to on better than the business needs. Needless to say, requests for this sort of involvement from manglement were met with responses along the lines of "stop being awkward and just do it".
Oh yes, and did I mention there was no budget for anything either - so if we'd determined that the backup systems we then had weren't adequate, we didn't have a budget to improve anything !
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2025
