Re: Good point
It seems to be a tragic reality that despite the fact that we are, as employees, often asked to write up disaster plans I have never known one to actually be fully implemented after the fact.
I'm sure a lot of us will have done the "What if..." threat planning game.
"What if I'm locked out of this device?" -> We have backups, taken nightly.
"What if we can't retrieve the nightly backup?" -> We have off-site backups that are taken weekly.
"What if we can't boot the device to restore the backup?" -> We have a backup-restore boot option available on the network.
"What if the network is down?" -> Our configuration management will repair it.
"What if the configuration management is down?" -> We have a backup of the config management master, taken weekly, that can be restored from a physical disk.
so, now, everything has gone completely to shit, and what's the procedure?
Step 1: Send an engineer to the server room with a copy of the config management master server backup disk. It will restore a working config management system.
Step 2: Order the config management start to repair the network based on known physical states such as mac addresses and port numbers. If that fails - and the only way it can is hardware failure, and we keep spares - then when it fails we'll know where the physical failures took place because we know the physical location of every asset. It's recorded in the config management system.
Step 3: Go replace the thing and go back to step 2.
Step 4: We now have a working network and can reach all the physical hosts, including the backup host. If the backup host is not responding, order the config management system to rebuild the backup host and then reconnect it to the storage targets.
Step 5: Now we have the backups back, attempt to reach the downed host (or potentially hosts) if you can't, then order the config management system to rebuild those hosts. Either everything is good now or we can't reach it so...
Step 6: If the host is somewhere off network, ask the employee who's actually got hands on it to boot it into recovery mode, where it will reconnect to the VPN using the recovery certificate and now we're back to Step 5.
See, none of that is hard to understand. This sort of thing ought to be standard procedure absolutely everywhere because hundreds of people like myself have played this exact "What if?" scenario through and thought out how to deal with every step in it hundreds of times over the last... pick a fucking number... years. We all know what we ought to do, and by this point there should be thousands of examples of it actually being done. This should not be hard at this point.
How many people actually do it tho? Not. Fucking. Many. I've worked at precisely one place where we ran through that little thought experiment once a year and the entire team tried to think of ways that the process could fail and if we came up with any opened work tickets to try and fix them. Everywhere else I've worked has Done the "What if?" at some point, although usually only once and only because a new manager rocked up and wanted to see it done, but then none of them ever actually implemented any of the recommendations.
Actual disaster recovery procedure always seems to end up as some ultra low priority that people love to talk about, but not actually do anything about.