Re: "an opportunity to preempt this threat"
Sooo 2000s (and yes I did have test servers back in the day)
Nowadays you have snapshots for your VMs and backups that can be restored quickly and replicas of your VMs. You rarely have to go back to tape these days.
Yesterday I restored my home PBX because (salient facts):
* I'm an IT bod - obvs we have a home PBX
* Wife does dog boarding at home
* We have electric underfloor heating, a fairly open floor plan and 75% of ground floor is laminate. U/F is connected to its own consumer unit (UK) with ten zones
* IT gear is mostly in the attic - VMware esxi (Dell T420) and router and switch and other stuff
* IT gear in the attic is powered off one of the U/F circuits with a UPS
* ... leaky dog ... ... very leaky dog ...
I have yet to 100% confirm that a dog killed one room's underfloor heating but it is rather likely. I've got five out of ten zones switched off via the consumer unit and one of the switched off zones is where they pissed. Whilst working out what the hell was happening I flicked power off and on a few times.
The Dell VM host decided that its battery backed cache on the RAID had lost data. I'm not sure exactly why because I monitor all my gear and my customers with Icinga1 and 2 (migration ongoing) and none of the logs indicated a problem. Power went out, UPS depleted and box crashed. Outage was about 12 hours but the battery on RAID should have managed several days at least.
I had three MariaDBs on that box and two went wibble. I recovered my PBX from backup (Veeam), my Nextcloud was fine and Zoneminder needed a parameter to ignore knackered data in INNODB and I also had to delete the redo logs (which I thought should have saved me - nope) and I was able to dump the zm databases to files. I recreated my ZM db and dumped the data back in.
Backups are quite handy,