Re: Business Continuity
I'd be willing to best most(90%+) of the customers did not take their own backups, just like most likely most office 365 customers don't take their own backups. Quite surprising really (maybe I shouldn't be surprised).
Mirrored sites can still be compromised, if anything it may be easier, compromise one site and the replication automatically compromises the other site(s) for you (depending on how it was compromised and what kind of replication). Failures can also replicate, data corruption can destroy multiple sites as fast as your replication can send it.
ISP going down and security compromise are very different things. Myself I have been involved with 3 primary storage array(SAN) failures in my career, all of them took multi day recovery efforts, all lost some data with a risk of total data loss, and in all cases the company did not have good backups, ALSO in all cases the company chose not to immediately invest in better protection going forward following the near disaster. All 3 situations were the most scary of my career as well, and in the two I was directly involved with I pulled an unbelievable amount of monkeys out of my ass to get the systems working again. The first one was early in my career and I was on the ops/app team not the backend team so I just waited while they worked to fix the issue. But I was the one to report the issue to everyone, will always remember the Oracle DBA telling me he almost got into a car accident when he read my emergency alert sent to everyone on that Sunday at around lunch time(with output from the HPUX Oracle systems showing "I/O error" on several mount points from the df command). Spent about 32 hours on a conference call for that, probably my longest ever conference call.
I've been fortunate never to have been involved in a serious security incident(have had to deal with a few stupid hacks from unmaintained systems that I was asked to help with over the years).
I run my stuff pretty well, though nothing is perfect, the best strategy (if possible) is try not to be a tempting target. Rackspace, hosting a lot of customer stuff is obviously not in a position to do that, so they have to deal with a lot more things than I.
Biting the hand that feeds IT
