Here we go again
IF I were employed, this would be hilarious to watch. As it is...
Most of what I know about HA & SRE, I learned at Google a decade ago.
This sounds like m$ is, at some level, trying to do things right. Real-world global HA means regional failover and a whole lot more. You do NOT drop customers just because a couple of regions go out. You shift them to healthy regions. (And your SREs are already working on the problem--see below).
"Monoculture" is not the problem here. It would be an absolute nightmare to try to manage parallel systems doing the same thing, both at an organizational and at an operational level. And the cost? Much, much, more than you think. Google developed an entire new language (go) that is torturous to work in because it ran fast, and at that scale, hardware costs outweigh salaries.
So, let's talk about what went wrong. Failure to test properly is the original root cause. (And HA locking sounds like a fascinating problem to work on.)
But how THAT failure turned into this?
1) Load on the affected servers rose for almost sixteen hours before failures--and no one noticed? Rule #1 of SRE: alert on yellow. Read that again. Proverbial klaxons go off if you loose half of your excess capacity, not when systems start failing. (And by "capacity", I'm talking about load availability across your fleet, not # of servers). It sounds like m$ didn't think there was a problem until the support cases were coming in.
2) Recovery from massive failures is also hard, BUT this has been a known issue for well over a decade (I would guess two, but I KNOW Google had to learn the hard way in 2015).
It turns out, the solution for mass failure recovery is the same as preventing the mass failure in the first place: quota. To quote Simon, "My job is to keep the network up. That I allow users on at all is because of my good nature." Read THAT again.
Quota servers actually are quite magical. Turns out, you really CAN drop tens of thousands of requests a second a server--if you keep the logic simple enough, but especially if you are prepared to handle 0% availability to protect your back end as a core feature. And no, your admin interface does NOT share a port with user access.
Of course, you still need to monitor your quota farm for ITS health.
Now these systems are NOT supposed to be responding to generic traffic, but to traffic generated by their own clients, which brings me to my next point. "EXPONENTIAL BACKOFF! DO YOU SPEAK IT?"
Indeed, to defend against this scenario, you can do even more than exponential backoff. As a problem develops, quota servers can signal clients to back WAY off.