Two things are alarming, one rack for all four internet connections - that's just crazy, everybody has been using diversely routed fibre to a number of racks for years now, so Cloudflare obviously didn't even know they were reliant on one rack. Not knowing is worse than it being that way, we all have older infrastructure which need to be upgraded or rebuilt but Cloudflare obviously missed this because they didn't now about it.
Secondly they were dithering over switching to DR because they admit switching back is difficult. It shouldn't be difficult - switching back is part of the DR plan, especially when it concerns your main control system. For such a critical piece of control they should be able to swap between two sites very quickly, both directions.
I like Cloudflare very much and am a customer spending a lot of money with them but they have had their pants pulled down in public and it's shown that perhaps they've grown too quickly and not put enough effort into reviewing key items of infrastructure and processes to ensure resilience is maintained at all times.
Biting the hand that feeds IT
