Where's the HA and DR?
Sure, shit can happen - although as nearly everyone else has said above - proper testing would have prevented this.
But what struck me is they have a single cluster. Is there not a mirrored version elsewhere using different infrastructure, where the changes get applied later? Sounds nuts to me that a BANK is depending on a single tech stack that they don't fully understand without a different stack running in a different environment in a different datacentre.
Like their debrief, but I wouldn't let them hold my money.
Biting the hand that feeds IT
