UniSuper Google Cloud outage caused by an unfortunate series of events

Google's Cloud CEO Thomas Kurian has weighed in on the UniSuper fiasco and confirmed that UniSuper's Private Cloud subscription was accidentally deleted. In a joint statement with UniSuper CEO Peter Chun, Kurian admitted that an "inadvertent misconfiguration" during the provisioning of UniSuper's Private Cloud services …

[elsergiovolador]

Someone else's

At the end of the day, cloud is just someone else's computer stored away somewhere you don't have access to.

Then someone you don't know can just go in and delete your data, because they had a bad day or they were tripping still after a wild weekend with mates.

You don't know if they just given in to their intrusive thoughts telling them to write 'rm -rf /home' or whatever.

You also don't know if someone had a fight with a hubby over the phone whilst cancelling someone's subscription and you just happened to have a similar name.

[Notas Badoff]

Re: Someone else's - Abend's Observation

I really liked commenter Abend0c4's observation:

"Many cloud systems are actually just distributed single points of failure"

[Mike 137]

"UniSuper had duplication in two geographies as a protection against outages and loss"

Using the same account on the same provider -- a classic predictable single point of failure. Even if they'd used two separate accounts on Goo cloud, the problem would not have existed.

When will we finally learn what redundancy really means?

[Gene Cash]

Re: "UniSuper had duplication in two geographies as a protection against outages and loss"

> When will we finally learn what redundancy really means?

From the article (which you seem to have not actually read)

"Fortunately, UniSuper had backups at another cloud provider"

[Mike 137]

Re: "UniSuper had duplication in two geographies as a protection against outages and loss"

"From the article (which you seem to have not actually read)"

Fallacious assumption that missed my point entirely - actually I did read it, and the fact remains that replication using a common account does not equal redundancy if the account goes down, so regardless of any other measures taken that specific one was completely pointless.

[An_Old_Dog]

Re: "UniSuper had duplication in two geographies as a protection against outages and loss"

... rather like how RAID-1 (mirroring) is not backup.

[NoKangaroosInAustria]

Re: "UniSuper had duplication in two geographies as a protection against outages and loss"

But it's not really like RAID-1 now is it? these are not independent disks attached to the same device but disks in different devices in two geographically separate locations. No one would have faulted them if they stopped there but thankfully, they had an additional provider to backup their main backup provider, so kudos to them.

[jonsg]

Strange anti-cloud emphasis

"In the meantime, UniSuper's woes remain a lesson for companies leaping cloudwards. Someone clicking the wrong button, a previously unknown bug, an unforeseen series of events, or a combination of all three could have dire consequences for a business."

There is absolutely no difference in this, between being on own estate or in the cloud. Either way, someone can goof up, or a natural disaster can happen, or a systems failure, and bad stuff occurs.

Like any sensible and well-prepared company, UniSuper's IT had mitigated against as much of the risk as humanly possible. Not only had they replicated across two regions, they were replicating to a second supplier. If they were using own estate, doubtless they would have done the same, with replication across two geographically well-separated data centres and use of a separate backup domain, which is exactly analoguous, except it would have involved a heck of a lot more CapEx and having to manage multiple redundant network links through different providers, to avoid a single point of failure between the sites.

And yet this is an excuse to bash cloud use? Weird.

[Anonymous Coward]

Re: Strange anti-cloud emphasis

“There is absolutely no difference in this, between being on own estate or in the cloud.”

The Cloud: that's where you pay someone else to do software design, maintenance and upgrade on their hardware /s

[arsechitect]

Why so defensive ?

I don't see it as bashing the cloud use. The agility and ability to move fast is a double-edged sword that can break things easily.

"There is absolutely no difference in this, between being on own estate or in the cloud."

The difference is that when you send the "delete everything" command in the cloud it can be completed within minutes.

This is not the case for on-prem. Most likely you have to send the "delete everything" command separately to your server / storage / networking / infra

Backup most likely would be done by tape so deleting those backup would take days.

"If they were using own estate, doubtless they would have done the same"

Why are you making this assumption?

If they have the money to run in multi-cloud environment, then most likely they will have enough funds to fund renting rack-space in different locations.

On this specific incident, if UniSuper run on prem the damage would be contained and restoration could be within hours.

[UnknownUnknown]

Re: Strange anti-cloud emphasis

“UniSuper Google Cloud outage caused by an unfortunate series of events”

Literally or Metaphorically ?

[Roger Lipscombe]

An "inadvertent misconfiguration"?

"an "inadvertent misconfiguration" during the provisioning of UniSuper's Private Cloud services resulted in the deletion of the subscription."

This is word salad (I should know; I've written customer-facing incident reports in my time). Sounds to me like someone fat-fingered a Terraform plan (or similar) against the production account. I don't know why they're getting to shift some of the blame onto Google.

[Anon the mouse]

Re: An "inadvertent misconfiguration"?

They have a $125billion fund (not sure if AUS or USD), so enough money for Google to be willing to share in the blame.

[Anonymous Coward]

UniSuper

Where do they operate? What do they do? Presumably they are an organisation of some size..? There is a distinct lack of context in this article.

[notyetanotherid]

Re: UniSuper

... context is provided by the linked story earlier in the week: https://www.theregister.com/2024/05/08/google_cloud_misconfiguration_takes_australian/

[Benegesserict Cumbersomberbatch]

Re: UniSuper

Every Australian employer is obliged to put a fraction (currently 11%, up to a limit of $27500/year) of each employee's wages into a retirement superannuation fund which becomes accessible to the employee when they retire. This has been the case since the early 90s.

The funds under management as a result of this scheme reached AUD3.5 trillion in 2022.

Unisuper started as the industry fund for tertiary education. They are the 5th-biggest such fund and the $120b mentioned in the article is in AUD.

[Anonymous Coward]

Misconfiguration or malice?

With Google laying off people in an attempt to enhance shareholder value, is it possible that this is not misconfiguration at all but rather a deliberate act by a disgruntled former employee?