Microsoft says bug, sorry, 'a latent defect' in Safe Deployment Process system downed Azure Active Directory

Microsoft has blamed a software bug for the service disruption on Monday and Tuesday that affected customers using Azure Active Directory-dependent applications. From about three hours, from 2125 UTC (1425 PDT) on September 28, 2020 to 0023 UTC (1723 PDT) on September 29, 2020, Azure Active Directory (AD), Microsoft's cloud- …

[A Non e-mouse]

Outage Notifications

What's more, it has a plan to prime its automated communications pipeline to get outage information to customers within 15 minutes, so they spend less time in the dark

I've lost count of the number of times we've logged support calls with Microsoft and being told there are no service problems to suddenly seeing an announcement saying the issue (that we were having) has been resolved.

[sabroni]

Re: Outage Notifications

So this 15 minute turn around will be helpful then.

What a bunch of bastards!

[Version 1.0]

Just another day

If you think about the cloudy environment that everyone is working in and using, then it's surprising and a real tribute to everyone that it works as well as it does. I'd buy them a beer.

[Joe W]

Re: Just another day

And that's why we don't do that in our company. Yet (I guess, as the license models et c. that allowed us to do certain things are being phased out, which poses many additional challenges - thank $(higher being) I'm not involved in that stuff).

[Anonymous Coward]

New buzzword

So now we have to add "latent defect" to the cloudy Buzzword Bingo Board.

[sanmigueelbeer]

Re: New buzzword

undocumented feature Latent defect

New phrase. Gotcha.

[Anonymous Coward]

Re: New buzzword

How about "Yet Another Microsoft Fuckup" or YAMFU?

[fidodogbreath]

Re: New buzzword

War is Peace

Freedom is Slavery

Ignorance is Strength

Bug is Feature

[Shadow Systems]

Re: New buzzword

OH! Oh! I wanna play!

Microsoft is competent.

Trump is sane.

Wearing trousers is fine.

Vogons are caring, compassionate, diplomatic, helpful, honest, & sociable!

*Head explodes in sarcasm*

[Dan 55]

Re: New buzzword

Wearing trousers is fine, isn't it? Or is today no-trousers day and nobody told me?

[My other car WAS an IAV Stryker]

Re: New buzzword

With Shadow Systems, EVERY day is no-trouser day.

[Warm Braw]

Re: New buzzword

While undiscovered: latent defect

When triggered: blatant defect

[Anonymous Coward]

Re: New buzzword

Canon in its products calls them "phenomenon"... "fixes a phenomenon that may occur during..."

[Psmo]

Anti-nominative determinsm

I'd be wary of anything named the Safe Deployment Process.

[Anonymous Coward]

Re: Anti-nominative determinsm

Sounds like something the coyote would buy from Acme.

[Chris G]

Re: Anti-nominative determinsm

The Acme version is agile, it can be adapted to deploy anvils or large boulders.

[TimMaher]

Re: Anti-nominative determinsm

Does it come with a short fuse?

[martynhare]

Re: Anti-nominative determinsm

Doesn’t let’s encrypt rely on ACME for validating certificate remewal entitlements?

[sanmigueelbeer]

Re: Anti-nominative determinsm

I'd be wary of anything named the Safe Deployment Process.

Duck season!

[bombastic bob]

Re: Anti-nominative determinsm

Duck season!

Rabbit season!

Duck season!

Rabbit season!

Rabbit season!

Duck season!

...

[bombastic bob]

Re: Anti-nominative determinsm

what if you're deploying actual (physical) Safes? You know, like those electronic 'smart' safes that transfer info to the bank. (when money and banking is involved, you can't have bugs)

However, Micros~1 Cloud Services is apparently like a breeding ground for them.

[Andy The Hat]

Is it just me ...?

"The disruption occurred, Microsoft says, because a bug in its Azure AD's Safe Deployment Process rendered it unsafe: the safeguard pushed through a crash-inducing update into production, bypassing the usual verification process, and ultimately broke AD."

I have no idea what this says. Can somebody translate it into English please?

My take is that an update was applied which shouldn't have because it was buggy enough to ultimately take down the system *and* it was applied by protection software which was bug ridden enough to do exactly the opposite of what it was supposed to do ... or have I got the wrong end of a very sticky stick?

[Dan 55]

Re: Is it just me ...?

They like testing in live on a subset of servers and seeing where the smoke rises, but it went wrong and it got sent to all servers and it turned into a dumpster fire.

[Anonymous Coward]

Re: Is it just me ...?

> I have no idea what this says. Can somebody translate it into English please?

I'm guessing it means that MS have a process that is supposed to ensure that changes to AD are done in a safe way. However there was a bug in the change process, rather than in the change itself, which then somehow broke the change or failed to apply it fully.

A car analogy might be the manufacturer failing to fit the narrow-necked fuel filler pipe that prevents a large diameter nozzle diesel pump being put into an unleaded-only car. The prevention device is missing so the driver isn't prevented from accidentally filling-up with the wrong fuel.

[Pascal Monett]

Microsoft and its rings

Everything has rings at Borkzilla. They have been well thought-out and cleverly implemented, with iron-clad protocols and processes to respect, and yet they regularly blow up and spew stuff over every ring at the same time.

Windows Update had the same problem no too long ago, remember ? It was the Fast Ring update that spewed over every other ring and borked a whole slew of PCs.

It's a very good idea, really. Except it's Borkzilla, so Murphy's Law stands proud.

Icon for the ring, obviously.

[IT Hack]

Re: Microsoft and its rings

Rings? Sounds a b it goatse-ey to me...

[TRT]

Re: Microsoft and its rings

Five rings for the Azure Lords, doomed to die...

[Eric Kimminau TREG]

ring slipped, spewage results

So the spewage protection ring slipped allowing spewage over everything. Got it.