back to article Microsoft says bug, sorry, 'a latent defect' in Safe Deployment Process system downed Azure Active Directory

Microsoft has blamed a software bug for the service disruption on Monday and Tuesday that affected customers using Azure Active Directory-dependent applications. From about three hours, from 2125 UTC (1425 PDT) on September 28, 2020 to 0023 UTC (1723 PDT) on September 29, 2020, Azure Active Directory (AD), Microsoft's cloud- …

  1. A Non e-mouse Silver badge
    Mushroom

    Outage Notifications

    What's more, it has a plan to prime its automated communications pipeline to get outage information to customers within 15 minutes, so they spend less time in the dark

    I've lost count of the number of times we've logged support calls with Microsoft and being told there are no service problems to suddenly seeing an announcement saying the issue (that we were having) has been resolved.

    1. sabroni Silver badge

      Re: Outage Notifications

      So this 15 minute turn around will be helpful then.

      What a bunch of bastards!

  2. Version 1.0 Silver badge
    Pint

    Just another day

    If you think about the cloudy environment that everyone is working in and using, then it's surprising and a real tribute to everyone that it works as well as it does. I'd buy them a beer.

    1. Joe W Silver badge

      Re: Just another day

      And that's why we don't do that in our company. Yet (I guess, as the license models et c. that allowed us to do certain things are being phased out, which poses many additional challenges - thank $(higher being) I'm not involved in that stuff).

  3. HildyJ Silver badge
    Devil

    New buzzword

    So now we have to add "latent defect" to the cloudy Buzzword Bingo Board.

    1. sanmigueelbeer Silver badge
      Thumb Up

      Re: New buzzword

      undocumented feature Latent defect

      New phrase. Gotcha.

      1. Anonymous Coward
        Anonymous Coward

        Re: New buzzword

        How about "Yet Another Microsoft Fuckup" or YAMFU?

      2. fidodogbreath Silver badge

        Re: New buzzword

        War is Peace

        Freedom is Slavery

        Ignorance is Strength

        Bug is Feature

        1. Shadow Systems

          Re: New buzzword

          OH! Oh! I wanna play!

          Microsoft is competent.

          Trump is sane.

          Wearing trousers is fine.

          Vogons are caring, compassionate, diplomatic, helpful, honest, & sociable!

          *Head explodes in sarcasm*

          1. Dan 55 Silver badge
            WTF?

            Re: New buzzword

            Wearing trousers is fine, isn't it? Or is today no-trousers day and nobody told me?

            1. My other car WAS an IAV Stryker Silver badge
              Devil

              Re: New buzzword

              With Shadow Systems, EVERY day is no-trouser day.

    2. Warm Braw Silver badge

      Re: New buzzword

      While undiscovered: latent defect

      When triggered: blatant defect

    3. LDS Silver badge

      Re: New buzzword

      Canon in its products calls them "phenomenon"... "fixes a phenomenon that may occur during..."

  4. Psmo
    Paris Hilton

    Anti-nominative determinsm

    I'd be wary of anything named the Safe Deployment Process.

    1. Anonymous Coward
      Anonymous Coward

      Re: Anti-nominative determinsm

      Sounds like something the coyote would buy from Acme.

      1. Chris G Silver badge

        Re: Anti-nominative determinsm

        The Acme version is agile, it can be adapted to deploy anvils or large boulders.

        1. TimMaher Silver badge
          Happy

          Re: Anti-nominative determinsm

          Does it come with a short fuse?

      2. martynhare

        Re: Anti-nominative determinsm

        Doesn’t let’s encrypt rely on ACME for validating certificate remewal entitlements?

    2. sanmigueelbeer Silver badge
      Happy

      Re: Anti-nominative determinsm

      I'd be wary of anything named the Safe Deployment Process.

      Duck season!

      1. bombastic bob Silver badge
        Trollface

        Re: Anti-nominative determinsm

        Duck season!

        Rabbit season!

        Duck season!

        Rabbit season!

        Rabbit season!

        Duck season!

        ...

    3. bombastic bob Silver badge
      Devil

      Re: Anti-nominative determinsm

      what if you're deploying actual (physical) Safes? You know, like those electronic 'smart' safes that transfer info to the bank. (when money and banking is involved, you can't have bugs)

      However, Micros~1 Cloud Services is apparently like a breeding ground for them.

  5. Andy The Hat Silver badge

    Is it just me ...?

    "The disruption occurred, Microsoft says, because a bug in its Azure AD's Safe Deployment Process rendered it unsafe: the safeguard pushed through a crash-inducing update into production, bypassing the usual verification process, and ultimately broke AD."

    I have no idea what this says. Can somebody translate it into English please?

    My take is that an update was applied which shouldn't have because it was buggy enough to ultimately take down the system *and* it was applied by protection software which was bug ridden enough to do exactly the opposite of what it was supposed to do ... or have I got the wrong end of a very sticky stick?

    1. Dan 55 Silver badge

      Re: Is it just me ...?

      They like testing in live on a subset of servers and seeing where the smoke rises, but it went wrong and it got sent to all servers and it turned into a dumpster fire.

    2. Anonymous Coward
      Anonymous Coward

      Re: Is it just me ...?

      > I have no idea what this says. Can somebody translate it into English please?

      I'm guessing it means that MS have a process that is supposed to ensure that changes to AD are done in a safe way. However there was a bug in the change process, rather than in the change itself, which then somehow broke the change or failed to apply it fully.

      A car analogy might be the manufacturer failing to fit the narrow-necked fuel filler pipe that prevents a large diameter nozzle diesel pump being put into an unleaded-only car. The prevention device is missing so the driver isn't prevented from accidentally filling-up with the wrong fuel.

  6. Pascal Monett Silver badge
    Angel

    Microsoft and its rings

    Everything has rings at Borkzilla. They have been well thought-out and cleverly implemented, with iron-clad protocols and processes to respect, and yet they regularly blow up and spew stuff over every ring at the same time.

    Windows Update had the same problem no too long ago, remember ? It was the Fast Ring update that spewed over every other ring and borked a whole slew of PCs.

    It's a very good idea, really. Except it's Borkzilla, so Murphy's Law stands proud.

    Icon for the ring, obviously.

    1. IT Hack

      Re: Microsoft and its rings

      Rings? Sounds a b it goatse-ey to me...

    2. TRT Silver badge

      Re: Microsoft and its rings

      Five rings for the Azure Lords, doomed to die...

  7. Eric Kimminau TREG
    Thumb Up

    ring slipped, spewage results

    So the spewage protection ring slipped allowing spewage over everything. Got it.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2021