back to article Microsoft 'fesses to code blunder in Azure Container Apps

A code deployment for Azure Container Apps that contained a misconfiguration triggered prolonged log data access issues, according to a technical incident report from Microsoft. The incident, which began at 23:15 UTC on July 6 and ran until 09:00 the following day, meant a subset of data for Azure Monitor Log Analytics and …

  1. DJV Silver badge

    So...

    Microsoft STILL doesn't understand that test and production systems should be kept completely isolated from each other. Sheesh, when will they learn!

    1. Lil Endian Silver badge
      Facepalm

      Re: So...

      Sheesh, when will they learn! -- Hahahahaaa!

      That the SPOF was telemetry is most amusing to me. That the telemetry is a known SPOF that can bork-loop without raising huge red flags...? And all that nicely straddling test and production systems. Nice!

      Makes me wonder: Who telemetries the telemetry?!

      1. b0llchit Silver badge
        Coat

        Re: So...

        Who telemetries the telemetry?!

        Easy, the watched watchers watching watches watch ticking!

  2. Pascal Monett Silver badge
    Trollface

    So, fat-fingered configuration, again

    Seems to me that these cloud thingies work fine when humans don't interfere.

    Hmm, reminds me of BOFH somehow . . .

  3. FlamingDeath Silver badge

    Microturd

    Microturd and code blunder, imagine my shock

  4. that one in the corner Silver badge

    they interpreted this as a configuration change ... and restarted as well

    Why is it so hard for programmers to think of diff'ing the incoming config against the currently active one?

    No, no, let me guess: there is a timestamp field for when the config is sent out and because that has progressed by five to ten seconds...

    Sigh.

    1. OhForF' Silver badge

      Repeating the same actions and expecting different results...

      After a service fails and fails again within 5 seconds of the restart what is the chance it will run fine after another restart with the next 5 seconds?

      Micros~1 should have learned to (rate) limit the automatic restarts for services that fail directly after initialization a long time ago.

      1. Anonymous Coward
        Anonymous Coward

        Re: Repeating the same actions and expecting different results...

        As they failed for decades to write decent code I find people's expectation that they have a clue about decent deployment processes rather puzzling.

        Sigh.

    2. Lee D Silver badge

      Re: they interpreted this as a configuration change ... and restarted as well

      More importantly: Why is there not a little thing somewhere saying "Hey, this service that never normally restarts is suddenly doing it every 5-10 seconds"?

      Simple statistics to detect frequency changes and unusual behaviour like that.

      Or whoever is in charge of this global, business-critical, spans-all-customers, service should have at least some kind of alert and then go "WOAH! Everyone stop and tell me what's changed in the last 10 minutes!".

  5. ecofeco Silver badge

    Microsoft

    Microsoft gonna Microsoft.

    1. Anonymous Coward
      Anonymous Coward

      Re: Microsoft

      Why Gates decided to name the company after a personal anatomical detail I'll never understand..

  6. Will Godfrey Silver badge
    Happy

    Well, Well

    They didn't drop the ball this time... just refused to let go of it.

    1. Luiz Abdala
      Joke

      Re: Well, Well

      More like throwing the ball upwards every 10 seconds and grabbing it again.

      1. Will Godfrey Silver badge
        Thumb Up

        Re: Well, Well

        Good point.

        Oh, and it must have been one of those attached to elastic to make sure it came back

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like