back to article Exchange Online and Microsoft Teams went down in APAC because Microsoft broke itself

Microsoft's flagship cloudy productivity services are down across the Asia-Pacific region. "Our initial investigation indicates that there our service infrastructure is performing at a sub-optimal level, resulting in impact to general service functionality" states an advisory time-stamped 12:41PM on December 2. The incident …

  1. Pirate Dave Silver badge

    Wasn't there a major fiber cut in Egypt a few days ago? I wonder if Microsoft's infrastructure is responding poorly to that...

    1. Anonymous Coward
      Anonymous Coward

      More likely they just install the monthly security updates.

      1. Tim99 Silver badge
        Trollface

        Er, I don’t think it’s Tuesday…

  2. ecofeco Silver badge

    Bwahahahahahaha

    Never change Microsoft, you crazy, fun loving pranksters!

    By which I mean, you useless twats.

    1. NATTtrash
      Trollface

      Re: Bwahahahahahaha

      Never change indeed.

      Microsoft appears not to know what's wrong.

      'nuff said... So maybe..?

      We have successfully restarted a small portion of the affected systems...

      Ah... "Did you turn it on and off already?"

  3. Mayday
    Windows

    "Microsoft’s flagship cloudy productivity services are down across the Asia-Pacific region."

    So we are at normal service levels then?

  4. Lil Endian Silver badge
    Coat

    Have you tried...

    ...turning it off and on again?

    You are Roy Trenneman, and I claim my £100 winnings!

    Thank you! Jen x

    [Mine's the one with Grand Designs in the pocket!]

    1. Zippy´s Sausage Factory
      Joke

      Re: Have you tried...

      This will help. Track 5: "Coffin Fodder". It got me through some dark times...

  5. Anonymous Coward
    Anonymous Coward

    "Put it in the cloud", they said

    "It'll be more resilient", they said

  6. A Non e-mouse Silver badge

    For those that don't realise, Teams is heavy reliant on Exchange Online. Problems with Exchange will always cause problems for Teams.

    1. chivo243 Silver badge
      Devil

      I've never touched Teams, and Exchange 2003 (circa 2005) was my last dealing with the devil! I guess, I won't miss out on anything...

      1. 43300 Silver badge

        On-prem Exchange was much less hassle than cloudy Exchange! Once on-prem was set up it normally just worked other than the occasional dodgy patch (which could usually be identified and removed fairly quickly). Cloudy Exchange is a law unto itself and you can never predict when they might break it, or introduce an "improvement" which causes something you use to stop working.

        1. Trixr

          Co-signed. Exchange 2010 was fine, and 2016 was when everything you wanted in terms of HA and so on worked very nicely. I had more uptime over my 16 years managing Exchange on-prem than MSFT has managed to achieve for one calendar year in the APAC region.

          I had one server stop delivering mail for about 6 hours - small region, about 10% of the user base (fixed with a stinking reboot, after exhausting every other troubleshooting option). And that is literally all, other than dodgy CUs, as you say.

  7. Roger Kynaston
    Happy

    DNS?

    it is always DNS. Someone likely updated a zone file incorrectly.

    1. Anonymous Coward
      Anonymous Coward

      Re: DNS?

      We discovered a misconfiguration in some of Office 365's DNS records. We logged a support call and it took MS several months to fiix their DNS. If it is DNS, don't hold your breath for MS to fix it quickly.

      1. Strahd Ivarius Silver badge

        Re: DNS?

        DNS = Network

        Microsoft = Software

        enough said...

  8. Pascal Monett Silver badge
    Coat

    "this incident is getting worse"

    So, the rot is spreading.

    When will it become sentient ?

    Mine's the one with the sentient yoghurt in the pocket.

    1. Anonymous Coward
      Anonymous Coward

      Re: "this incident is getting worse"

      I think sentience must have occurred in the non-MS parts of the infrastructure (because, let's face it, with Microsoft code you have at least no worries about accidentally creating SkyNet) and they have taken action to isolate the infection.

      At present, MS representatives are busy bribing the owners of the functioning infrastructures to implement .Net or similar viruses, so disservices will be resumed shortly.

      Yes, I haven't had my coffee yet. Why do you think? Grr.

    2. Fred Flintstone Gold badge
      Thumb Up

      Re: "this incident is getting worse"

      Well, at least you have culture..

      Have a nice weekend :).

  9. Kevin Johnston

    Just had Teams go silent here in a EMEA based system, is it too much to hope that is linked so I can spend the day working instead of sat on meaningless calls/chats?

    Damn, barely had time to click Submit before 'it came back'...there goes today's achievements

  10. Dan 55 Silver badge

    "Users may be unable to search Microsoft Teams"

    No change there then, but at least they're finally admitting it.

    1. Captain Scarlet
      Coffee/keyboard

      Re: "Users may be unable to search Microsoft Teams"

      I feel for the poor souls who found something and then spent an age trying to get back to the end of a massive chat window

      1. David 132 Silver badge
        Thumb Up

        Re: "Users may be unable to search Microsoft Teams"

        Ah, so it’s not just me then who thinks that the Teams UI sucks donkey balls through a small straw in terms of navigability.

  11. TaabuTheCat

    Advice given should be advice taken

    Please run sfc /scannow and if that doesn't fix it please reinstall the operating system.

    1. Strahd Ivarius Silver badge

      Re: Advice given should be advice taken

      I think a gdprupdate /force is needed somewhere

  12. that one in the corner Silver badge

    Which "State level hacking team"

    are going to get the blame this time?

    "Attacking Microsoft is attacking Global Peace. This appears to be a new player, calling itself 'The Jason Malconfig Line 12'"

    1. P. Lee

      Re: Which "State level hacking team"

      Russians.

  13. Antron Argaiv Silver badge
    Pint

    I love carefully crafted org-speak

    "Our initial investigation indicates that there our service infrastructure is performing at a sub-optimal level, resulting in impact to general service functionality"

    Nice work, that.

  14. This post has been deleted by its author

  15. Bearshark

    New verbiage for sh*t being down

    Network Infrastructure DOWN...

    We don't like the word DOWN anymore. Here's the new industry verbiage:

    "service infrastructure is performing at a sub-optimal level"

    Brilliant! Ship it!

    1. 43300 Silver badge

      Re: New verbiage for sh*t being down

      'Down' is negative, and modern software marketing people don't like negatives. See also the prompts trying to get you to agree to something: 'Yes' or 'Maybe later'. There isn't a 'fuck off and don't ask me again' option.

  16. AVR

    My memories of Exchange servers are a lot worse than that. Some fairly catastrophic problems caused by putting too much into public folders for one, some really awkward problems relating to starting up a new server when the others went down in an earthquake, occasional slow synchronisation and message transfer between servers. It was a while ago I was involved with that admittedly.

  17. Brett Weaver

    "Legacy"

    We got rid of the team that supported the software that eventually failed. Cost our customers, but we've only been out of university for six months so anything that works but we don't understand is "legacy"

    1. Trixr

      Re: "Legacy"

      I am very very prepared to bet that it was something to do with basic auth, which they recently deprecated with extreme prejudice. If so, the irony is rich.

  18. PeterM42
    Facepalm

    Ah! - Is that anything to do with......

    ..... Office 364½ ?

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like