back to article Microsoft Azure: It's getting hot in here, so shut down all your cores

Microsoft has warned that a "subset of customers in South Central US" may experience Azure problems today after cooling issues sent the servers scurrying for the shutdown button. The warning was first raised by Microsoft at 09:29 UTC as pretty much everything in the South Central US region went offline thanks to a temperature …

  1. Bronek Kozicki

    Ouch

    feeling hot here ...

  2. Anonymous Coward
    Anonymous Coward

    How Does Cloud Work Again?

    And the support page doesn't load either... @AzureSupport inform us, please!!

    The support page doesn't load? That's odd. I could have sworn that a properly designed and implemented cloud service could failover to a different region.

    1. Anonymous Coward
      Anonymous Coward

      Re: How Does Cloud Work Again?

      Ah yes, about all those promises of built in redundancy and no risk of single point of failure; just sign up to run all your IT in our cloud service and save lots of money - honest!

      1. Jay Lenovo
        Facepalm

        Re: How Does Cloud Work Again?

        If the cloud communicates at nearly the speed of light, I imagine the data is getting re-routed through the MS Data Center on Pluto.

        It takes light about a half a day to circle back.

        A hard day for keeping the nines.

        1. Pascal Monett Silver badge

          Re: "A hard day for keeping the nines"

          Well five nines was shot before Easter, four nines are in the toilet as well.

          The only question is how long will three nines last ?

          The Cloud (TM) is really not the best place to host your data.

        2. bombastic bob Silver badge
          Trollface

          Re: How Does Cloud Work Again?

          the cloud would work better if Micro-shaft could write EFFICIENT CODE like they USED to.

          Compare ".Net" and "UWP" to how snappy Win '95 was, by comparison. In 16Mb of memory even!!!

          And that pretty much explains it all. Micro-shaft, STOP it with the BLOATWARE! Abandon ".Not", "UWP", "The Metro", C-pound, and THE SPYWARE!!! [then you might find your servers won't overheat because they're no longer working against themselves, ya know???]

          MS fanboi downvotes welcome. heh. But, you *KNOW* I'm *RIGHT* about this!!!

          1. Arctic fox
            Headmaster

            Re: How Does Cloud Work Again?

            Actually Bob I strongly suspect that your downvote totals have at least as much to do with your obsessive use of caps lock as your opinionate certainties.

          2. deadlockvictim

            Re: How Does Cloud Work Again?

            Bomb. Bob» the cloud would work better if Micro-shaft could write EFFICIENT CODE like they USED to.

            When was this exactly? I've been using Microsoft products since Excel v2 on the Mac way back in the early nineties. And even then, Microsoft products were resource hogs.

            When I think about it now, there are only 2 Microsoft products worth paying for: Excel & SQL Server. Maybe Windows Server.

            I do agree with the sentiment of your post though.

          3. Anonymous Coward
            Anonymous Coward

            Re: How Does Cloud Work Again?

            @Mr. Attention-Whoring-Random-Caps-Gimmick; As the other person noted, Microsoft weren't noted for their resource efficiency, even back then.

            If Windows 95 was "snappy" at the time, that's probably as much because it was doing much less by modern standards. And also because despite the improved multitasking and better interface, it was still based on MS-DOS (#) and Windows 3.1 under the skin- rather than the NT-based newer versions- which would be grossly unsuitable for any remotely modern use.

            I'm the last person you could accuse of being an MS "fanboi"- I loathe them more than ever with the blatant spyware that is Windows 10- and it's true that modern software is often horribly efficient in general- but your (tediously bombastic) nostalgia says nothing of real insight either.

            FFS, if efficiency was all that mattered, we'd ditch the OS altogether and hit the bare metal directly. There's a reason that's gone out of fashion, though...

            (#) i.e. A ripoff/clone/workalike of CP/M (a minimal- by modern standards- OS designed in the mid-70s for early 8-bit CPUs) which required numerous subsequent modifications in order to overcome its obsolete-at-birth architecture and ended up being a pig-fugly hacked about mess.

          4. RyokuMas
            FAIL

            Re: How Does Cloud Work Again?

            "Abandon ".Not"... C-pound..."

            Name-calling aside: what, pray tell, would you propose as an alternative to these? C# continues to rank highly in the most in-demand programming languages (usually around #5 in the charts right now).

            Unless of course you have some magical way of instantly training all these .NET developers...

      2. TechDrone

        Re: How Does Cloud Work Again?

        Cloud providers give you the tools to build redundant services, but don't do anything automatically. And since you have to pay extra to get the redundancy, you end up spending even more money to protect yourself from your suppliers failings.

        From the cloud suppliers point of view I guess it's doubles all round.

    2. phuzz Silver badge

      Re: How Does Cloud Work Again?

      Really, Microsoft should be using AWS to host their status page (and vice versa). It's the only way to be sure that you can survive a complete failure of all your systems, by using your competitors.

  3. Wellyboot Silver badge
    Happy

    XKCD>>

    https://xkcd.com/908/

    natch

    1. Anonymous Coward
      Anonymous Coward

      Re: XKCD>>

      https://xkcd.com/7/

  4. Doogie Howser MD

    Marketplace is shafted too

    "The portal is having issues getting authentication tokens for Microsoft_Azure_Marketplace."

    For the love of fuck.

  5. Korev Silver badge
    Coat

    Over heating datacentres? Did they let the Hotmail in again?

  6. MrKrotos

    Hahahahahahahha

    Ahahahahahahhahaha

    That is all :P

  7. Charlie Clark Silver badge
    Coat

    Should've used DeepMind?

    To manage the temperature at the data centres.

  8. MikeGH

    " 16:35 UTC" isn't that a really odd deadline to set?

    1. dank_army

      The first status update was 12:35 UTC - ms updates every 4 hours

    2. francis.mondia.et

      I think its gonna give them enough time for the sysadmin dads to tell their families dad's not gonna be home for dinner (and possibly breakfast) ;)

  9. PghMike

    Auth services affecting multiple regions

    Apparently a lot of login requests get forwarded through the South Central region, so other regions may be affected. I'm sitting here in Pittsburgh and some things are messed up on some networks.

  10. Steve Davies 3 Silver badge

    So MS expect us to trust them with our Business?

    Oh sorry, MS does not work on Labor Day.

    All is good then.

    Nah. /s /s /s

  11. Anonymous Coward
    Anonymous Coward

    Reputation Damage

    Outages seams to be an on-going issue, specifically with Azure and not other providers in terms of frequency.

    This is going to start to impact the reputation of Azure if they keep messing up.

    1. Claptrap314 Silver badge

      Re: Reputation Damage

      Maturity matters. Lot at Amazon's failure rate at this stage, and you will see that had a lot more. Of course, we expect to learn from each other's mistakes.

      Of course, Microsoft has a reputation for poor quality, and life is not fair, so....

    2. PeterM42
      Alert

      Re: Reputation Damage

      Er, .......Does Azure have a good reputation?

      1. cob2018

        Re: Reputation Damage

        Not to my knowledge. However, I think that it is possible for them to go from bad to worse.

        1. hplasm
          Coat

          Re: Reputation Damage

          Azure wish I'd chosen AWS!

  12. Claptrap314 Silver badge

    Cascading Failure

    This mode of failure strikes me as rather odd. It should not be overly difficult to engineer a cooling solution to a data center with 10% excess capacity, and this was hardly the hottest day of the year.

    If they're pushing things to the limit, however, and shifting loads into other data centers....

    1. swschrad

      how Big Cooling works

      it is highly likely that there is a big-ass installation of cooling water assemblies outside to dump the center heat. I humbly suggest that San Antonio also being nicely warm, the filters inside before the actual air conditioning chillers are slathered with algae, having cleaned them before in other places. guys, stagger your maintenance so they don't all clog at the same time, ok?

  13. Martin M

    Ouch

    But the thing that made me chuckle was the inline ad: “Azure - migrate your on-premises workloads to the cloud with confidence.”

    Might want to pause that campaign for the moment, marketing peeps...

    1. Dan 55 Silver badge

      Re: Ouch

      Why? It's not false marketing, you can be confident Azure's going to go down.

      1. Ken Moorhouse Silver badge

        Re: It's not false marketing

        “Azure - migrate your on-premises workloads to the cloud with confidence.”

        It is if it is the process of migration itself that is failing.

  14. Anonymous Coward
    Anonymous Coward

    As firms become more and more Cloud dependant

    How good will Microsoft be at scaling things up, anyone wonder...

    Will Cloud reliability be 50% marketing.... 49% fingers crossed.

    1% delivered reliability as promised on vendor favored contracts.

    1. Anonymous Coward
      Anonymous Coward

      Every single momentary cloud hiccup or lag should count as part of total reliability / uptime

      Right now that granularity is massaged out of the official numbers and presentations given by 'Someone Else's Computer' salesmen. Blaming old in-house infrastructure and external Telcos comes first anyway.

      Next time CEO's are pitched Cloud services, they should ask why everything from e-Sports to Fortnite doesn't run in the Cloud... Its telling how competitive video gaming is years away from being hosted there.

  15. khanmo01

    Down for me at the East Coast

    Came to work downtown Manhattan at 9 am. Its now past 3 pm. Can confirm that I have no access to Sharepoint or Teams all day.

    1. deadlockvictim

      Re: Down for me at the East Coast

      No access to Sharepoint? Oh dear.

  16. whitepines
    FAIL

    So will anyone be getting refunds for damages, loss of business, etc.? Or even refunds for the time the service (Office 365, nay Office 364, ah well let's just call it Office Sometime) was out?

    Or is this another area where perhaps some new legislation for rights of the tenant needs to happen?

    Of course, since we use our own, open source systems, no effect here other than an uncontrollable urge to laugh at those glued to Microsoft's "too good to be true" deals!

    1. Anonymous Coward
      Anonymous Coward

      Why is it the overall health is addressed in the portal but not your organization specifically?

      Whether it is a large or small subset of users, down is down.

      Knowing a majority may be doing fine, doesn't address my business obligations.

  17. hayseed

    The American South is not the UK

    > At one point OneDrive for Business had also gone for some tea and biscuits.

    Iced sweet tea (probably with a little lemon) and gravy biscuits (a baked bread) here.

  18. Anonymous Coward
    Anonymous Coward

    Up, but doesn't know it

    Server running in West US 2, had no issues all day. But, the monitoring and logs for that server have an almost 4 hour gap where all stats went to zero.

    First saw that zero flat-line and almost choked. Got onto the server and checked the logs there, not a blip.

  19. Anonymous Coward
    Anonymous Coward

    Our Azure services are hosted in Europe, so in theory haven't been affected by this outage.

    For some unexplained reason though, our hosted Mac build servers are in the US. And yep, they've been down all day.

    1. Anonymous Coward
      Anonymous Coward

      A little update... once the affected machines were back up and running, according to Microsoft, there were 123 million builds queued in that data centre.

  20. Sir Loin Of Beef

    Now that your stuff is in the "cloud" you get to wait for someone else to fix it. Good luck with those wait times!

  21. JSIM

    Azure Blues.

    Who's going to write the song? It's about time for a song.

    1. Doug 4

      The song has already been done.

      Driftin' Dreamin'

      In an Azure mood,

      Stardust gleamin'

      Thru my solitude:

      Here in my seclusion,

      You're a blue illusion

      While I'm in this Azure interlude.

      I'm not wanted I'm so all alone;

      Always haunted

      By the dreams I own;

      But, though Im tormented

      I must be contented

      Driftin' Dreamin'

      In an Azure mood!

      Driftin' Dreamin'

      In an Azure mood,

      Star dust gleamin'

      Thru my solitude:

      Here in my seclusion,

      You're a blue illusion

      While I'm in this Azure interlude!

      Azure lyrics © EMI Music Publishing

  22. Anonymous Coward
    Anonymous Coward

    Follow the sun...

    One time where 'follow the sun' isn't such a good idea!

  23. fedoraman
    Coat

    Blue Sky Thinking

    It always amused me to think that you get most Azure (i.e. clear blue sky), when there are *no* clouds

  24. WibbleMe

    What happend to the technology of making blocks of ice and running pipes over them, if the cloud runs on hot air then it is

  25. one crazy media

    Cheap off-the-shelf hardware. You get what you paid for. Next shut down will be caused some crappy software developed using Agile one-line stories.

    Who came up with the bright idea that one can develop quality (I know quality and software, right!) software using Agile.

    Yep, Agile is great for changing products and prices on an E-commerce web site.

  26. sebbb

    What about boffins complaining though?

    I mean, I'm amazed about how people are shouting at Azure Support on twitter like "It is unacceptable!!!" "Our production system has been down for 10 hrs!!".

    Did you not think that you should have had redundancy across multiple regions for your production systems (I'm talking about self-engineered solutions of course)? Anything can fail anytime, spend that money or keep off those "clouds".

  27. steviebuk Silver badge

    "We want....

    ....to be Infrastructure free. The cloud never goes down".

    Yeah right. Considering our own data centre hadn't been down in probably over a year. The only other time it went down was through hardware failure of new kit out of our control. Ooo, bit like how the cloud is out of our control.

    When will they realise the cloud is just "Someone else's computer you have no control over" and if you are small fish, you're at the bottom of the support list of companies to get back online.

    And "Infrastructure free" isn't a fucking term that exists! Stop making shit up!

  28. Snarf Junky

    Well that's a bit of a balls up isn't it

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like