back to article Oracle Cloud, Netsuite, and Azure go down, hard, Down Under

Oracle, Netsuite, and Microsoft's clouds have gone down, hard, in the Sydney, Australia, region likely due to an issue at a datacenter provider in which both are tenants. The Big Red Cloud first advised customers of an outage at 2129 Sydney time (1229 UTC) on Wednesday, and 29 minutes later wrote to inform customers that the …

  1. formerpommie
    FAIL

    It's not just Oracle

    I'm at the brown end of this and can tell you that (at least) Microsoft Azure and SAP have also been impacted by this outage.

    It has provisionally been attributed to a cooling failure caused by a power surge. (this may change)

  2. Anonymous Coward
    Anonymous Coward

    One of our clients logged a ticket about a DB error in Micropay. Looking at the Micropay IP address they are in Sydney & MS are reporting SQL services are still having problems.

    Today is payday, so they are not very happy.

  3. Clausewitz4.0 Bronze badge
    Black Helicopters

    Linkedin was also down, actually terribly slowyy

    Linkedin was also down a few hours ago - probably this was the cause

  4. Somone Unimportant

    I seem to recall a Sydney-based datacentre being flooded some years ago, with much the same effect.

    tap tap tap...

    Yep. AWS Sydney, June 2016.

    Wonder if it's the same site?

    1. nullacritter

      June 6th, 2016, and the flooding was at a nearby zone substation that caused the Colocation providers' DRUPS to try to feed the substation. I don't know what site caused this issue, but that one was not it.

  5. Anonymous Coward
    Anonymous Coward

    My company too.

    My company was affected by this too. Our monitoring systems are Azure hosted and were hosed too. We should move one monitoring systems to a different DC

    1. richardcox13

      Re: My company too.

      > We should move one monitoring systems to a different DC

      No, you should be running in more than one DC, or at least more than one AZ in the DC.

      A different DC will have its own set of vulnerabilities to an outage. As long as you are in a single AZ you will go down with that AZ.

  6. eldakka
    Facepalm

    The Big Red Cloud first advised customers of an outage at 2129 Sydney time (1229 UTC) on Wednesday, and 29 minutes later wrote to inform customers that the outage had started earlier than its first emailed advisory – at 1015 UTC.

    Oracle’s second email delivered the mixed message that: “We are still investigating an issue in the Australia East (Sydney) region that is impacting multiple OCI services. We have identified root cause of service failures and are working to mitigate the issue.”

    Well, not for those who host their email in those data centres.

  7. deadlockvictim

    Common connection

    Is there a common connection?

    is the single cause of failure Sydney's electrical grid?

  8. Anonymous Coward
    Anonymous Coward

    At least Azure Availability Zones work as documented (unlike Google's) - all the impacted VMs in AU East I'm responsible for all failed over correctly to one of the other functional AZs.

    It seems most of these outages are due to design and cost choices at the customer layer rather than a complete failure of cloud substrate. If as a customer you're not considering a DC as a point of failure then you will have an unplanned outage at some point during your operations.

  9. yogidude

    The cost of doing business

    "proactively powered down a small subset of selected compute and storage scale units, to avoid damage to hardware."

    So the combined penalties that Azure might fork out for an outage on their cloud is less than the cost of a selection of compute and storage. Nah. I reckon they powered down that hardware because the data on that hardware might have become irretrievable. Hardware can be replaced. Data is a different liability altogether

    Sounds like we haven't heard the whole story. Mind you after Azure token signing keys were exfiltrated earlier this year, we are getting used to not hearing the whole story from Microsoft.

    1. Anonymous Coward
      Anonymous Coward

      Re: The cost of doing business

      I'm sure they are strategising right now on what to say, it's what businesses do

      Avoid accountablilty where possible, profit

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like