back to article Salesforce fell over so hard today, it took out its own server status page

Salesforce is digging itself out of a multi-hour outage right now that it has blamed on a DNS issue. At one point today, the IT breakdown was so severe that its status page was pretty much inaccessible for netizens, and staff resorted to posting updates on their help and training sub-site. "Salesforce is experiencing a major …

  1. Anonymous Coward
    Anonymous Coward

    It was the engineer in the DC with the keyboard

    On one of the webcasts they said that it was an engineer deploying a emergency DNS fix across the network. The 'fix' took down most of their datacenters.

    Scary how a emergency fix is rolled out globally in one go...

    1. Tomato Krill

      Re: It was the engineer in the DC with the keyboard

      Coming soon to an episode of Who Me?….

    2. Michael Hoffmann
      Coat

      Re: It was the engineer in the DC with the keyboard

      Forgive me Paul Vixie for I too have sinned. Let s/he who hath never Quickly Changed An NS Record Before Nipping Off To The Pub throw the first kill -HUP!

      Icon is putting my coat back on at the pub after getting called back to the office.

      1. Korev Silver badge
        Coat

        Re: It was the engineer in the DC with the keyboard

        That's a dead CERT

    3. Korev Silver badge
      Coat

      Re: It was the engineer in the DC with the keyboard

      >On one of the webcasts they said that it was an engineer deploying a emergency DNS fix across the network.

      Will they CNAME and shame?

      1. Korev Silver badge
        Coat

        Re: It was the engineer in the DC with the keyboard

        I'm struggling to think of A nother DNS pun now

        1. Steve K

          Re: It was the engineer in the DC with the keyboard

          That's some sort of A RECORD?

        2. Chris J

          Re: It was the engineer in the DC with the keyboard

          Salesforce in a bit of a BIND

      2. Anonymous Coward
        Anonymous Coward

        Re: It was the engineer in the DC with the keyboard

        Someone will soon PTR finger at the culprit.

        1. MyffyW Silver badge

          Re: It was the engineer in the DC with the keyboard

          Might make my leadership team perform a reverse lookup ferret on their enthusiasm for Salesforce. SRV them right.

          [who am I trying to kid, they'll still sign-up]

    4. JimC

      Re: It was the engineer in the DC with the keyboard

      DNS that doesn't roll out globally across the org makes for extremely bizarre faults! I remember having some issues in the early days of Microsoft DNS.

    5. tip pc Silver badge
      Thumb Up

      Re: It was the engineer in the DC with the keyboard

      There is an “On Call” or “Who, me” webcast?

    6. Cederic Silver badge

      Re: It was the engineer in the DC with the keyboard

      Welcome to the world of DevOps, or, if you want to be paid a little more, DevSecOps.

      This is why continuous delivery never became popular in the Finance industry.

    7. yetanotheraoc Silver badge

      Re: It was the engineer in the DC with the keyboard

      "an engineer deploying a emergency DNS"

      The part of the manager who told him to go ahead with the fix will by played by William Atherton. He's always excellent playing the character the audience loves to hate.

      W.A. "I'm ordering you to make that change.

      Engineer "Are you sure? This hasn't been tested."

      W.A. "Do it now."

  2. nintendoeats Silver badge

    We really need to come up with another acronym for either Domain Name Service or Denial of Service. This article wasn't quite what I was expecting :p

    1. SgtFalstaff

      Sometimes they are one and the same

  3. Bibbit

    Business-speak I’ve missed?

    I’ve not heard “out of impact” before? Is this a new one or have I just been living in a cave?

    I’m still not comfortable with the phrase “going forward” so it’s probably just me.

    1. Pascal Monett Silver badge

      Re: Business-speak I’ve missed?

      You're on the cusp, the bleeding edge.

      It's a bit like Preschoolers, they blabla and they blabla and, every now and then, they make a new noise they like.

      Meanwhile, the adults translate their sounds into actual words.

    2. Anonymous Coward
      Anonymous Coward

      Re: Business-speak I’ve missed?

      @Bibbit

      No it isn't just you. I have never heard of it either. It's just more pseudo intellectual bullshit made up by people who want to sound knowledgeable. My pet hate is "reached out".

      1. Tabor

        Re: Business-speak I’ve missed?

        If your pet hate is “reached out”, that’s because nobody has “circled back” to you yet. First heard that one 7 years ago, and I still haven’t encountered anything in the wild that makes me shudder more.

        1. Throatwarbler Mangrove Silver badge
          Trollface

          Re: Business-speak I’ve missed?

          Let's sync up offline and have a sidebar conversation. I'm sure I can meet your ask for more egregious business jargon.

          1. MyffyW Silver badge
            Coat

            Re: Business-speak I’ve missed?

            I would invite you to come for a quick scuba in my thinktank, but I'm not sure we should be sharing the same corporate waterfall during COVID-19.

            [Coat's already on, zipped up, hood up, ready for a deluge]

            1. RegGuy1 Silver badge

              Re: Business-speak I’ve missed?

              "Now, as you all know, I’m not here..."

              https://en.wikipedia.org/wiki/Drop_the_Dead_Donkey

            2. TeeCee Gold badge

              Re: Business-speak I’ve missed?

              Excellent. I admire the way your little rivulets of analogy combined to form a veritable ocean of truth.

              1. Psmo

                Re: Business-speak I’ve missed?

                I thought you weren't supposed to merge the streams in case of splashback?

        2. sorry, what?

          Re: Business-speak I’ve missed?

          @Tabor, for me it's the Indian-centric abuse of the word "Revert". They use it to mean "respond", but that's not what it means... #sigh#

    3. Anonymous Coward
      Anonymous Coward

      Re: Business-speak I’ve missed?

      Whenever I hear the word "utilize" I quit listening. USE it or lose your audience.

  4. Anonymous Coward
    Joke

    If only...

    If only Salesforce had some sort of ticketing software available to them whereby a proposed change like this could be logged and the risk and impact of a mistake fully evaluated as part of the approval process.

    1. Anonymous Coward
      Anonymous Coward

      Re: If only...

      Have you seen the price of that software? Who's going to pay that?

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like