Salesforce fell over so hard today, it took out its own server status page

Salesforce is digging itself out of a multi-hour outage right now that it has blamed on a DNS issue. At one point today, the IT breakdown was so severe that its status page was pretty much inaccessible for netizens, and staff resorted to posting updates on their help and training sub-site. "Salesforce is experiencing a major …

[Anonymous Coward]

It was the engineer in the DC with the keyboard

On one of the webcasts they said that it was an engineer deploying a emergency DNS fix across the network. The 'fix' took down most of their datacenters.

Scary how a emergency fix is rolled out globally in one go...

[Tomato Krill]

Re: It was the engineer in the DC with the keyboard

Coming soon to an episode of Who Me?….

[Michael Hoffmann]

Re: It was the engineer in the DC with the keyboard

Forgive me Paul Vixie for I too have sinned. Let s/he who hath never Quickly Changed An NS Record Before Nipping Off To The Pub throw the first kill -HUP!

Icon is putting my coat back on at the pub after getting called back to the office.

[Korev]

Re: It was the engineer in the DC with the keyboard

That's a dead CERT

[Korev]

Re: It was the engineer in the DC with the keyboard

>On one of the webcasts they said that it was an engineer deploying a emergency DNS fix across the network.

Will they CNAME and shame?

[Korev]

Re: It was the engineer in the DC with the keyboard

I'm struggling to think of A nother DNS pun now

[Steve K]

Re: It was the engineer in the DC with the keyboard

That's some sort of A RECORD?

[Chris J]

Re: It was the engineer in the DC with the keyboard

Salesforce in a bit of a BIND

[Anonymous Coward]

Re: It was the engineer in the DC with the keyboard

Someone will soon PTR finger at the culprit.

[MyffyW]

Re: It was the engineer in the DC with the keyboard

Might make my leadership team perform a reverse lookup ferret on their enthusiasm for Salesforce. SRV them right.

[who am I trying to kid, they'll still sign-up]

[JimC]

Re: It was the engineer in the DC with the keyboard

DNS that doesn't roll out globally across the org makes for extremely bizarre faults! I remember having some issues in the early days of Microsoft DNS.

[tip pc]

Re: It was the engineer in the DC with the keyboard

There is an “On Call” or “Who, me” webcast?

[Cederic]

Re: It was the engineer in the DC with the keyboard

Welcome to the world of DevOps, or, if you want to be paid a little more, DevSecOps.

This is why continuous delivery never became popular in the Finance industry.

[yetanotheraoc]

Re: It was the engineer in the DC with the keyboard

"an engineer deploying a emergency DNS"

The part of the manager who told him to go ahead with the fix will by played by William Atherton. He's always excellent playing the character the audience loves to hate.

W.A. "I'm ordering you to make that change.

Engineer "Are you sure? This hasn't been tested."

W.A. "Do it now."

[nintendoeats]

We really need to come up with another acronym for either Domain Name Service or Denial of Service. This article wasn't quite what I was expecting :p

[SgtFalstaff]

Sometimes they are one and the same

[Bibbit]

Business-speak I’ve missed?

I’ve not heard “out of impact” before? Is this a new one or have I just been living in a cave?

I’m still not comfortable with the phrase “going forward” so it’s probably just me.

[Pascal Monett]

Re: Business-speak I’ve missed?

You're on the cusp, the bleeding edge.

It's a bit like Preschoolers, they blabla and they blabla and, every now and then, they make a new noise they like.

Meanwhile, the adults translate their sounds into actual words.

[Anonymous Coward]

Re: Business-speak I’ve missed?

@Bibbit

No it isn't just you. I have never heard of it either. It's just more pseudo intellectual bullshit made up by people who want to sound knowledgeable. My pet hate is "reached out".

[Tabor]

Re: Business-speak I’ve missed?

If your pet hate is “reached out”, that’s because nobody has “circled back” to you yet. First heard that one 7 years ago, and I still haven’t encountered anything in the wild that makes me shudder more.

[Throatwarbler Mangrove]

Re: Business-speak I’ve missed?

Let's sync up offline and have a sidebar conversation. I'm sure I can meet your ask for more egregious business jargon.

[MyffyW]

Re: Business-speak I’ve missed?

I would invite you to come for a quick scuba in my thinktank, but I'm not sure we should be sharing the same corporate waterfall during COVID-19.

[Coat's already on, zipped up, hood up, ready for a deluge]

[RegGuy1]

Re: Business-speak I’ve missed?

"Now, as you all know, I’m not here..."

https://en.wikipedia.org/wiki/Drop_the_Dead_Donkey

[TeeCee]

Re: Business-speak I’ve missed?

Excellent. I admire the way your little rivulets of analogy combined to form a veritable ocean of truth.

[Psmo]

Re: Business-speak I’ve missed?

I thought you weren't supposed to merge the streams in case of splashback?

[sorry, what?]

Re: Business-speak I’ve missed?

@Tabor, for me it's the Indian-centric abuse of the word "Revert". They use it to mean "respond", but that's not what it means... #sigh#

[Anonymous Coward]

Re: Business-speak I’ve missed?

Whenever I hear the word "utilize" I quit listening. USE it or lose your audience.

[Anonymous Coward]

If only...

If only Salesforce had some sort of ticketing software available to them whereby a proposed change like this could be logged and the risk and impact of a mistake fully evaluated as part of the approval process.

[Anonymous Coward]

Re: If only...

Have you seen the price of that software? Who's going to pay that?