Salesfarce to Failsforce: Salesforce database blunder outage enters day three as fix falters

Three days on, Salesforce.com has yet to fully recover from an outage that began on Friday. Fifteen hours and eight minutes after an errant database deployment script granted past and current users of the company's Pardot B2B marketing automation system full read and write access to all data, prompting the cloud CRM giant to …

[Anonymous Coward]

Meh!

> And now the weekend is over…

I had no idea what 'Pardot' was until this outage happened. I looked it up: apparently it's a B2B marketing system.

Before the outage: Salesman to boss: I've just sent 100,000 'valuable marketing messages' and had one response.

During the outage: Salesman to boss: I've only managed to send 200 emails manually... and had one response.

[Anonymous Coward]

Well

This didn't actually affect pardot, it was the main salesforce instance that got borked. The ICO is going to rip them a new one - people logging into their service will have seen everybody else's data - medical service companies, financial services, the lot.

[Anonymous Coward]

Re: Well

On the call, I believe they said the expanded permissions only affected users within an org, not across orgs. So if that is accurate, then one tenant shouldn't be able to see another tenant's data.

Of course, a lot is riding on that being an accurate assessment, which at this point, who knows.

[Anonymous Coward]

Re: Well

Sort of true, sort of not. Anyone using communities (people from other orgs accessing only a small subset of data on yours) will also have had their profiles elevated. So there is massive potential for GDPR breach there, and the ICO will certainly have something to say about that.

The cloud solves everything right?

[Anonymous Coward]

Exactly

... and that's how we found out there was an issue (11 hours before they fessed up). A friendly user sent us a screenshot saying this doesn't smell right. We closed the community. I watched as they ran multiple scripts in the background over-writing the permissions in the profiles multiple times. Luckily no need for the ICO for our dataset, but you can bet a lot of letters were sent on friday for those holding any sensitive info.

It's Salesforce world tour Thursday in London. I'm going to ask them some very awkward questions.

[ecofeco]

I never get tired of saying it

So how's that cloud thing workin for ya?

[Anonymous Coward]

Re: I never get tired of saying it

Third quarter earnings look great!

Actual things of substance, not so much.

[Wilco]

Re: I never get tired of saying it

Pretty well, thanks. Of course, I have proper rollback scripts for any change I make to a live system.

[Dr Who]

Re: I never get tired of saying it

Crikey, haven't we moved on from this sort of thing yet?

You might as well ask how that IT thing is going for ya? Abacus, slide rule and paper spreadsheets were just so much more reliable. Never had a power outage with those. Or a virus.

Embrace the new, make it work for you and learn valuable lessons from those who fail to make it work for them.

[trevorde]

Re: I never get tired of saying it

"Embrace the new" means trusting that a third party API will always be available and work as advertised. The rug can be pulled out at any moment, and there's nothing you can do about it - just ask any dev who's used the Twitter APIs.

[Sir Loin Of Beef]

Re: I never get tired of saying it

Nope. Why put your data in the hands of someone else to bork?

[LordHighFixer]

Re: I never get tired of saying it

Embrace the new, are you high, this is re-invent the old. A place, not in my company, where computers live, that I send my work to and get the results back. Welcome the Pre 1970's computing!!!

[Anonymous Coward]

Spare a thought for the teccies...

The ones at SFDC who are working 24x7 to try and undo this idiocy while the bosses breathe down their necks and the customers continue to call in to ask 'Is it done yet?'.

No excuse for the cretin(s) who broke it, but just remember that there's someone out there surviving on coffee and haribo, who probably hasn't seen wife and kids for 96 hours straight, desperately working to fix it.

This is my territory and I almost took a job at SFDC a year ago that would have put me right on the front-line in this.

AC because $CurrentEmployer doesn't know I interviewed at SFDC!

[fnusnu]

Re: Spare a thought for the teccies...

"just remember that there's someone out there surviving on coffee and haribo, who probably hasn't seen wife and kids for 96 hours straight, desperately working to fix it."

And that's probably why it hasn't been fixed.

[Sir Loin Of Beef]

Re: Spare a thought for the teccies...

Why? That is what a tech does - I am one. There are times when you work long hours. Just remember that when you are bored out of your mind.

[fnusnu]

Re: Spare a thought for the teccies...

The National Transportation Safety Board (NTSB) has found fatigue to be a causal or contributory factor in accidents in every mode of transportation and has issued almost 80 fatigue-related safety recommendations since 1972. The National Aeronautics and Space Administration (NASA) Ames Fatigue Countermeasures program has addressed fatigue in aviation through research and other activities since 1980.

https://www.aviationpros.com/home/article/10386580/a-matter-of-fatigue-how-it-can-affect-performance

https://safetyalliancebc.ca/the-dangers-of-fatigue-in-the-workplace/

[alferdpacker]

Re: Spare a thought for the teccies...

Quite. The fact that they keep fixing it and then breaking it suggests they should maybe take a step back.

Easier said than done, of course. I always get sucked in and forget to eat etc.

[Anonymous Coward]

Re: Spare a thought for the teccies...

Spare a thought for the customer service reps. They had nothing to do with causing the issue, but are the ones taking the brunt of customer ire.

[Anon Coward (there are nutters out there - I've worked with them)]

MIM Bridge

That is one MF of a MIM bridge I would not like to be part of!

Actually forget MIM - whats the next level after MIM?

[Anonymous Coward]

Schadenfreude

I'm experiencing just ever such a small amount of schadenfreude that this is happening...

"Salesforce" is one of those sort of ridiculous gung-ho macho testosterone-overloaded company names that you really just so want it to get its due comeuppance one of these days.

(Likewise for embarrassingly awful sneering and patronising names such as certain companies named after monkeys. Can they be next, please?)

[Anonymous Coward]

Re: Schadenfreude

Drawing a total blank. Bonobo.com? Macacque Technology Services? Bald Uakari, Inc.?

[Anonymous Coward]

Re: Schadenfreude

Imagine an infinite number of monkeys typing out an infinite number of (usually fairly content-free and unreadable on mobile phones because of the space/formatting uselessly reserved for large pointless images that you never actually download) "subscriber contact" emails, and you would be closer to the mark.

On second thoughts, maybe the monkey reference does relate to the email writers after all, and isn't a cheap insult towards postal workers?

(Anyway, at least bonobos seem to know how to prioritise the important things in life)

[Anonymous Coward]

Re: Schadenfreude

WTF have Gorillaglass got to do with it?

[CloudMonster]

#popcornemoji

[Tom Paine]

Like a Saleforce into the night...

YouTube "LCD Soundsystem:: Where Are Your Friends Tonight?"

[Claptrap314]

Maybe I'm just REALLY slow

But would it have not worked to roll back the transaction? User permissions are going to be orthogonal to almost all other activity. ?????

[Anonymous Coward]

Re: Maybe I'm just REALLY slow

I was on one of the bridge calls and was shocked when one of the SFDC people said something like (I'm paraphrasing and I hope I'm recalling this correctly) "We couldn't roll back what the script did."

What the F?

You allowed a script to be run in production, in a multi-tenant environment, that affected permissions on data, and you didn't have a rollback plan if the s**t hit the fan?

Somebody needs to be fired over that one. Not the person who ran it, because they were probably just doing their job. But the person who allows such scripts to be run in production with no rollback plan needs to be fired.

They kept apologizing on the call. You know what that means? Nothing. Stop apologizing and fix your damn processes and institute mandatory rollback ability on ALL scripts run in production!

[Kevin McMurtrie]

Re: Maybe I'm just REALLY slow

Hmmmm... Maybe the "script" destroyed everything and the current state is a rollback that doesn't include some important bits.

[Daedalus]

Bozocalypse, salesforce edition.

Another domino falls. You think this is bad? This is presumably an environment where (theoretically) they knew what they were doing. Assuming everything isn't farmed out to those fine people who like to "do the needful", that is. Those big city admin networks in the US don't have anybody who knows what they're doing, and they're paying (literally) the price.

Too much technology, too few technologists.