Salesforce? Salesfarce: Cloud giant in multi-hour meltdown after database blunder grants users access to all data

Unlucky Salesforce customers have been unable to reach the service since 0956 PDT (1656 UTC) on Friday, thanks to a ham-handed database deployment. Shortly thereafter, the cloud CRM biz said that it's looking into an issue linked to current or past users of its Pardot B2B marketing automation system. It seems the US tech …

[Marty McFly]

Two problems....

1) It happened.

2) The underlying design is such that it *could* happen in the first place. That is the bigger problem. We have been led to believe the cloud providers will hold our data securely, it won't be comingled with others, etc. One database error, and suddenly the data is shared.

There is no cloud. There is only "someone else's computer".

[Brian Miller]

Re: Two problems....

There is no cloud. There is only "someone else's computer".

More like someone else's CSV file!

[a_yank_lurker]

Re: Two problems....

I assume the underlying database is a relational database. From all the ones I have played with granting user permissions and modifying there permissions is fairly easy (exact syntax may vary some) if you have admin privileges. So 2 will happen again and again and again...

[Anonymous Coward]

Re: Two problems....

"So 2 will happen again and again and again..." - especially when you employ the cheapest, most unskilled tech folk you can find.

[Doctor Syntax]

Re: Two problems....

And especially when it's some other businesses' transactions at risk, not your employer's. The essential requirement for a DBA is paranoia and it's hard to feel that they're out to get you when you know it's somebody else entirely that's at risk.

[a_yank_lurker]

Re: Two problems....

Cheaper and unskilled will only increase the frequency of 2. When the non-occurrence relies on no one making a mistake it will happen again and again.

[sorry, what?]

Re: Two problems....

While Salesforce does run on Oracle, behind the scenes, it abstracts DB access and has their own profile permissions mechanism. No one (outside Salesforce) has direct access to the DB.

[gabbs]

Re: Two problems....

It's a meta database and the data wasn't exposed out of the database to other customers only essentially within the same customers org. So if they had set up their internal permissions to stop specific users from seeing specific data in the org this was suddenly visible to the internal user.

[sorry, what?]

Hang on: what does "all data" mean here?!?

Reading between the lines and looking at the supposed "all data" grant, in Salesforce terminology that means all data in YOUR ORG that belongs to YOUR BUSINESS. It doesn't mean all data on the DB across all businesses that share the instance. Can we have some clarifications here please?

[Wayland]

Re: Hang on: what does "all data" mean here?!?

All your data. The context is obvious, all your data on this database.

Just because a database server can host thousands of databases does not mean it should. It would have been safer if each customer had their own virtual server running a database but I suppose people wanting to get on Oracle would need a piece of the big dog not the whole little dog.

[sorry, what?]

Re: Hang on: what does "all data" mean here?!?

Some of the commentards have been suggesting that "all data" means all the data in the database, which is, of course, multitenanted, hosting data from numerous "orgs" (think of these as virtual servers). My point is - I suspect this isn't the issue but rather that sharing rules have been negated within orgs allowing all users (of those orgs affected) to access data for that org without sharing rules being applied.

While this could be a significant issue, especially if it affected communities on Salesforce, it is far less big a deal than some suggest - the data from one business was probably not exposed to other, unrelated businesses.

[Cederic]

Re: Hang on: what does "all data" mean here?!?

The context was not obvious, and it's nice to get clarification.

[Potemkine!]

Re: Two problems....

We have been led to believe the cloud providers will hold our data securely,

A saleman's lie. Data security always rely on the customer in the end.

[Anonymous Coward]

GDParrrggh....

Rubs hands together before going down the shops to buy some popcorn ....

This is going to be fun !

Yet another one to add to my list when I find myself yet again infront of someone who thinks I'm bonkers for not embracing the magic cloud.

[SWCD]

Re: GDParrrggh....

Don't suppose you'd mind sharing some of those via Pastebin or similar? I always end up misplacing old stories and never get my list above three :-(

[Anonymous Coward]

Re: GDParrrggh....

@SWCD

You're right, I should probably write them down for posterity. Maybe I could even get them printed out onto business cards and I could just hand them out - might need to use 0.2pt font size to fit them all on at the rate things are going ... ;-)

I'm a little short on spare time at the moment, but maybe I'll start a doc and add to it in drips and drabs.

[Shadow Systems]

At the AC, re: a list of URL's.

Instead of listing all the individual URL's on a business card, make it a single URL to a site you keep updated with all the individual links.

The business card will be easier to read, the list/site will be easy to update, & you won't have to reprint your cards every time you add a new link to the list.

Here, go enjoy a pint on me... before you get your arse busy creating that site so we can read all those examples! =-D

[el_oscuro]

I once got something this in a deployment script

grant select, insert, update, delete on scott.emp to dev_user;

grant select, insert, update, delete on scott.dept to dev_user;

grant select on hr.orgs to dev_user;

grant create session, create table, create view, grant any role, create sequence, create synonym, create procedure to dev_user;

This was for supposed to be production. Needless to say, I shit canned it.

[Anonymous Coward]

Lying bastards

We noticed this at 10am UTC. We shut our own instance and watched as they fucked around trying to fix this without telling anyone for 8 hours. Cunts.

[NeilPost]

Re: Lying bastards

So for some, what has been taken offline includes ServiceCloud so if you are a support organisation your global service desk is nuked and you are trying to make do with GoogleSheets and other manual workarounds.

D’oh.

[Anonymous Coward]

Re: Lying bastards

I've been developing solutions in Salesforce for support organizations for almost 10 years. I wouldn't exactly label myself as a fanboi, although I'm probably close to one. The platform certainly has quirks and irritations you encounter, but I've always been able to develop useful solutions on the platform.

Although unplanned outages are devastating to support orgs since they need real time access to data to solve customer issues, in the years I've worked on the platform, I've only seen maybe two or three unplanned outages that lasted more than short time. This is the first one I've seen this big, and yes, if we had our support org on production right now, it would have been hell. Luckily, we're still in development mode before moving them off another system, so they weren't impacted by this outage.

The one thing I do criticize Salesforce on is the trust site. It sounds good on paper, but unlike it's name, I can't trust it. It's too slow to update. Our sandbox instance kept showing green for a long time even after several of us couldn't log in. Downdetector's site was more informative than Salesforce's own trust site in figuring out there was a problem with our org.

Salesforce should have tons of metrics and tests going on in real time to determine the health of an org, and that should be directly reflected on their trust site. Whoever is in charge of the trust site needs to be held accountable for making it more real time and useful when something like this happens. I think Downdetector is an incredibly useful site, but I shouldn't have to go outside Salesforce's own systems to get useful info on an outage. That's just unacceptable.

[gnasher729]

Re: Lying bastards

To be honest, in this situation shutting down access to everyone (and telling them only when _everything_ is shut dow) seems to be the right thing to do. Otherwise, some idiot would say "Hmmh, they shut down my access because the enabled everyone to access everything. So let's say what I can still access. " Or they call a mate at a different company to let them check if they can delete everything.

[sitta_europea]

Re: Lying bastards

You got that exactly right.

[Compuserve User]

Wash, Rinse, Repeat.

It will happen time and time again until we get back to a local server mindset. I have never trusted cloud and thanks to Experian I have to be ever more diligent with my data. Maybe the next fad is pen names? This Salesforce drama was something that could have been avoided, but penny pincher IT management and poor HR practices are to blame. Get the data local and then worry about security locally. DO NOT TRUST YOUR DATA TO CLOUD.

[Manuel Breschi]

I predicted something like this happening 3 years ago...

https://www.linkedin.com/pulse/salesforcecom-outage-beginning-end-manuel-breschi/

"... The next level of data disruption I expect to happen some day - at this point probably sooner than later - is when a sales rep will start her working day by handling service requests, orders and any kind of activity and will just realize by the end of the day that those leads were not actually her company's ones, but those of another enterprise, because the same database instance lost the information about the tenant and data would be visible to every customer sharing the same space. Ooppss!"

[gabbs]

Re: I predicted something like this happening 3 years ago...

I'm afraid your still waiting... Salesforce didn't share data with other customers of Salesforce the issue was contained to within the orgs. IE other Salesforce users may have seen other data in the same org from other users of the same company.

[Tech Lead]

The outage is small compared to the data breach here. Granting access to all users to all data across the entire system from one script should make every customer on SalesForce run for the hills. SalesForce had no choice but to shut everything down as now they and all of their customers may have broken data privacy law (HIPPA, GDPR, etc.)

[fredesmite]

Remember - The CLOUD

Is nothing more than renting someone else's computer that other people are on too

[GinBear]

Not all cloud providers are created equally

All of you naysayers of the cloud amuse me.

Yes, a cloud implementation is renting computing in somebody else’s data centre, but that doesn’t mean it is more prone to breaches etc than your own data centre, it just means that you need to manage that risk somewhere else.

For example, you could go with a cloud based provider that will implement single tenant systems (just your stuff on the databases you have rented) rather than multi-tenant such as the Salesforce model

[Roland6]

Re: Not all cloud providers are created equally

>For example, you could go with a cloud based provider that will implement single tenant systems (just your stuff on the databases you have rented) rather than multi-tenant such as the Salesforce model

One question: Who has sysadmin access rights?

About the only cloud you are 'safe' on is one where you are running your own VM's and the cloud providers admins can only prevent your VMs from running.

'Safe' with cloud is a relative term, obviously I suggest running your own VM's is at one end of the scale and having your data within a multi-tenant system such as Salesforce is at the other, with single tenant somewhere in the middle. However, in all cases you are (to varying degrees) at the mercy of third-party sysadmins.

[GinBear]

Re: Not all cloud providers are created equally

‘Safe’ with your own locally hosted clouds assumes that you and your colleagues know more about managing enterprise systems than organisations that only do that.

And those cloud providers known that their reputation will be ruined if they don’t do it properly so tend to invest in getting things right and controlling access to your data... thats excluding salesfarce of course

Of course you may be one of the hugely experienced it bods out there that knows everything and have the experience to back it all up...

[Roland6]

Re: Not all cloud providers are created equally

>‘Safe’ with your own locally hosted clouds assumes that you and your colleagues know more about managing enterprise systems than organisations that only do that.

No, actually it is a matter of risk and mitigation in what you can do in the event of the untoward happening, in this case the sysadmins giving inappropriate persmissions to a third-party organisation.

As you acknowledge, the 'experts' at Salesforce made mistake/got it wrong then in resolving the mess you lost access not only to data but also to an automated business process; lesson: as an organisation using Salesforce etc. what are your risk mitigation and business continiuty strategies...

Personally, I suspect good cloud vendors will up their game, and so put in place safegaurds that will reduce both the risk of this happening again and if it does mitigations. However, this doesn't alter the need for a user organisation to have done a risk assessment and prepared mitigations...

Aside: I think we are agreeing its about risk and its management, in which the quality of the in-house team is just factor in this assessment.

[GinBear]

Re: Not all cloud providers are created equally

I have to say, I agree completely @Roland6, it is about risk and mitigation.

Cloud vendors will up their game, some are already way ahead of Salesforce (take a look at ServiceNow as an example).

The thing that amuses me is that all the cloud naysayers seem to believe that keeping the hosting internally mitigates the risk, when at the end of the day there will be SysAdmins working internally that either accidentally or malevolently can do just as much harm as a SysAdmin working for a cloud provider. Add to this the fact that the cloud provider will have much greater focus on managing this risk and have invested far more in avoidance and mitigation because it is core to their business.

[yoganmahew]

Re: Not all cloud providers are created equally

It's not about knowing more about enterprise systems, it's about knowing more about your own business and setting your recovery priorities accordingly. Unless you are a super large customer, you have no specific priority with a SAAS vendor on a shared system. Their low priority service interruption could be catastrophic for you.

As it was, Salesforce had an issue with part of their system, borked everyone, said "here's how you fix it yourself", ran a fix that only worked if you hadn't tried to fix it yourself, and patted themselves on the back.

(The last bit is yet to come ;-) ).

[FozzyBear]

Can't find it

The Related "Who, Me!" article

[Anonymous Coward]

To Cloud or Not To Cloud

So... The moral of the story is to use an on-premise CRM?

[steamdesk_ross]

Separating DBs

My recent experience with a major cloud provider is that proper DB separation isn't something that is particularly easy achievable or well documented, especially in their "off the peg" auto-scaling environments. You still really have to roll your own mechanisms to achieve this. For our project we ended up blowing a large amount of a recent project budget on implementing this better than the environment was pushing you to. That came as quite a surprise, I'd have thought that by now DB compartmentalisation would be de riguer!

I strongly suspect that a large number of service providers who really ought to have absolute DB and CDN separation don't. I won't name names, but having researched this in depth I think that this same mistake could happen for some of the largest:

financial/bookkeeping service providers,

note/document storage apps,

CRM systems,

mail providers,

DDOS prevention providers.

... all of these I investigated when I was trying to get advice on how best to do this ourselves. It's very hard to find any companies which advertise that they use proper DB separation/encryption so that client data is compartmentalised. Why wouldn't you advertise it? Because you know that you are sat on monolithic DBs and CDNs which *can* be compromised in a big way by a single small mistake.

[jeremylloyd]

Re: Separating DBs

It is exceptionally hard to hyper-scale a SaaS system where each client has a unique single-tenant database. You get problems with backups taking very much longer, CDP with large numbers of databases, product upgrades which involve schema changes, load scaling across multiple product releases because you can't role out schema changes quick enough, etc etc. It's expensive to manage.

Multi-tenant solves many of these problems, but the trade-off is security. When you screw up, you screw up big-time.

[Mr Sceptical]

Did someone at Salesforce annoy the BOFH and revenge was served Aliens style? -->>

Alternatively, meet 'John the sometime dba' in a future episode of, "Who, Me?"

I'll skip the popcorn, but enjoying the show anyway!

[-teacup ordinance]

no consequences

as was said previously: ""So it will happen again and again and again..." - especially when you employ the cheapest, most unskilled tech folk you can find."

and the reason it will happen again and again is that there are no consequences for the toad that hired the cheap unskilled tech. the tech will get thrashed but the manager and director and VP above will still get bonuses even though everyone in the company knows who really is to blame.

Thsi type of farce has played out in every one of the companies I have worked for in my 30 years : / without exception.

until some higher up loses a bonus because they created a bad situation by faulty management ( Mr. Fawlty, Mr. Fawlty) it will continue as an endemic disease in all companies.

* warning: no sarcasm was used in this posting. end

[Tom Paine]

Re: no consequences

Thsi type of farce has played out in every one of the companies I have worked for in my 30 years : / without exception.

May I politely suggest the possibility that you're picking the wrong employers?

No doubt there are plenty of industries and orgs where senior managers cover up for each other in this sort of situation, but in anywhere of any scale the seething piranha pool of hungry ambitious execs eager to climb a ladder of knives sticking out of other people's backs mean "you're only as good as your last outage". At least two of my past employers ditched senior execs after major outages (Director, Operations and CIO, respectively.) So it does sometimes happen.

It will also depend what sort of impact this event has on the bottom line. If they end up as GDPR test cases and get a huge fine -- or if there's a long-term loss of customer confidence and they start switching to competitors in significant numbers, senior heads could certainly roll.

[Anonymous Coward]

They are full of it in some ways - They're being transparent but transparent like a muddy pond.

As someone who works heavily with Salesforce (anon so I don't get the paddle) I can say that they really cocked up with this one. They claim or formerly claimed not to do this type of stuff outside of a maintenance window. From what I can tell they added some enhancements to their Pardot product and were trying to change access to what objects the built in Pardot account could see. This impacted any org that ever had Pardot installed - even as a trial. Again - they have maintenance windows for product updates but as they get larger and larger they seem to be doing more one off updates outside of those windows.

For all the technical people here's a Salesforce primer - the idea of permissions do not exist as you or I would know SQL flavor of the week/Oracle permissions. They have some type of permission black magic voodo layer that runs on top of the Oracle DMBS and handles the multi-tenancy and organization (the company using Salesforce) level permissions. So the permissions they cocked up were org level object permissions. Salesforce permissions allow you to grant CRUD access to objects (tables) and for those toplink/hibernate folk you'll get Salesforce's object terminology. I only noticed the problem when my clients started pinging me that anyone without admistrator permissions could do anything in Salesforce. I have no idea if they accidentally granted read/modify all on objects (still organization level) to individual users because honestly most companies actively using Salesforce don't care if people can do too much, only that they can't do anything.

But that explaination still doesn't hold water with me because they didn't just revoke the modify all/read all permissions on these users, they revoked all permissions on these users. They removed all CRUD object level access to the point where some users couldn't even log in. Those that could were not able to do anything in the system. They did all of this in a way that was not correctable through the UI. So when I tried to grant read access to helpdesk users on the case object, I just received a nasty red Oracle error. They left the system in a place where it wasn't even admin correctable.

The whole time the site that they have to provide system status (trust.salesforce.com) was showing a big green check box that everything was wonderful. Yet when I went to go log a case with Salesforce they had a message that they were having these problems. Good enough for case deflection, not good enough for the marketing site to show how much uptime they have.

I still think Salesforce is a decent product, but I think everyone who hears cloud doesn't realize what they're actually selling. This is more like you paying someone to run their software - same as if I paid an ISV to use a product that ran on their premises. It's not like AWS, Rackspace, et al. where you run a virtualized machine that you can do anything you want with it. I do have some problems with the way they've outsourced their support who just read the help documentation back to me, or the way I think some of these screwups are the result of them trying to continue to show year over year growth when they have already sold to must people who are willing to buy.

[dotdotdotcom]

Oopsie

Looks like someone forgot to add the WHERE clause to the script before they hit execute.

[NiceCuppaTea]

Hosting / Cloud providers don't learn from others mistakes (or even their own sometimes) even with a single tenant solution a tired/inattentive sysadmin can have devastating effects, i still shudder when i think of all the VPS instances that 123 reg nuked a few years ago whilst trying to delete inactive VM's

[Anonymous Coward]

All your base are belong to us... had to say it...