ML fails: Loyalty prediction? Not really. And bonus prediction? Oh dear You may never have heard of the NPS but if you have ever answered a question such as: "How likely are you to recommend our company/product/service to your friends and colleagues?" you have entered the Net Promoter Score zone. The Net Promoter Score is used to gauge the loyalty of a firm's customers and, since its introduction …
I didn't used to mind taking part in these surveys for the account managers of our main suppliers. Then they slowly started leaning on me and said I needed to give them a 9 or 10 otherwise they were in trouble. Having read the article, I can understand why.
I strongly dislike being told what answer to give for a question so I now decline to take part. My guess is that many of their other customers are taking the same line and they're getting fewer and fewer responses, making it harder and harder for them to achieve their target score. (Which in turn puts more and more people off and....)
Then they slowly started leaning on me and said I needed to give them a 9 or 10 otherwise they were in trouble. Having read the article, I can understand why.
Whether the sales droid gets a bonus is not the real issue: once you start giving them a 9 or 10 then the marketing department will start shouting from its web site that ''90% of customers consider us excellent'', which will pull in more people who will then also be pressured to give artificially high scores.
So: what is really going on here is that companies are using the social pressure (''that nice man who sold me a car wants to give his daughter a nice birthday present, what possible harm is there is me lying on this questionnaire ?'') to artificially raise the company approval score; this acts to the detriment of (potential) customers. The value of the commission paid to the salesman is trivial compared to the extra sales that are generated.
Just another way of generating fake news.
A consequence that the article authors did not understand.
It's not fake news, it's marketing.
Marketing should be dealt with in the same way as unwanted dog poo.
At first ignore it.
If it continues, investigate further and confirm the source.
If it still continues, apply fire to the source. A LOT of fire.
Slowly, the world will become a better place.
I wouldn't rate anyone at "10" for unless they were absolutely perfect in every way (which is impossible) and I wouldn't rate "1" unless they were a physical manifestation of pure evil. Neutrality is of course "5" as anyone who has ever seen a bell curve knows instinctively.
1. Ultimate Evil
2. Imbecility
3. Incompetence
4. Sincere effort
5. Average
6. Good
7. Very good
8. Excellence
9. Preternatural
10. Godlike
Neutral when the scale has 10 possible values means that you *ought* to treat 5 and 6 as equal. Otherwise, your neutral is 4 steps away from Evil but 5 steps away from Godlike.
This is why if you _want_ to offer a neutral answer option you ought to have an odd number of values on your scale.
(Note that the NPS system has a 0 which you've not assigned a value to, but apparently it's worse than Evil)
Zero? In that case, I can revise the scale by moving 1-4 down a place and creating a new "Adequate" rating at position 4 i.e:
0. Ultimate Evil
1. Imbecility
2. Incompetence
3. Sincere effort
4. Adequate
5. Average
... makes no difference to the frontline victims though because it just adds greater granularity of fail. Of course, I'm being honest here. If I actually get wind of someone using this kind of thing to scam specific employees out of their wages then I'll happily be completely dishonest in whatever way is most effective for gaming the system. I'm only honest when the rating is applicable to an organisation as a whole, such as habitually rating EE between 1-2 (on the scale above).
Yes. But.
The last time you bought something online (eg. App, auction site, mail-order, hotel, etc) - did you go for the supplier with 3 stars out of 5, or 5 stars out of 5?
As consumers, we should be satisfied with a supplier who has a score of 3 out of 5. But, we're not. We see that as a failure. As business owners, we have to ensure that our feedback score is always 5 out of 5 - or we lose business.
So, we find that the scale has been re-calibrated. Maximum score doesn't mean "God Like". It now means "Everything went OK. The parcel arrived on time, and wasn't broken - and I received the item I paid for".
....that the authors of this piece have never worked for a large organisation with multiple layers of management.
Otherwise it wouldn't/shouldn't have come as a surprise to them that some notional objective(ish) measurement tool has instead been transmogrified into a tool to support/extol managements objectives rather then measure their effectiveness.
Did you really expect anything else?
> Otherwise it wouldn't/shouldn't have come as a surprise to them...
I didn't spot where the article expressed any suprise. Just a reasoned explanation of what is going wrong, so that we can all understand how it is going wrong, not just that it has been going wrong.
Perhaps also a modicum of sadness that those who actually need to understand an article like this will be precisely those who would never read it - and it was ever thus.
"Since the probability of any one of your customers being a promoter is 0.7..."
I probably haven't had enough coffee to fight through my cold, but I don't get that at all. If you model the customer as equivalent to rolling 1d10 then the chance of them being a promoter is 20% (a roll of 9 or 10); with a 10% chance of them being neutral and a 70% chance of them being a detractor.
Alternatively, if the P/N/D distinction accurately captures human behaviour, you could argue there's a 33% chance of each outcome.
Where the hell does 70% chance of being a promoter come from?
From the employer. Higher up in the article:
Your employer explains that you can get your bonus if you achieve 70 per cent Promoters, 20 per cent Neutrals and 10 per cent Detractors (70P,20N,10D).
Your employer has told you you can achieve your bonus if you hit these figures. The rest of the article is pointing out that it's untrue.
"Your employer explains that you can get your bonus if you achieve 70 per cent Promoters, 20 per cent Neutrals and 10 per cent Detractors (70P,20N,10D)."
Maybe I don't understand statistics but how does that state that the chance of any one customer being a promoter is 70%? It merely states what you will get as a result of customers being a Promoter or not.
On reflection, and slightly less cold, I concede that the boss's target can be reused as a probability distribution. If the salesman's making that target exactly each month, and you pick a sale at random, then it will have those probabilities of being P, N or D. In effect the saleman acts as a convolution on the underlying probabilities.
"Your employer explains that you can get your bonus if you achieve 70 per cent Promoters, 20 per cent Neutrals and 10 per cent Detractors (70P,20N,10D)."
That's asserting you can achieve a bonus if the number of promoters are 0.7 * number_of_sales and the number of neutrals are 0.2 * number_of_sales. It has nothing to do with 0.7*0.7 or 0.7*0.2 which are the chance of two 70% events happening or the chance of a 70% and a 20% event happening.
If you're right, and that's the reason they've used 0.7, then the stats is very wrong.
It might have been sensible to point out that Net Promoter Score is actually a trade marked term, and the technique, although widely copied is actually a proprietary one.
Other than that, the article is a fair assessment of the bungled and widespread mis-applications of the tool. At the individual the article is correct, but even at company level it is flawed, because there is another level of failure, and that is that it assumes that "not recommending" is a standard level of detractor, and that's rubbish. When a customer is really disaffected, it isn't that they "wouldn't recommend" a company, they actually become quite vocal telling people to avoid a company (like I do for Virginmedia). NPS completely fails to capture those real detractors. It likewise misses out the real advocates, who individually will score a company 9 or 10, but who are a small subset of those reported 9-10s.
The real appeal of NPS is twofold: It gives a single number, so ideal for target setting and "performance measurement", and it is easily manipulated.
I gave a scathing response to a carrier.
I pointed out the management were incompetent, the call logging process was flawed, the first two weeks of so called service and continued denials were shockingly bad.
After a few weeks (and threats to move thousands of pound of business elsewhere) did it finally get through to the real tech team. In 24 hours the issue was resolved.
Guess who got the bad score?
Yup the poor sods that actually fixed the issue.
needless to say another scathing emails and phone call followed.
"Guess who got the bad score?
Yup the poor sods that actually fixed the issue."
This is a very real problem. There have been times when I would like to send a message of dissatifaction to megacorp for sending me through too many hoops to get to a techie who knows their stuff. I definitely don't want to adversely affect the techie, but I still want to get the message through that the support structure stinks.
In theory it sounds like a good idea to feed back your degree of satisfaction with a product or service. In practice I have had enough of filling in endless surveys every damn time I buy a pair of socks, especially if (as the article says) the questions can't be taken at face value and the answers are used to downvote someone's bonus.
How likely am I to recommend this company to a friend?
0 - because I don't have any friends?
0 - Because this isn't the sort of thing my friends and I talk about?
0 - because I know my friends won't ever use this sort of company or service?
Even when I'm really pleased with a company, it's probably less than a 1 in 5 chance that I'll tell anyone, for at least one of the reasons above.
Hack me off enough and I'll go round reading every story on El Reg looking for an opportunity to mention how useless [redacted] are in the comments.
This.
The other day I called Barclaycard Merchant Services to find out how to perform some tests on a PDQ terminal that was intermittently failing to connect when processing transactions and I got put through to an automated survey at the end of the call asking how likely I was to recommend their services. To be fair they did ask during the call if i didn't mind that happening.
Given my job role there's no scenario I can imagine where my view of payment processing providers would be sought or need to be offered. They honest answer would be 0 - it's never going to happen.
I played along though and gave them a 6 because they guided me through the testing procedure efficiently (it worked this time) but offered no suggestions as to why it was sometimes failing.
This was classed as dissatisfaction on their scale and they then wanted me to put into words why I felt this way. I suppose that's a way of obtaining more meaningful feedback but I wasn't prepared to offer any more brain power than that required to think of a number and press a button so I hung up.
EXACTLY! As a former non-PHB of mine said "If you do someone a really good job, if the subject HAPPENS to come up in conversation, they MIGHT mention that you did them a good job. If you do them a bad job, they will bring up the subject in every conversation they have with anyone and tell them you're a screw-up."
My doctors texts me this after every visit.
"How likely am I to recommend the surgery to friends and family"
Vodafone have started randomly texting me too.
"How likely am I to recommend Vodafone to friends and family"
It's more a question of how likely am I to just delete the text upon receipt.. very likely.
not relentlessly nagged for by the service supplier.
My record is 15, including 5 pieces of snail mail from the dealer about a car service earlier this year.
This ended up annoying me so much I ended up giving them a score of 0 with the comment about the nagging. Got a phoned apology form the service manager, who then had the hide to ask me to rescore the actual service. I hung up. As they offered an extended-extended warranty if I serviced with them, and this is now expired, I will no longer use the dealer service.
Just curious - how does a 'nil' response score ?
This post has been deleted by its author
Frank Bettger had it all figured in the 1930s. He analysed the streets that bought most insurance, and allocated his time pro-rata. It didn't take long before he was getting 80% conversions from 20% effort.
I mentioned this once at a sales pitch an NPS expert firm were giving to our board. But techies don't get consulted on spunking money to marketeers.
Well, except that I now know that, by not being a customer at all, I arguably cause less front-line damage than by patronising the outfit.
Don't you want to be a customer just so you can wear the Wetherspoons customer uniform? You know, a saggy, stained grey tracksuit, with bottoms having a knee level crotch so that as you waddle it looks like you've shat yourself? Fag in hand, three days of stubble, and ideally a proper ratboy face?
The company I work for used to have this kind of question on their customer portal and once every month or so if you logged in, it asked you how high you would rate it.
There was a bug that caused me to get the survey every time I logged in (probably the way I was binning the cookies).
I just answered 10 every time and when the followup question joyfully thanked me for such a high score and gave me the opportunity to fill in a free text box with the reason, I took great pleasure in typing "I work for the company and part of our bonus is based on the NPS. Bad targets drive bad behaviours!"
I rarely do, but I became aware of a Chinese company that sells products (for a commercial kitchen, not IT related) for much lower prices than the competition. That alone wasn't enough to get me to recommend them, but the products are of very high quality as far as I can tell and I'm able to get service on them from a local supplier should it be necessary.
The best part is that when I google the products everyone on the internet is selling them for the same price as that local supplier so I don't feel like I have to choose between a better deal that screws the local guy who I will need to count on for service or overpay to guarantee I have a happy supplier who will provide quick service if it is necessary.
I still wouldn't feel like I could give them a 10, or even a 9, because I've only had a couple items for a few months - too early to gauge reliability. The price I paid was less than double what I'd pay for used "guaranteed for 90 days" equipment, and almost half of what I would have paid for new equipment from other companies so I felt it was worth taking a chance on a new player. If I'm still happy after a year or two then it's a 9, and if it is trouble free for 5+ years then I'll call it a 10. I guess I'm hard to please, but the NPS scores don't seem to jive with my recommendation threshold since they would consider 8 to be 'neutral'.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
