This is the crux of the issue - CEOs do need a simple metric to summarize performance in each of their business units in order to evaluate performance and prioritize company resources. Metrics are only useful for decisions if they are easy to understand, and consistently measured. Just as Earnings is used to measure profitability, and Revenue is used to measure growth, NPS is the best they've come up with so far for customer experience. Of course none of these 3 metrics is perfect, but they still need to be measured and complimented by more detailed reporting. To better understand Earnings you look at changes in the component parts, revenue and cost. To better understand NPS you look at the mix of scores (1s, 2s, 3s, etc) and follow up questions like actual referrals, which differ by business (unlike NPS which can be applied more broadly).
The author devotes most of the article pointing out that there are areas where follow up questions would give a richer understanding of customer experience, which of course all NPS proponents would agree with.
What he could do a better job of is trying to convince us why executives shouldn't even attempt to summarize their company's performance in customer experience, just like they summarize lots of other complex activities to report to shareholders. Too many companies focus only on hard metrics like revenue and profits, which is why they find NPS a helpful way to steer the company's focus back to the customer.
He would tell you that NPS is only like earnings or revenues if we allowed either to have 50% or more of their data filled with arbitrary numbers, not audited data collected from state-licensed specialists who would lose their job if it was discovered the data was manufactured whole cloth.
The author would also tell you that NPS is easily gamed and there's no checking on whether that is or not. He wrote extensively in the article the various techniques that folks can game the numbers. If this is a number reported to shareholders, shareholders should insist (No, Demand!) that the numbers be corroborated by a neutral third party that will accept liability for any errors. (No surety insurer will guarantee such a liability, for the risk of error or misrepresentation is way too high.)
As you stated, most use follow-up questions to get a richer understanding of the customer. What the author would tell you is that it's clear the NPS recommendation question taints those followup questions and diminishes their validity and inherent value. If the true goal is to learn a richer understanding of customer experience, there are many better ways to achieve it.
In other words, the author believes if executives want a simple metric that is better than NPS, a random number generator is the fastest and cheapest way to achieve it. Why bother with customers at all, if all you're going to do is squander your interaction with them on such a foolish metric.
— The author.
These types of sweeping statements aren't a helpful way for the author to advocate his broader point.
The strengths of the article lie in the more detailed points which bring to light some great examples of how NPS, and surveys in general are mis-used (and some examples that don't actually manifest themselves as real problems in daily use very often).
My thoughts on the examples chosen above:
>50% or more of their data filled with arbitrary numbers
I'm not sure what this refers to, but the way the NPS equation works, every respondent's score matters and mathematically impacts the overall NPS (each one feeds in as either a promoter, neutral, or detractor). Some of the author's own recommended questions include only 3 answer inputs also.
>Not collected from state-licensed specialists
Almost all operational data collected by companies for management reporting is not collected by state-licensed specialists, but is still useful.
>Easily gamed
All survey questions could be gamed in the ways similar to the article examples. A good executive will make sure the survey is asked in the same way of his own organization as of those of peers, and in the same manner over time. He or she won't let agents do things like cherry pick which customers to survey, otherwise the money he invests in the survey wont' actually help him run his company.
>Shareholders should insist the numbers be corroborated by a neutral third party
Correct, companies who are serious about measuring NPS accurately do often hire third parties to run surveys (see example of JD Power running NPS: http://www.jdpower.com/press-releases/bain-certified-net-pro...)
>NPS recommendation question taints followup questions
All survey questions can be tainted by preceding questions. When writing a survey it is fairly straightforward to A/B test the order to make sure this isn't a major factor.
This seems perfectly reasonable to me. Outcomes matter -- not effort -- and reaching 6 is not the outcome NPS wants.
It’s like “Capital A” Agile. Sure, it’s easily gamed, but only only screwing over yourself when you do that.
None of your arguments here are based on data. Do you have some evidence that a measured NPS score proved that the metric is bad? The WP link you posted to criticisms is all arguing relative merits. None of them are particularly strongly opposed, and none claimed that NPS doesn't work.
> The author would also tell you that NPS is easily gamed
Do you have data showing NPS scores being gamed?
Easily gamed and actually gamed are two completely different things. Having tried to measure NPS before, I found that 0 people appeared to be gaming the system, my customers told me honestly that my product was mediocre.
To suspect that the polls are being gamed, you assume there's something in it for the respondent, right? What benefit do you think there is for respondents to answer dishonestly?
> In other words, the author believes if executives want a simple metric that is better than NPS, a random number generator is the fastest and cheapest way to achieve it.
I hate to say it, but this kind of hyperbolic statement is having the opposite of the intended affect, it's reflecting on the author.
This is not how it's used in practice. Meaning: No company measures performance based solely on an NPS metric. NPS is one data point among many used to measure company performance.
There's no more sense in demonizing NPS as there is in worshipping it.
Nope. Feynman phrased it well:
"Poets say science takes away from the beauty of the stars - mere globs of gas atoms. I too can see the stars on a desert night, and feel them. But do I see less or more? The vastness of the heavens stretches my imagination - stuck on this carousel my little eye can catch one - million - year - old light. A vast pattern - of which I am a part... What is the pattern, or the meaning, or the why? It does not do harm to the mystery to know a little about it. For far more marvelous is the truth than any artists of the past imagined it. Why do the poets of the present not speak of it? What men are poets who can speak of Jupiter if he were a man, but if he is an immense spinning sphere of methane and ammonia must be silent?"
For sure, NPS has its limitations. Broadly speaking, I think it's a bad metric for something like asking random strangers how much they like their iPhone. But if you're trying to figure out the strength of, for example, an ERP software company's customer franchise, NPS is really useful, for two reasons:
1. When you're talking to people who know a product well, and you're talking to them in a professional context (e.g., a scheduled 15-minute interview that is part of their work day), they tend to be thoughtful about the 0-10 rating.
2. The follow-up question traditionally used in NPS surveys, "What could [company] do to make that score a 10 instead of [number]?" almost never fails to provide valuable feedback. I believe this is because of the way the question is set up: first the respondent is asked to provide a broad indicator of their customer satisfaction, and then, with that number identified, they're asked for details as to why they chose that number. I have no background in psychology but I think it's pretty intuitive that this question-flow focuses the mind and prompts people to provide insightful constructive recommendations.
Asking "What can we do to make that score a 10" of existing customers is fine as long as you've got it balanced with some other work to help you gauge the thinking of the portions of the market that aren't already sending you a 6 figure annual check. But I bet many companies are lulled into a false sense of security by numerous answers to that sort of question that just request minor improvements to existing functionality. Probably frequently given by people who answered 9 to "would you recommend?" and then migrated to some hip new cloud offering a year later.
That broader context is outside the scope of the work I usually do, though -- I'm usually focusing much more specifically on understanding the strength of existing customer franchises.
Certainly, two customers who give 9-out-of-10 scores might have very different levels of satisfaction. Sometimes it's a 9 because "I love the product and the customer service is great but on principle perfection is impossible and I don't give 10s" and sometimes it's a 9 because "the product is pretty good and does everything we need, but our main sales rep is really hard to get ahold of". This is why, in practice, I find that more insight comes out of the "why not 10" questions than the pure NPS number.
What? No!!! Any self-respecting statistician would know that the mean of a quantity where addition is not well-defined is meaningless.
Use the median!!!
The reason is obvious. The entire theory is that for your business to be successful you need to have exceptionally satisfied customers who will PROMOTE your business.
All you have to do is read the name of the thing to understand exactly how and why it works, which apparently everyone besides this author can do.
Not that the mean is the only (or even the most useful) statistic.
For example, 6 on the NPS scale would be less likely to recommend compared to 7 and 7 would be less likely than 8. However, the gap between 6 and 7 and the gap between 7 and 8 may not be equal. If you were to get the mean of 6, 7, and 8, you would get a value of 6, but there is no guarantee that the average of the participants' likelihood to recommend was actually equal to a 6.
Measuring the mean would work if the interval between the numbers were equal. See here for details: https://en.wikipedia.org/wiki/Level_of_measurement
There goes the Dow Jones average.
The Dow Jones was invented in the 1890s to be easy to calculate. It is widely panned and is not used by anyone in industry for anything serious.
Several marketing scholars have pointed out the psychometrically undesirable properties of the metric and there is conflicting evidence on e.g. the metric's predictive validity, e.g. in relation to company success
(All in all that old HBR marketing article contains very little details to back up the claim that it is a good and valid metric.)
If you can go with behavioural outcomes for measuring success, e.g. purchases, I think that that will always be more powerful than what a user says in a survey.
And if you can then causally link (e.g. through experiment) something you can influence (e.g. some dimension of "UX) to that outcome you actually have a quite good decision tool too.
NPS does not provide that.
> On a scale of 0-10, how likely are you to recommend Netflix to a friend or colleague?
0
> You answered 0 to the last question. Why wouldn't you recommend Netflix?
It's 2018, people. Everyone I know either already uses Netflix, used to use Netflix, or doesn't own a computer. If I started going around making a recommendation like that, they would think I'm a prat.
The kind of situation I'd expect to see more often in real life is that NPS scores are inflated for fun luxury goods, because they naturally inspire more enthusiasm, and deflated for essentials, because very few people are even capable of getting excited about socks.
Mostly I was meaning to hint at, where I've seen things like NPS go off the rails is when people assume they measure what the marketing pitch says they measure. The reality is invariably more subtle.
(I do keep an exception for NPS surveys on automated interactions. No, Barclaycard, I will absolutely not recommend you to friends based on the text message you sent me about my increased credit limit, are you literally insane?)
The only time I answer less than 10 for a CS person is if they’re actually rude or not trying to be helpful. I much prefer the surveys that ask multiple specific questions that split the interaction between the rep and the outcome.
0-6 counts as a-1
7-8 counts as a 0
9-10 counts as a +1
NPS is an average of these numbers so if every user gave a seven or eight, the NPS would be zero (which is a mediocre but not terrible NPS score)
https://cdn.business2community.com/wp-content/uploads/2016/0...
How about "Your friend has decided to ditch cable and try using a video streaming service. How likely are you to recommend Netflix?"
It's even more critical with services that have very specific use-cases. How about a dermatologist? I don't think many people would say they recommend this type of service without some context. So, "If someone you know develops a similar condition, how likely are you to recommend DermatologyCo?"
> How likely are you to refer a friend to a position at <company>?
The answer is 0. That doesn't mean I think it's a terrible place to work, it means that I don't know anyone who is both interested and qualified.
Similar findings were made by Google with the YouTube rating system that was switched from 1 to 5 stars to a simple upvote/downvote after realizing that most users are polarizing towards the extremes: https://techcrunch.com/2009/09/22/youtube-comes-to-a-5-star-...
To be fair, everything I've read about NPS says don't use it in isolation. The article covers this with the "why?" question. You need to follow up with more detailed questions to get any real benefit from it. But that reduces the survey to a simple tool for flushing out your unhappy users. It gets more complicated because, like unhappy families, everyone is unhappy in their own way.
Some additional things that muddy the waters:
1. Cultural fit - not giving a good score in seen as impolite. As a result you have to be pretty, damned unhappy before considering anything else. When you're users have reached that point you've probably lost them regardless of what you do.
2. Time of use - new users will be enthusiastic and give good scores because new users are enthusiastic and give good scores. By the time they have used the platform for long enough to make a valuable contribution they have generally lost the will to tell you in detail why your product is great or sucks.
2. Sample size, sample size, sample size. The math here is not precise but it's precise enough to tell you if your sample size is too small. The main effect of too small a sample size is the variation in the score is so great it is impossible to infer anything and you may as well abandon the approach lest you end up chasing shadows trying to make something better.
3. Internal use - simply don't do this - ever. There's a vast amount of politicking and employee uneasiness associated with this. Nothing is ever anonymous no matter what is said publicly.
So, in the absence of anything else NPS is probably a reasonable measure if only to generate a conversation internally and for trying to gauge whether your efforts are moving the needle and in which direction.
IMO, NPS doesn't provide a good enough signal to be "the one number" you should care about, but it recognizes the actual numbers don't matter all that much. I've seen the same question used with a 5 point Likert scale which I imagine is easier to work with. You'd still have to deal with people responding like "0/10 can't think of any friend I'd recommend this to" :/
Growth is a single number, and NPS is measuring growth, not UX.
I'm no big lover of NPS, but this analysis is awful! He claims it's bad because he doesn't understand it. That's not very scientific either.
I'm not sure NPS works well at all, but the idea behind it is obvious. It's a growth metric. The goal of NPS is to tell you how many new customers an existing customer will refer to you over the lifetime of their account.
This is just like population statistics. NPS is trying to measure your customer birth rate by asking how many customers are (or intend to be) pregnant. It's not an accident that there are only two thresholds, and it's wrong to conclude that these thresholds indicate a problem with the method.
If a customer recommends less than one new customer during the entire time they're with you, then you have a replacement rate less than 1, and you're losing customers over time. If you have a replacement rate between 1 and 2, and your customer lifetime is long (say, years) then you aren't growing fast. If your replacement rate is 2 or higher, and your customer lifetime is short (say, weeks) then you are growing more than 200% month over month virally, without the need for marketing.
What the people who designed NPS did, I am sure (meaning I'm speculating, but giving the strongest possible interpretation), is measure some responses and compare it to the number of actual referrals, then drew the lines where the referral rates cross from negative growth to neutral grow, and from neutral growth to positive growth. That's what I would do. And it seems plausible that people who give a score of 6 or less won't end up referring anyone, on average.
Sadly, the article doesn't conclude with any real alternatives for measuring growth. Since NPS is an indirect growth metric, the better answer may be to simply measure your growth directly. That means understanding engagement and activity, not just counting how many accounts exist, but other than that, counting your active customers is a single number that will reliably tell you growth, and can't be gamed by your customers -- it can only be gamed by yourself and your team.
Connect the dots for me on how NPS measures growth. Where does it tie to growth at all?
> NPS is trying to measure your customer birth rate by asking how many customers are (or intend to be) pregnant.
Horrible analogy, but ok. I'd say, if there's any equivalent, it's asking how many people think they are likely they might get pregnant ever.
> What the people who designed NPS did, I am sure (meaning I'm speculating, but giving the strongest possible interpretation), is measure some responses and compare it to the number of actual referrals, then drew the lines where the referral rates cross from negative growth to neutral grow, and from neutral growth to positive growth.
They didn't do anything like that.
> And it seems plausible that people who give a score of 6 or less won't end up referring anyone, on average.
It does seem plausible. It isn't validated by any science, but it's certainly plausible. (Like the earth is plausibly flat.)
> Since NPS is an indirect growth metric, the better answer may be to simply measure your growth directly.
Agreed!
> They didn't do anything like that.
Now I'm really confused by your statements. I just read the link to the original source that you posted on hbr.org. What Reichheld described is exactly what I said above, he correlated survey responses against actual growth rates, and drew the lines between negative and positive growth rates. Not only that, he asked the question multiple different ways, and found out which question statistically landed the most accurate answers.
Why are you claiming they didn't do that? Are you saying the article is lying about the data they used to come up with NPS?
Instead of investigating the possibly legitimate reason NPS people might be doing this, you put up a straw man argument about all respondents giving the same score. The likelihood of all respondents in a large survey all giving 8's is very, very close to zero. The likelihood of your NPS score suddenly flipping from 0 to 100 is very, very close to zero.
So you got my analogy and suggested an alternative, but you still don't see how probability of referral (or birth) is an indicator of business (or population) growth? You do seem to get it, so I don't understand what you're missing. I'm not sure how to (or if I need to) explain it better.
Polling a bunch of people how likely they are to refer a friend is like sampling the derivative of the growth function you want to estimate. If everyone responds accurately and tells the truth, and they refer people at the rate they said they would, you can use the data to predict your growth.
The fact that NPS puts the negative growth line at 60% says, to me, that they concluded that people inflate their self-reporting referral probabilities.
There is a mapping between what people report, and what they do. NPS might have the mapping wrong, but there is a mapping. I don't expect the NPS mapping to be very accurate, but if it's wrong I'd like to hear why. You haven't explained why it's wrong because you don't seem to understand why it might be right.
> Horrible analogy, but ok. I'd say, if there's any equivalent, it's asking how many people think they are likely they might get pregnant ever.
I don't understand what you're arguing (or why), you're splitting a very fine hair here, the difference between what you suggested and what I said is subtle at best. The NPS question is how probable are you to recommend this service to a friend. Someone with a low probability is likely to recommend to 0 friends. Someone with a (self reported) medium probability may be likely to refer 1 friend. Someone with a high probability may be likely to recommend 5 friends.
Asking a yes/no how likely one is to ever get pregnant would be a worse proxy for population growth than asking how many pregnancies you expect in your lifetime. The NPS question doesn't exactly ask either of those, it can be interpreted either way.
> They didn't do anything like that.
So what did they do? Your post ignores that question and argues it's purely bogus. I don't even know what they did, and I don't buy that NPS is pure fiction with nothing at all to back it. I totally would buy that the NPS scale was based on a small sample, and that it doesn't fit many companies very well.
> Like the earth is plausibly flat.
Not sure I get where the snark here is coming from. There exists an average response to this survey that is between 1 and 10 where below that number, statistically people will not refer anyone. What is that number? Why does 6 seem as plausible to you as the earth being flat?
I am giving them the benefit of the doubt, I assume it was based on more than a guess.
Now, instead of assuming, I'm going to look it up.
It turns out they described what they did, and it seems that it was based on some actual data:
"So what would be a useful metric for gauging customer loyalty? To find out, I needed to do something rarely undertaken with customer surveys: Match survey responses from individual customers to their actual behavior—repeat purchases and referral patterns—over time. I sought the assistance of Satmetrix, a company that develops software to gather and analyze real-time customer feedback—and on whose board of directors I serve. Teams from Bain also helped with the project.
"We started with the roughly 20 questions on the Loyalty Acid Test, a survey that I designed four years ago with Bain colleagues, which does a pretty good job of establishing the state of relations between a company and its customers. (The complete test can be found at http://www. loyaltyrules.com/loyaltyrules/acid_test_customer.html.) We administered the test to thousands of customers recruited from public lists in six industries: financial services, cable and telephony, personal computers, e-commerce, auto insurance, and Internet service providers.
"We then obtained a purchase history for each person surveyed and asked those people to name specific instances in which they had referred someone else to the company in question. When this information wasn’t immediately available, we waited six to 12 months and gathered information on subsequent purchases and referrals from those individuals. With information from more than 4,000 customers, we were able to build 14 case studies—that is, cases in which we had sufficient sample sizes to measure the link between survey responses of individual customers of a company and those individuals’ actual referral and purchase behavior.
"The data allowed us to determine which survey questions had the strongest statistical correlation with repeat purchases or referrals. We hoped that we would find at least one question for each industry that effectively predicted such behaviors, which can drive growth. We found something more: One question was best for most industries. “How likely is it that you would recommend [company X] to a friend or colleague?” ranked first or second in 11 of the 14 cases studies. And in two of the three other cases, “would recommend” ranked so close behind the top two predictors that the surveys would be nearly as accurate by relying on results of this single question."
As this article identifies correctly, the more important question is the harder-to-quantify “why?”.
Armed with this, the fluctuating and heavily biased score will have more meaning and context. Using that data to improve the service/product for the next cycle may do more to realistically reflect sentiment change than any weirdly random maths.
Of course, this still ignores what someone actually does do. But it’s possible to track certain cases of referral and actually to encourage trackable referrals using the correct approach. This is a more solid measure (if possible) than any hypothetical gauge of future behaviour - again as the article points out.
I totally agree that NPS feels like placing faith too much in something magical. However, I don't feel totally convinced by this article; when people calculate NPS, do they really ignore any other analysis on the raw input (i.e. all 0's vs all 6's)? Yes, there are totally exceptional datasets that make the NPS look like it hasn't move; has that been a problem for people in the wild?
Also, is picking one ecommerce customer out of the data enough to say that there's no correlation between NPS and future behavior?
I've seen NPS used as a way of keeping a pulse on a community; if it drops sharply, something is clearly wrong in a way that normal monitoring can't surface.
If this is your goal (and it's a good goal), there are way better questions than NPS to use here. I'd go with a simple "How did we do today?" question, versus the convoluted NPS mechanism.
NPS is not harmful, poor leadership and failure to correctly manage a feedback program is.
The real lessons are in the verbatims.
- Yes, I would bring up how much I like [product]
- Yes, if the subject of [product category] came up
- Yes, if my friend had a problem that [product] could solve
- No, because I don't care enough about [product category]
- No, because I don't like [product] enough
- No, because I dislike [product]
Which is a terrible metric that pretty much nobody should use.
1. Create a strong worded post that attracts attention
2. Attach to this post your sales pitch.
But the NPS number is more a number for executives than people working directly on the product. NPS doesn't give UX people anywhere near enough information to do their job. None of these top-line numbers have enough information for that.
I found myself thinking about this: http://www.ianbicking.org/blog/2016/04/product-journal-data-... – there is a real dysfunction that the author is seeing, and is common. Top-down process design means that executives design the work process and give it to their managers, and managers design the process and give it to their reports, and you end up with processes that aren't designed to help individuals do their own jobs, instead everyone is supporting someone else's job. Executives make broad decisions: is this product succeeding? If we continue our current approach where will that take us? Are the teams performing well? Someone who is doing UX shouldn't be worried about these questions, they should be concerned about specific details, because those specific details are what a UX designer can change.
That being said, I agree that a three point scale would be better in most cases.
The focus on the number proves the saying: when a metric turns into a goal, it ceases to be useful. The goal should be "dollars earned." All the metrics should just assist in improving this metric. Else you'll get the bad statistical hacking described. Those hacks are not just for NPS; I think we've all seen them from various companies.
Of course, dollars earned also ends up being gamed (see also: Enron) but at least there are usually better safeguards against this in most organizations. And increasing dollars earned are usually a good end state.
For a big company (Forbes 500), or one with an effective monopoly (cable companies, airlines), or one with big enough backers (some SV startups) it may not matter. In those cases, the company doesn't need any single individual, and the individual doesn't have the power to really hurt the company.
For your local "mom and pop" company in a small town with only a few employees, the difference between a 6 and a 0 might be massive, perhaps to the point of staying in business or not.
You can choose to go down and focus on the 0s too, but they help to often spot bigger pain points that could shape the product.
Again, the thing to remember with any number is we optimize for what we measure. NPS can be helpful but you need to have the NPS feedback machine running smooth or else you are going to spend your time building/running that vs. actually doing your thing.
The simplicity of NPS also encourages its implementation rather than the most common alternative of measuring nothing.
That's the thing though: The progress from 0 to 6 is not reflected in any score.
The theory of NPS is that what matters is what people say about you. If people from 0-6 are all going to say negative things when asked, then lumping them in the same bucket is reasonable. It may not be as good as a more subtle scale, but it may be much better than thinking the numbers are linear.
It's possible that one could come up with a mapping that's even better, of course. But NPS is simple enough that even executives understand it. A marginally more-accurate number that nobody understands is probably worse, because people will trust it less. The point of the NPS score is not theoretical accuracy, it's motivating change.
