Your boss walks in with a chart of the last 12 months of transactional Net Promoter survey results and he’s not happy!
The score went down last month and he want’s to know why. Looks like you’ll have to hunt around to find a reason for the change; or will you?
Just because your survey score has gone down, or up, doesn’t mean that there has actually been a change in the overall business NPS. It might just be a fluke of the sample you have collected. The change might be within the Margin of Error.
What is Margin of Error?
When you run a survey, say NPS, you are trying to determine the NPS of all your customers. The problem is that you are never able to collect a response from every single customer. In reality you make do with a sample; maybe 10% of your customers respond. So, rather than calculating the NPS of all your customers you are only estimating it based on the customers who responded.
Now, by random chance in this survey you might get responses from a set of very happy or very unhappy customers. This results in the actual score being lower or higher than you the score for the sample you have collected.
The problem is: how do you know how close your estimate is to the actual NPS?
You can discover this by calculating a Margin of Error. This will tell you that you can be, say, 95% certain that the NPS for all your customers is between your sample score plus the Margin of Error and the sample score minus the Margin of Error.
So before you break out the hard hats and wait for the blame game to start you need to determine if a real change has occurred. The problem with Net Promoter is that the statistics that you normally use for survey scores don’t work so well for NPS.
However, there is an approach that you can use to determine if the change is significant. This post “How can I calculate margin of error in a NPS result?” provides a very good and detailed response to the question. Unfortunately, if you’re not statistically inclined reading the post may not help very much. So here I will take you through the process step by step.
Calculating Margin of Error for Net Promoter®
First you need to know more than just the score, you need the actual number of Promoters, Detractors and Neutrals in your sample:
- #P is the number of promoters
- #N is the number of Neutrals
- #D is the number of Detractors.
Now we calculate the total respondents:
#T = #P + #N + #D
First calculate NPS. In this case we are getting a -1 to +1 score which is really NPS / 100:
NPS = #P/#T – #D/#T
Now work out the Variance of the sample NPS using the discrete random variable approach:
(1 – NPS) ^ 2 * #P/#T + (0 - NPS) ^ 2 * #N/#T + (-1 - NPS) ^ 2 * #D/#T
Now calculate the Margin of Error (MoE) for your sample:
MoE = SQRT(Var(NPS)) / SQRT(#T)
Remember that this is the MoE for a -1 to +1 NPS so to get this back to the same range as your normal NPS you need to multiply it by 100.
Using Margin of Error
The easiest was to use Margin of Error is add error bars to your charts. Simply add a couple of rows of values to your chart:
NPS + MoE and NPS – MoE
Another approach is to perform the calculation for two different samples and end up with two MoE.
To compare two such results you need to account for the possibility of error in each. When survey sizes are about the same, the standard error of their difference can be found by a Pythagorean theorem: take the square root of the sum of their squares. [source]
Using this approach you can use this information to determine if your score probably (95%) changed between samples thus:
If ABS(NPS1 – NPS2) > 2 * SQRT (MoE1 ^ 2 + MoE2 ^ 2)
Then there has probably been a real change in the population NPS, where 2 is an approximation for the t statistic for the 95% point.
If that still looks like it’s too much maths you can download our handy dandy NPS Margin of Error Calculator spread sheet. All you need to do is enter the number of #P, #N and #Ds for each sample and it will tell you if the score has really changed and even provide a chart for you.
How many responses do you need for NPS?
Lets’ do a couple of real world examples to get a feel for how different volumes of responses relate to the MoE in your NPS analysis.
Each of the examples generates an NPS of 10 but it takes a relatively large number of responses (at least 100) to start to reduce the statistical uncertainty in the score.
- 10 responses
- 4 Promoters
- 3 Neutrals
- 3 Detractors
MoE: 26.3 points
- 100 responses
- 40 Promoters
- 30 Neutrals
- 30 Detractor
MoE: 8.3 points
- 1000 responses
- 400 Promoters
- 300 Neutrals
- 300 Detractor
MoE: 2.6 points
Other Statistics for Net Promoter
You can also use Chi-Squared tests on Net Promoter. For more information on using this technique check out this post: Using Chi-Squared tests on Net Promoter Data.