Your boss walks in with a chart of the last 12 months of transactional Net Promoter survey results and he’s not happy!
The score went down last month and he want’s to know why. Looks like you’ll have to hunt around to find a reason for the change; or will you?
Just because your survey score has gone down, or up, doesn’t mean that there has actually been a change in the overall business NPS. It might just be a fluke of the sample you have collected. The change might be within the Margin of Error.
In this post we’ll run through how to calculate all of the statistical values you need, or you can skip to my free Excel spreadsheet and just use that.
What is Margin of Error?
When you run a survey, say NPS, you are trying to determine the NPS of all your customers.
The problem is that you are never able to collect a response from every single customer. In reality you make do with a sample; maybe 10%, 20% or whatever of your customers respond.
So, rather than calculating the NPS of all your customers you are only estimating it based on the customers who responded.
Now, by random chance you might get more responses from very happy or very unhappy customers. This results in the actual score being lower or higher than you the score for the sample you have collected.
The problem is: how do you know how close your sample estimate is to the actual NPS?
You can discover this by calculating a Margin of Error.
With Margin of Error, you can be, say, 95% certain that the NPS for all your customers is between your sample NPS plus the Margin of Error and your sample NPS minus the Margin of Error.
If you’ve ever tried to understand the statistical terms around error and confidence intervals it’s easy to become confused. So just quickly I want summarise the meaning of each for clarity.
Standard Error – is a generic term that applies to many types of sample statistics (mean, standard deviation, correlation, etc). When we talk about Averages, the Standard Error is equal to the Standard Deviation of the sample.
Margin of Error – the amount within which you are confident (e.g. 90%, 95%) that the population mean is above or below the sample mean.
Confidence interval – this is the range within which you are confident (e.g. 90%, 95%) the population mean lies
Here is how those three terms are related.
Firstly Margin of Error and Standard Error:
Where Z is a value related to level of confidence, 90%,95% etc and is approximately 2 for a 95% level of confidence.
So before you break out the hard hats and wait for the blame game to start you need to determine if the change in score is greater than the Margin of Error, i.e. outside the Confidence Interval.
Calculating Margin of Error for Net Promoter®
The problem with Net Promoter is that the statistics that you normally use for survey scores don’t work so well for NPS.
However, there is an approach that you can use to determine if the change is significant. This post “How can I calculate margin of error in a NPS result?” provides a very good and detailed response to the question.
Please note that the above referenced post equates Standard Error and Margin of Error. This is a little unusual and in the rest of this post we have preserved the approach in that post but used the terms Standard Error and Margin of Error in the more common way.
If you’re not statistically inclined, reading the post may not help very much. So here I will take you through the process step by step.
First you need to know more than just the score, you need the actual number of Promoters, Detractors and Neutrals in your sample:
- #P is the number of promoters
- #N is the number of Neutrals
- #D is the number of Detractors.
Now we calculate the total respondents:
Next calculate a version of NPS that is -1 and 1. We don’t multiply by 100 as we normally do. Here is the equation:
Now determine the Variance of the sample NPS using the discrete random variable approach:
Now calculate the Standard Error for your sample:
And, as above, our Margin of Error is approximately 2 times this value so:
Remember, this is the MoE for a -1 to +1 NPS so to get this back to the same range as your normal NPS you need to multiply it by 100.
Using Margin of Error
One way to use Margin of Error is add error bars to your charts. Simply add a couple of rows of values to your chart:
- NPS + MoE, and;
- NPS – MoE
Another approach is to perform the calculation for two different samples and end up with two Standard Errors you can compare.
To compare two such results you need to account for the possibility of error in each. When survey sizes are about the same, the standard error of their difference can be found by a Pythagorean theorem: take the square root of the sum of their squares. [source]
You can now use this information to determine if your score probably (95% confidence) changed between samples. Again the 2 in this formula is an approximation for the 95% confidence Z value of 1.96:
If that looks like it’s too much maths you can download our handy dandy NPS Margin of Error Calculator spread sheet.
All you need to do is enter the number of #P, #N and #Ds for each sample and it will tell you if the score has really changed and even provide a chart for you.
How many responses do you need for NPS?
Lets’ do a couple of real world examples to get a feel for how different volumes of responses relate to the MoE in your NPS analysis.
Each of the examples generates an NPS of 10 but it takes a relatively large number of responses (at least 100) to start to reduce the statistical uncertainty in the score.
- 10 responses
- 4 Promoters
- 3 Neutrals
- 3 Detractors
MoE: 52.6 points
- 100 responses
- 40 Promoters
- 30 Neutrals
- 30 Detractor
MoE: 16.8 points
- 1000 responses
- 400 Promoters
- 300 Neutrals
- 300 Detractor
MoE: 5.8 points
Other Statistics for Net Promoter
You can also use Chi-Squared tests on Net Promoter. For more information on using this technique check out this post: Using Chi-Squared tests on Net Promoter Data.