Your boss walks in with a chart of the last 12 months of transactional Net Promoter® survey results and he’s not happy!

The score went down last month and he want’s to know why. Looks like you’ll have to hunt around to find a reason for the change; or will you?

Just because your survey score has gone down, or up, doesn’t mean that there has actually been a change in the overall business NPS. It might just be a fluke of the sample you have collected. The change might be within the Margin of Error.

## What is Margin of Error?

When you run a survey, say NPS, you are trying to determine the NPS of all your customers. The problem is that you are never able to collect a response from every single customer. In reality you make do with a sample; maybe 10% of your customers respond. So, rather than calculating the NPS of all your customers you are only estimating it based on the customers who responded.

Now, by random chance in this survey you might get responses from a set of very happy or very unhappy customers. This results in the actual score being lower or higher than you the score for the sample you have collected.

The problem is: how do you know how close your estimate is to the actual NPS?

You can discover this by calculating a Margin of Error. This will tell you that you can be, say, 95% certain that the NPS for all your customers is between your sample score plus the Margin of Error and the sample score minus the Margin of Error.

So before you break out the hard hats and wait for the blame game to start you need to determine if a real change has occurred. The problem with Net Promoter is that the statistics that you normally use for survey scores don’t work so well for NPS.

However, there is an approach that you can use to determine if the change is significant. This post “How can I calculate margin of error in a NPS result?” provides a very good and detailed response to the question. Unfortunately, if you’re not statistically inclined reading the post may not help very much. So here I will take you through the process step by step.

## Calculating Margin of Error for Net Promoter®

First you need to know more than just the score, you need the actual number of Promoters, Detractors and Neutrals in your sample:

- #P is the number of promoters
- #N is the number of Neutrals
- #D is the number of Detractors.

Now we calculate the total respondents:

#T = #P + #N + #D

First calculate NPS. In this case we are getting a -1 to +1 score which is really NPS / 100:

NPS = #P/#T - #D/#T

Now work out the Variance of the sample NPS using the discrete random variable approach:

Var(NPS) =

(1 – NPS) ^ 2 * #P/#T + (0 - NPS) ^ 2 * #N/#T + (-1 - NPS) ^ 2 * #D/#T

Now calculate the Margin of Error (MoE) for your sample:

MoE = SQRT(Var(NPS)) / SQRT(#T)

Remember that this is the MoE for a -1 to +1 NPS so to get this back to the same range as your normal NPS you need to multiply it by 100.

## Using Margin of Error

The easiest was to use Margin of Error is add error bars to your charts. Simply add a couple of rows of values to your chart:

NPS + MoE and NPS - MoE

Another approach is to perform the calculation for two different samples and end up with two MoE.

To compare two such results you need to account for the possibility of error in each. When survey sizes are about the same, the standard error of their difference can be found by a Pythagorean theorem: take the square root of the sum of their squares. [source]

Using this approach you can use this information to determine if your score probably (95%) changed between samples thus:

If ABS(NPS1 – NPS2) > 2 * SQRT (MoE1 ^ 2 + MoE2 ^ 2)

Then there has probably been a real change in the population NPS, where 2 is an approximation for the t statistic for the 95% point.

If that still looks like it’s too much maths to do, then you can download our handy dandy NPS Margin of Error Calculator spread sheet. All you need to do is enter the number of #P, #N and #Ds for each sample and it will tell you if the score has really changed and even provide a chart for you.

## Other Statistics for Net Promoter

You can also use Chi-Squared tests on Net Promoter. For more information on using this technique check out this post: Using Chi-Squared tests on Net Promoter Data.

Andre says

Hello,

I have a question about the values involved in the calculation.

In the worksheet is launched only the total value of Promoters, Detractors and Neutrals, but there is no comparison with total sent to research (information A and B).

Example:

A – Total base – 25.000 visits per month

B – Total sent to research – 6.000 visits (records with e-mail)

Total responses: August 2012: Total responses – 1.711 (Detractors – 467 , Neutrals – 351 , Promoters – 893)

The margin of error is not related to total visits compared with total answers too?

Adam Ramshaw says

Andre,

Thanks for your question.

No the NPS Margin of Error is not directly related to the number of visits compared with the total answers as we are talking about a sampling approach.

For very small samples there may need to be some adjustment but most NPS samples are large enough to use the formula in the post.

Adam

Sergio says

Hi Adam,

I´m sorry, but I still don´t get it… so you mean the MoE would be the same even if the total base would be 1711 visits (the total of aswered survey)? In this case, the MoE shouldn´t be zero?

Whit Wilson says

This is really great–thanks!!

Adam Ramshaw says

Whit,

Glad you liked it.

Adam

Lauren says

I have tried to download the calculator and it doesn’t appear to have been sent or gone into my spam filters.

Adam Ramshaw says

Lauren,

That is probably because on the form you unselected “Request Download”. Try again with that box selected.

Adam.

DPMCooney says

Hi,

I’m not sure I follow the pooled variance:

Why do you have it as if greater than 2 * SQRT (MoE1 ^ 2 + MoE2 ^ 2) as apposed to just SQRT (MoE1 ^ 2 + MoE2 ^ 2)?

Thanks,

Adam Ramshaw says

Dom,

Technically, it should be 1.96 for the two-sided hypothesis test at level alpha=0.05. We have used an approximation for the calculator

(Thanks to thomas t john for correcting me on this point)

Is that clear?

Adam

DPMCooney says

Hi,

I think. So in your calculator MoE for difference (Cell C15) is 2.18 (with the default NPS sample data). But what you’re saying is it is infact 2*2.18 in this case (4.37)?

Sorry to be a pain, but the cells are locked and I just like to understand the math behind it.

Thanks,

Adam Ramshaw says

Dom,

It might be easier if you looked at this blog post as it explains the calculation in more detail: https://www.genroe.com/blog/how-to-calculate-margin-of-error-and-other-stats-for-nps/5994

Tom says

If I am understanding correctly what you are trying to do… the “2 times” in the check “If ABS(NPS1 – NPS2) > 2 * SQRT (MoE1 ^ 2 + MoE2 ^ 2)” comes from the standard normal. Technically, it should be 1.96 for the two-sided hypothesis test at level alpha=0.05.

Based on this, the following clarification you provide is not correct: “It is 2 x the difference because you have to make sure the difference between the two scores is twice the MoE. One for each score.”

Adam Ramshaw says

Tom,

Yes you are correct. The 2 is an approximation.

I’ll correct my earlier comment.

Adam

Pam says

Adam,

Thanks for that clarification. I was wondering where the Z(sub C) was from the standard formula for E = Z(sub C) * Sigma / n.

thanks for the clarification. I wanted to make sure that MoE for NPS was no different than any other typical MoE calculation.

Again thanks,

Pam

Cindy says

I am trying to compare NPS scores for my client relative to its competitors. My client and competitor scores are coming from respondents in the same sample. Do you have a test for this? Also, you are testing at 95% LOC. Do you have a spreadsheet at 90% LOC?

Lisa says

Hi,

I do have 50% of respondents in 7 and 8 – so I do not have a normal curve but a shifted one. How do I calculate then?

Thanks a lot

Tena says

We could check whether the NPS has changed ….how do we check whether it has improved statistically ?

Adam Ramshaw says

Tena,

Thanks for dropping by.

If you use the Margin of Error process described here then a change (see last equation) indicates a significant change.

The easiest thing is probably to just download the Excel calculator above. That tests for statistical improvement in two ways (MoE and Chi-Squared) and gives you a simple read out of whether a significant change has occurred: Yes or No.

Catherine Good says

I have always understood that the MOE you reference is only applicable to random samples of customers. We do not sample from our population of customers. We invite every customer to participate and about 5% do. Can we still use the MOE calculations you recommend?

Adam Ramshaw says

Catherine,

That’s a good question (sample Vs census) and not one I’ve considered before. I expect that as you are receiving a 5% response rate that can be taken as a sample and the same approach

Regards,

Adam Ramshaw

Paul V says

I don’t understand why we need to assume detractors, passives and promoters are discrete random variables? Aren’t they merely groups from an existing continuous variable?

If we have the raw data (the 0-10 ratings from respondents) – wouldn’t the standard error of NPS just be: Standard Deviation of response values /SQRT(sample size).

Adam Ramshaw says

Paul,

Thanks for stopping by.

The point you make is a good one and has been suggested by another person in the past.

I’m not a statistics professional and to date I’ve been influenced in the treatment of the data by someone I trust in this area: a Professor with a specialisation in data analytics.

My concern on the approach you note is that NPS is a net proportion (the net is the concern so the range changes from 0->100 for the input score to -100->+100 for the outcome) and that the proportions of the 0-10 scale are different for each group.

Would you care to comment?

Regards,

Adam Ramshaw

Vanessa says

Hi Adam,

Could I use the formula: If ABS(NPS1 – NPS2) > 2 * SQRT (MoE1 ^ 2 + MoE2 ^ 2)

to check if the diffrence is significant, if my NPS is for example a 3 month rolling score?

So, for example:

NPS1: January to March

NPS2: February to April

Adam Ramshaw says

Vanessa,

You can use the same statistics for comparing any two sets of scores but they should be on the same basis as your 3 month rolling averages are.

Regards,

Adam

Vanessa says

Thank you for your answer,

I’m trying to understand why the Margin of error for NPS would be smaller than for a simple average.

I have an NPS for a sample of 75 cases and the margin or error is around +/- 7

The Margin of error for a sample of 75 cases is around 12

How can I explain that the NPS that fluctuates more than a normal average can have a smaller MoE? or maybe I’m missing something?

Adam Ramshaw says

Vanessa,

The key thing to note is that MoE for NPS changes with the proportion of Neutrals/Detractor/Promoters and that common MoE calculators use a standard worst case 50% proportion in the calculation.

So the NPS calculator is a more accurate estimate.

If you try a few different example sets of N/D/P you’ll see the change in MoE.

Regards,

Adam Ramshaw

paul goodhew says

HI Adam, you give the margin of error calculation as:

MoE = SQRT(Var(NPS)) / SQRT(#T)

But isn’t this just the calculation for the standard error? Shouldn’t you multiply this value by the z score corresponding to a 95, 90, 99% etc confidence?

i.e. MoE = SQRT(Var(NPS)) / SQRT(#T) * 1.96 for 95% confidence?

if so, then your spreadsheet is also wrong.

Paul Goodhew

Adam Ramshaw says

Paul,

I think that the question you have here is one of definitions. I’d direct you to the comments of the source article (http://stats.stackexchange.com/questions/18603/how-can-i-calculate-margin-of-error-in-a-nps-net-promoter-score-result) where the question you have asked is answered by it’s author.

Regards,

Adam Ramshaw

paul goodhew says

But you’ve suggested that

“The easiest was to use Margin of Error is add error bars to your charts. Simply add a couple of rows of values to your chart: NPS + MoE and NPS – MoE”

This would provide a margin of error with only 68% confidence which is neither standard practice nor useful. The margin of error calculation means nothing unless you specify a confidence level.

Given people are probably coming to this blog for a quick solution to their problem could I suggest you amend the calculation to calculate a 95% confidence level which is pretty standard in business, and probably what most people are looking for?

Larry says

Adam,

The calculator has been very helpful and handy for some of the work I do. My question is, how do interpret the result if MOE test and Chi-Squared test give me conflicting result? One says yes and the other says no?

Thanks,

Larry

Adam Ramshaw says

Larry,

Thanks for your comment.

Each test uses a different technique to determine it’s result so a split decision is very possible. What you do about it depends on the consequences.

If you want to be conservative you will wait until both tests say the same thing. If you want to be a bit more aggressive with your decision then a split decision is okay.

Like many things in statistics, there is not perfect answer.

Regards,

Adam

Roman says

If you multiply SQRT (MoE1 ^ 2 + MoE2 ^ 2) by 2 when testing significant differences between two NPS, so you need to multiply by 2 your MoE when you add error bars to your charts. In that way it will be NPS + 2*MoE and NPS – 2*MoE

I am confused, that MoE should be equal to half of the confidence interval (95%?), but in this case, you need to do the multiplication in the MoE formula.

MoE = 2*SQRT(Var(NPS)) / SQRT(#T)

Correct me if I’m wrong.

Best regards,

Roman A.

Adam Ramshaw says

Roman,

Sorry to take so long to reply but I’ve been trying to get a good answer for you. I feel that your answer it correct but I’m asking another source to verify.

When I have an answer I’ll post it in the article.

Adam

paul goodhew says

see my post from May 2016. You need to multiply the standard error by the appropriate Z score to give the margin of error for the desired confidence. For 95% confidence this is about 1.96 which is approximately 2.

Adam Ramshaw says

Paul,

Yes that’s my feeling as well but it doesn’t mesh with what the source post says; it basically says MoE = Standard Error. I’ve asked the posts original author to clarify his post or my understanding. Then I’ll update my post accordingly.

Adam

Daniela Castilla Orduz says

Hello Adam,

I just read your article and it is great. Thank you.

I have an important question related with the sample. I work in a big company. We have 42 different cinemas in the country. Recently, we started to use NPS but we do not know how to calculate tne sample per cinema. I mean, it is neccesary to collect an specific percent of answers based in the quantity of spectators? For example, if 20.000 customers assist to one of our cinemas each month, that means that we should apply the survey to 2%, 5% or maybe 10% percent of that group to guarantee that it is a representative sample?

I’d appreciate your help very much. Thank you.

Adam Ramshaw says

Daniela,

This is a good question – an I’m in the middle of writing a detailed blog post on it.

The short answer is no there is no minimum percentage to be “representative”. You’ll see the calculations on this page talk about numbers of responses not proportions.

You need to try to get a random sample though.

Regards,

Adam

Daniela Castilla Orduz says

Adam, I am so thankful because of your timely answer.

There are something that is not clear to me.

For example, if I have two NPS of two different cinemas in january.

Cinema 1: Spectators: 18509. Survey responses: 17. Responses percent: 0,1%. NPS: 76.

Cinema 2: Spectators: 44190. Survey responses: 200. Responses percent: 0,5%. NPS: 43.

My first question is: Can I do a valid comparison between the two cinemas?

And my second question is: This percents of survey responses are statistically valids?

I’d appreciate your help.

Thank you again!

Adam Ramshaw says

Daniela,

You can certainly compare these two cinemas. The problem you face is that the margin of error on the 17 response cinema will be very large so it might not be a useful comparison. You can use the estimator to understand how large that range it.

Note that the response rate is less important in these areas that the total number of responses. So, in general, the 17 < -> 200 comparison is more important than the 0.5% < -> 0.1%.

Regards,

Adam

Brendan says

This is brilliant, thank you Adam.

Adam Ramshaw says

Brendan — you’re welcome. Glad you found it useful.

Ann says

Hi Adam

What happens if a customer originally scores us low on NPS, but then, after feedback and conversation, wants to change their score. Does on merely replace their old score with the new higher score, or does the old score stand, but the new score is added.

Adam Ramshaw says

Ann,

That’s a good question. I’m not sure there is a perfect answer but personally I’d be inclined to let the first score stand on the basis that it required “feedback and conversation” to make the change. Unless you do this with every customer the first score is more representative of your current operations.

Adam

Daniela Castilla Orduz says

Hello Adam.

I had written to you before and your help was very useful.

I work in Cine Colombia, an important cinema company in Colombia.

We have been doing the NPS month to month for one year in 42 movie theaters across the country. Nowadays we have 22,000 surveys. Nevertheless we have some questions:

1. I want to illustrate the first question with an example:

We applied the NPS Survey in august in our 42 cinemas. In one of them we had 5,126 customers and 10 survey responses. In another theater we had 53,933 customers and 48 survey responses. Can we compare the NPS of these theaters in spite of the different number of responses or do we need similar samples?

2. Following the previous example we would like to know how can we determine the sample size for each theater? We have been using margin of error calculation based in your recommendations but the margin of some theaters is very high. What can you recommend us for this problem? Is it valid if we only collect 10 responses in one theater or do we need a minimun response quantity?

3. We are doing the question for each theater but not for the entire brand, I mean: “How likely is it that you would recommend [Theater 1] to a friend or colleague?

Is it correct to obtain the brand’s NPS (global NPS) from theaters’ NPS or the correct question should be How likely is it that you would recommend [the company] to a friend or colleague?

In advance I appreciate the help you could give us.

Regards,

Daniela.

Adam Ramshaw says

Daniela,

Thanks for your message. Just responding by your numbers:

1 – yes you can compare these NPSs but I would not. Your response rate is very low (0.2%) and I think that the chances your sample is skewed is quite high. While you can use such small samples sizes in certain circumstances, you need to actively ensure it is not skewed.

2 – yes – for small response volumes your margin of error will be high.

3 – yes to get the brand NPS you would need to change the question.

Overall, I would recommend that you need to get much larger sample sizes. At least 100 per location, but preferably more, before you can rely too much on the data.