In a recent post I discussed using the Margin of Error method to determine if the difference between two Net Promoter Scores® is probably real or just statistical noise. A sharp eye reader has identified that you can also use a Chi -Squared test to perform this test.

The advantage of the Chi-Squared test is that it is more discerning. For instance consider these two set of Net Promoter results:

- Sample A: Promoters=100,Neutrals=0,Detractors=100 => NPS= 0
- Sample B: Promoters=0,Neutrals=200,Detractors=0 => NPS= 0

You can see that there is something qualitatively different between these two scores. However, looking at the score and even using the Margin of Error calculation implies that the scores are no different.

## Enter the Chi-Squared Test

This statistical test can be used to determine if two sets of categorical data are different. Categorical data just means responses that are not numbers, e.g. Promoter or Detractor.

**Aside**: Interestingly the response from the “would recommend” question is numerical but by transforming the response into Promoters, Neutrals, Detractors the data becomes categorical.

The test basically works by looking at what we expect to happen and what actually happens, then looking at how similar they are.

This post is not trying to replicate a statistics course though so for more details on the actual calculation try starting at Wikipedia.

## Using the Chi-Squared Test

The chi-squared test is sensitive to shifts in the underlying values making up the NPS, not just the NPS score itself. So you can use this test to compare your Net Promoter samples.

Let’s go back and re-test our original data with this approach:

Chi-squared Value = 400.00

Critical Value = 5.99 (for 95% confidence, 2 degrees of freedom)

The Chi-squared value > Critical Value therefore there has been a change.

Once again that maths for this calculation, while not complex, can be a little difficult to follow. So we have updated our free download tool to be a Net Promoter Comparison Tester.

I would like to thank Darren Nicholson for being that sharp-eyed reader and laying out the Chi-Squared test.

Do you have other approaches that you would like to share with the NPS® community? Leave a comment below.

Adam Ramshaw has been helping companies to improve their Net Promoter® and Customer Feedback systems for more than 15 years. He is on a mission to stamp out ineffective processes and bad surveys.

Frank says

July 30, 2016 at 12:29 amThank you for this post. How did you arrive at the Critical Value 5.99?

Adam Ramshaw says

July 30, 2016 at 4:06 pmFrank,

This is the Chi-squared critical value for 95% confidence, 2 degrees of freedom.

Regards,

Adam

Stacey L Smith says

February 18, 2017 at 7:26 amWould you please help me with language in interpreting the test results in the Comparison Tester calculator? I’m not quite sure I understand; would you tell me if this is correct interpretation?

If the results for the MoE test are “Yes”, then is it correct to say “there is too much sampling error and you can’t have confidence in the results that the sample is representative of those who took the training. The results could be a fluke.”

“No” means less sampling error and you can have confidence in the results that the answers are not a fluke and the result of the intervention/training. The differences are real.

Adam Ramshaw says

February 20, 2017 at 7:46 amStacey,

Thanks for your message.

Your interpretation of the results is the

reverseof how it it should be read. YES means that the results probably (95%) different. NO means it is not clear they are different.Adam

Stacey says

February 20, 2017 at 1:44 pmThank you so much!

Andrew says

August 23, 2017 at 4:37 amCould be very difficult to explain if reporting only NPS

your example

A: 100,0,100: 0 NPS

B: 0,200,0: 0 NPS

Significantly different

A: 42,100,58: 8 NPS

B: 58,100,42: -8 NPS

Not significantly different

Adam Ramshaw says

August 23, 2017 at 7:45 amAndrew,

I agree, reporting only NPS can be a misleading. I’d generally suggest reporting at least response numbers and it’s also good to report splits.

Adam

Sailesh Dhital says

September 26, 2017 at 8:14 pmHello,

How did you calculate the below values?

Chi-squared Value = 400.00

Critical Value = 5.99 (for 95% confidence, 2 degrees of freedom)

Could you please explain this to me in details? Thanks in advance!

Adam Ramshaw says

September 27, 2017 at 10:32 amSailesh,

Sorry, this post is not trying to replicate a statistics course though. For more details on the actual calculation try starting at Wikipedia.

Adam