Most People Don’t Understand Sample Size


bigstock-young-business-man-holding-his-53002933You’ve spent weeks working through the numbers to unpick what customers are saying. After checking through the data and analysing a range of root causes, you have created a really practical plan to solve a key customer issue.

The PowerPoint presentation you’ve created nails each of the points you want to make. It starts right up front with the bad news: Net Promoter data for the business group in question.

Before you walked in, you were fully prepared for this meeting, so how is it that 3 minutes in you’re under attack on the very first slide? A senior manager is pointing accusingly at the screen: “I don’t believe that NPS – you don’t have a big enough sample size.”

This is a problem that I am increasingly hearing from customers. When their colleagues in the company don’t like the results from a Net Promoter or customer satisfaction survey they are blaming the sample size.

Don’t like the score their department has received? Blame it on sample size. Don’t like actions that are being recommended and want to drag the anchor on making change? Blame it on sample size.

More and more people are starting to hear about sample sizes and calculations and are using their half knowledge to frustrate the process. Typically, they have just enough information to be dangerous.

For anyone that really understands sample size and the way statistics operate, this is frustrating because it mischaracterises sample sizes and making decisions based on statistical data.

We use statistics to help us make decisions and the only information we really need is enough to distinguish between alternative outcomes. Any more than that is a waste of time and effort.

Here’s Why that Senior Manager is Upset

The reason that executive is so jumpy is that he or she is probably being rewarded based on customer satisfaction or NPS targets. This is a problem.

It’s not talked about very much but one of the key implications of all this sample size and confidence interval conversation is related to personal Key Performance Indicators (KPIs).

Typically, business KPIs are set as hard numbers: $x million in revenue, $y million in costs or $z seconds of average handle time. In all of these cases, hard single numbers are fine because you have access to the whole population, i.e. you can actually count every dollar of revenue.

The finance department is going to get the boot and the IRS is going to be very unhappy if you get revenue and cost wrong by even a few dollars.

On the other hand, CSAT and Net Promoter scores can only ever be estimates. When you state that the NPS for your business or department or division is 25, what you are really saying is:

The NPS of the sample we took was 25. Based on the attributes of the response we received, we can be 95% certain that the NPS of our customer base is between 21 and 28.

That is a very different statement and one that is at the real heart of the issues that we discussed in our introduction. If people are not confident in the robustness and accuracy of the customer feedback, data collection, and reporting process, they will refuse to believe the results and look for flaws in the system. Hence the large number of sample size questions.

If you are holding people to customer feedback goals then you should probably not be doing it the same way that you give revenue goals.

Rather than single numbers, you should opt for ranges, lowest estimated limits or even simple confirmed changes. For example, these would be better NPS KPIs:

  1. NPS for year 22-25
  2. NPS less the Margin of Error > 25
  3. NPS improved year on year

Note that (1) above get complex because you will also need to include minimum sample size and ratios of Promoters/Neutrals/Detractors.

In some ways, example (2) and (3) are the clearest sets of KPIs.

Getting You Back on the Front Foot

So let’s get back to basics and agree on what we are typically trying to achieve in a customer feedback context. I recently wrote about The only statistical analyses you need to use on customer feedback data and explored why you are only looking to perform two sorts of analyses on customer feedback data.

-Is the Score Different?
-Does Changing this Cause That to Change?

Most of these sample size questions are tied up in the first type of analysis. You are trying to determine if an average of a score has changed from one sample to the next.

Let’s examine what that means for sample size.

It is not generally well understood but there is no minimum sample size required for any particular set of statistical analysis. Rules of thumb such as needing a minimum of 100 responses are relevant to providing a “reasonable” level of confidence for a worst case scenario. They are not relevant to looking at a specific situation.

As an example, say you have this question in your customer survey:

How responsive is our sales staff, where 1 is unresponsive and 7 is very responsive?

One of the things you will want to know is has the score changed from the last survey. So what sample size do you need to check if a change has occurred?

Sorry, but we have to dive into some maths for a few seconds to make this point.

The equation for that sample size is:



  • n is the minimum required sample size
  • Z is related to how confident you want to be in the answer, normally expressed as 90% or 95%
  • σ is related to how much inherent variation there is in the population: does almost everyone give you a 5 or 6, or do you get a wide even range of scores all the way from 1’s up to 7’s.
  • E is the margin of error or size of the change you want to detect

Don’t worry too much about the details of the math, but what this says is that sample size (n) is not one number but a range of numbers. It changes with changes in the assumptions you make and the attributes of your customers:

  • n gets larger the more confident you want to be that you have detected a real change
  • n gets larger the more variation you have in your sample
  • n gets larger the smaller the change you want to detect.

So if “responsiveness” has gone up a lot since the last survey you will need a relatively small sample to identify the change. Alternatively, if the change has been small, you will need a bigger sample.

(Readers worried about using ordinal data in this way please read this)

There is no one right sample size. There is only the sample size that is able to detect what you want to detect.

You do need some statistics to demonstrate your position but they don’t have to be difficult. You could start with an easy Excel spreadsheet and a t-Test.

An Extreme Case: 5 Data Points May Be Enough

Let’s goes even further with the idea that sample size is not all it’s cracked up to be.

What if you had only five data points? Could you make any useful prediction with so few data points? The executive in your meeting would probably say no, but they’d be wrong.

Douglas Hubbard discusses the Rule of Five in his book How to Measure Anything.

Rule of 5

[Source](Hubbard, Douglas W. (2010-04-07). How to Measure Anything: Finding the Value of Intangibles in Business (p. 30). Wiley. Kindle Edition.)

Note that we are talking about median not mean (or average) here so it’s a bit different but the idea is still the same. With just 5 data points you can make some pretty strong statements about the underlying population.

If you take a sample of 5 customers and they each score your “responsiveness” between 4 and 6 on a 7-point scale, you can be pretty sure that the median score of all customers will be between 4 and 6.

The point being that you might well be able to make important decisions based on that limited amount of data.

Net Promoter is a Special Case

However, when you start to consider this idea of sample size you need to be aware that Net Promoter is a special case. The sample size calculators that you will find on the internet simple don’t apply because the statistics is different for NPS.

There are a tables provides by vendors in the Net Promoter consulting business (you can Google them) but they overestimate the sample size that you need because they only consider the worst case scenario.

Based on those tables, if, for instance, you had 112 responses, you might think that the sample size was only going to detect changes of +/- 11 points of NPS.

That is the worst case but if those responses looked like this, you could be confident in detecting changes of +/-5 points – more than twice as effective.

  • Number of Promoters: 86
  • Number of Neutrals: 22
  • Number of Detractors: 4

In this case, it is the large proportion of Promoters and the small proportion of Detractors that make the sample size more efficient.

Download this free Net Promoter specific calculator for the backup stats you need for that executive: Net Promoter® Comparison Tester

Getting Past Slide 1

So now, before you present, would be a good time to back to your presentation and think through the sample size question.

You should be prepared to discuss sample size and have the statistics to back up your position.

But, you should also have some sympathy for the senior manager and his or her not-so-perfect customer satisfaction goal.

Share viaShare on Google+Share on LinkedInTweet about this on TwitterShare on Facebook

Surprise: Rob Markey and I agree on Net Promoter® Benchmarking


delight-the-customerAnyone that has been reading this blog for more than a couple of weeks knows that the subject of Net Promoter benchmarking gets me fired up.

In talking to clients and prospects the question of “what’s a good Net Promoter score” almost invariably arises. Many times I have had to choose my words carefully when I tell people don’t waste your time on external Net Promoter® benchmarks.

I’ve also been careful to explain that not everyone agrees with my views on this topic.

One of the people who disagrees with me is Rob Markey (Bain Partner and co-author of The Ultimate Question 2.0). Clearly he knows something about the subject.

Well, recently when chatting to Rob for one of his Net Promoter System podcast interviews he took me to task on my anti-benchmarking views and, surprisingly, we came away in a violent agreement.

He outlined the specific scenarios where Net Promoter benchmarking is useful and I agreed with every one of his points.

[Read more...]

[Guest Post] 4 Insights Into Building a Better Organisational Structure With Customer Feedback


Auto_mechanic_toolsThe only reason we collect customer feedback, including Net Promoter ® is to understand how we can improve the customer experience and lift profits. Often this impacts the organisational structure, but driving change in this area can be difficult.

Beatrice Hofmeyr having identified this issue and is doing something about it. She is currently collecting practical, best practice techniques from real Australian organisations. In today’s guest post Beatrice provides some great early findings from the project.

Please welcome Beatrice Hofmeyr…

[Read more...]

The Practicalities of Giving Frontline Staff Net Promoter Targets


bigstock-Dart-in-bulls-eye-of-dartboard-16555100“If you don’t give us a 9 or 10 on the survey you receive it will mean we have failed”. On the surface it was an odd way to end my check out process at a well known hotel chain but one I suspect that many of us have experienced. It’s called score begging and it’s an indication of poorly set front-line customer satisfaction targets.

One of the critical success factors for NPS or customer feedback success is ensuring that everyone in the organisation has the score in their personal goals. But applying that idea to front line staff is difficult and if done poorly, as you can see, it drives the wrong behaviours.

To me it is clear that you must link NPS/CSAT to performance review outcomes at least as strongly as you link other hard metrics: revenue, average handle time (AHT), etc. If you don’t staff, quite rightly, deduce that NPS is nice but what you really care about is AHT and sales at all costs.

Not having a strong CSAT/NPS goal is at the heart of many issues in the customer experience. If you’ve ever been relentlessly handed off between operators in a contact centre you have felt the effects. You know they have a tight AHT goal and it’s more important to keep their personal AHT down by shuffling you to another operator than to solve your problem. [Read more...]

Customer Journey Maps Must Come Before Transactional Customer Feedback


Transactional customer feedback is a very effective way of improving business performance. With it you can diagnose problems and processes that are driving customers away and reinforce the drivers of customer loyalty in the business.

But how do you decide which transactions should be included?

What is a Customer Journey Map?

Customer Journey Maps document the interactions that customers have with your business before, during and after their relationship with you.

With so many practitioners using them, there are many different forms of customer journey maps; none of them right or wrong, just used in different ways.

[Read more...]

Writing the Perfect Customer Feedback Survey Invitation


bigstock-Vintage-Typewriter-41977552You already have a great survey invite subject line and now you need to follow that up with an email body that drives people to start the survey.

The invite doesn’t have to be long and complex, in fact it should be short and to the point but to be most effective it must include some key information.

At all times remember the goal of the invite: to persuade the respondent to provide their feedback on your organisation. That’s it. Nothing more. Don’t add words into the invite that do not directly help you achieve that goal. [Read more...]

Email Subject Lines that Drive Customer Feedback Survey Responses


bigstock-delete-button-18380024Before a customer can complete your meticulously developed customer feedback survey, they need to open the email invitation. You have precious few seconds to prevent their finger jabbing at the delete key and your subject line is your first defence. So make it a good one.

There is lots research into what drives higher general email open rates and we can leverage this to make sure that we can maximize the open rates of our client surveys. So let’s take a look.

[Read more...]

[Webinar] Using 5 Whys Root Cause Analysis on Customer Feedback


5 Whys is one of the most commonly used quality system tools. It is a simple and methodical way to identify the root cause of an issue.

When applied to Customer Feedback you can convert “interesting feedback” in to root causes and actions plans to drive improvement in your customer experience.

In this 30 minute webinar, we’ll teach you how to use this high value Customer Feedback tool:

  • Defining The 5 Whys Approach – What exactly is The 5 Whys Approach?
  • Customer Feedback Application – How to apply The 5 Whys Approach for Customer Feedback with practical examples.
  • Is it really for me? – When should you use The 5 Whys Approach?

[Read more...]

Post Call IVR Surveys: Popular But Not That Useful


After spending 15 minutes on the line to my bank the nice lady asks me if I will hold on the line after she hangs up to take a short survey. Sure what’s one more survey to someone who lives them 24/7!

After a couple of prompted button pushes to enter scores I get the chance to provide verbal feedback. I’m not the type to use circumlocution, I like to get straight to the point but I proceed with a relatively detailed account of a recent issue. I’m only half way through when I’m interrupted with a tone and being told that the recording has ended. Not a great experience.

Post call IVR surveys like this are the new in-thing with more and more organisations using the approach to gather transactional customer feedback.

The question is not whether they are possible, clearly they are, but whether they are useful. It turns out: not so much.

So let’s review how post call IVR surveys compare to email invites to web base surveys to see which is more effective.

[Read more...]

The Best Blog Posts of 2013

As we reflect on the year that was, we hope it has been a great and exciting year for each of you. I know that on our end it has definitely been a busy one but we wouldn’t have it any other way!

While 2013 is not over just yet, we thought that we’d gather together the most popular blog posts of the year. So sit back, relax and check out the posts which were the most engaging and interesting for the year.

[Read more...]