Most People Don’t Understand Sample Size

Equation

bigstock-young-business-man-holding-his-53002933You’ve spent weeks working through the numbers to unpick what customers are saying. After checking through the data and analysing a range of root causes, you have created a really practical plan to solve a key customer issue.

The PowerPoint presentation you’ve created nails each of the points you want to make. It starts right up front with the bad news: Net Promoter data for the business group in question.

Before you walked in, you were fully prepared for this meeting, so how is it that 3 minutes in you’re under attack on the very first slide? A senior manager is pointing accusingly at the screen: “I don’t believe that NPS – you don’t have a big enough sample size.”

This is a problem that I am increasingly hearing from customers. When their colleagues in the company don’t like the results from a Net Promoter or customer satisfaction survey they are blaming the sample size.

Don’t like the score their department has received? Blame it on sample size. Don’t like actions that are being recommended and want to drag the anchor on making change? Blame it on sample size.

More and more people are starting to hear about sample sizes and calculations and are using their half knowledge to frustrate the process. Typically, they have just enough information to be dangerous.

For anyone that really understands sample size and the way statistics operate, this is frustrating because it mischaracterises sample sizes and making decisions based on statistical data.

We use statistics to help us make decisions and the only information we really need is enough to distinguish between alternative outcomes. Any more than that is a waste of time and effort.

Here’s Why that Senior Manager is Upset

The reason that executive is so jumpy is that he or she is probably being rewarded based on customer satisfaction or NPS targets. This is a problem.

It’s not talked about very much but one of the key implications of all this sample size and confidence interval conversation is related to personal Key Performance Indicators (KPIs).

Typically, business KPIs are set as hard numbers: $x million in revenue, $y million in costs or $z seconds of average handle time. In all of these cases, hard single numbers are fine because you have access to the whole population, i.e. you can actually count every dollar of revenue.

The finance department is going to get the boot and the IRS is going to be very unhappy if you get revenue and cost wrong by even a few dollars.

On the other hand, CSAT and Net Promoter scores can only ever be estimates. When you state that the NPS for your business or department or division is 25, what you are really saying is:

The NPS of the sample we took was 25. Based on the attributes of the response we received, we can be 95% certain that the NPS of our customer base is between 21 and 28.

That is a very different statement and one that is at the real heart of the issues that we discussed in our introduction. If people are not confident in the robustness and accuracy of the customer feedback, data collection, and reporting process, they will refuse to believe the results and look for flaws in the system. Hence the large number of sample size questions.

If you are holding people to customer feedback goals then you should probably not be doing it the same way that you give revenue goals.

Rather than single numbers, you should opt for ranges, lowest estimated limits or even simple confirmed changes. For example, these would be better NPS KPIs:

  1. NPS for year 22-25
  2. NPS less the Margin of Error > 25
  3. NPS improved year on year

Note that (1) above get complex because you will also need to include minimum sample size and ratios of Promoters/Neutrals/Detractors.

In some ways, example (2) and (3) are the clearest sets of KPIs.

Getting You Back on the Front Foot

So let’s get back to basics and agree on what we are typically trying to achieve in a customer feedback context. I recently wrote about The only statistical analyses you need to use on customer feedback data and explored why you are only looking to perform two sorts of analyses on customer feedback data.

-Is the Score Different?
-Does Changing this Cause That to Change?

Most of these sample size questions are tied up in the first type of analysis. You are trying to determine if an average of a score has changed from one sample to the next.

Let’s examine what that means for sample size.

It is not generally well understood but there is no minimum sample size required for any particular set of statistical analysis. Rules of thumb such as needing a minimum of 100 responses are relevant to providing a “reasonable” level of confidence for a worst case scenario. They are not relevant to looking at a specific situation.

As an example, say you have this question in your customer survey:

How responsive is our sales staff, where 1 is unresponsive and 7 is very responsive?

One of the things you will want to know is has the score changed from the last survey. So what sample size do you need to check if a change has occurred?

Sorry, but we have to dive into some maths for a few seconds to make this point.

The equation for that sample size is:

EquationSource

Where

  • n is the minimum required sample size
  • Z is related to how confident you want to be in the answer, normally expressed as 90% or 95%
  • σ is related to how much inherent variation there is in the population: does almost everyone give you a 5 or 6, or do you get a wide even range of scores all the way from 1’s up to 7’s.
  • E is the margin of error or size of the change you want to detect

Don’t worry too much about the details of the math, but what this says is that sample size (n) is not one number but a range of numbers. It changes with changes in the assumptions you make and the attributes of your customers:

  • n gets larger the more confident you want to be that you have detected a real change
  • n gets larger the more variation you have in your sample
  • n gets larger the smaller the change you want to detect.

So if “responsiveness” has gone up a lot since the last survey you will need a relatively small sample to identify the change. Alternatively, if the change has been small, you will need a bigger sample.

(Readers worried about using ordinal data in this way please read this)

There is no one right sample size. There is only the sample size that is able to detect what you want to detect.

You do need some statistics to demonstrate your position but they don’t have to be difficult. You could start with an easy Excel spreadsheet and a t-Test.

An Extreme Case: 5 Data Points May Be Enough

Let’s goes even further with the idea that sample size is not all it’s cracked up to be.

What if you had only five data points? Could you make any useful prediction with so few data points? The executive in your meeting would probably say no, but they’d be wrong.

Douglas Hubbard discusses the Rule of Five in his book How to Measure Anything.

Rule of 5

[Source](Hubbard, Douglas W. (2010-04-07). How to Measure Anything: Finding the Value of Intangibles in Business (p. 30). Wiley. Kindle Edition.)

Note that we are talking about median not mean (or average) here so it’s a bit different but the idea is still the same. With just 5 data points you can make some pretty strong statements about the underlying population.

If you take a sample of 5 customers and they each score your “responsiveness” between 4 and 6 on a 7-point scale, you can be pretty sure that the median score of all customers will be between 4 and 6.

The point being that you might well be able to make important decisions based on that limited amount of data.

Net Promoter is a Special Case

However, when you start to consider this idea of sample size you need to be aware that Net Promoter is a special case. The sample size calculators that you will find on the internet simple don’t apply because the statistics is different for NPS.

There are a tables provides by vendors in the Net Promoter consulting business (you can Google them) but they overestimate the sample size that you need because they only consider the worst case scenario.

Based on those tables, if, for instance, you had 112 responses, you might think that the sample size was only going to detect changes of +/- 11 points of NPS.

That is the worst case but if those responses looked like this, you could be confident in detecting changes of +/-5 points – more than twice as effective.

  • Number of Promoters: 86
  • Number of Neutrals: 22
  • Number of Detractors: 4

In this case, it is the large proportion of Promoters and the small proportion of Detractors that make the sample size more efficient.

Download this free Net Promoter specific calculator for the backup stats you need for that executive: Net Promoter® Comparison Tester

Getting Past Slide 1

So now, before you present, would be a good time to back to your presentation and think through the sample size question.

You should be prepared to discuss sample size and have the statistics to back up your position.

But, you should also have some sympathy for the senior manager and his or her not-so-perfect customer satisfaction goal.

Share viaShare on Google+Share on LinkedInTweet about this on TwitterShare on Facebook
BusinessLeadersGuideCTA

Surprise: Rob Markey and I agree on Net Promoter® Benchmarking

delight-the-customer

delight-the-customerAnyone that has been reading this blog for more than a couple of weeks knows that the subject of Net Promoter benchmarking gets me fired up.

In talking to clients and prospects the question of “what’s a good Net Promoter score” almost invariably arises. Many times I have had to choose my words carefully when I tell people don’t waste your time on external Net Promoter® benchmarks.

I’ve also been careful to explain that not everyone agrees with my views on this topic.

One of the people who disagrees with me is Rob Markey (Bain Partner and co-author of The Ultimate Question 2.0). Clearly he knows something about the subject.

Well, recently when chatting to Rob for one of his Net Promoter System podcast interviews he took me to task on my anti-benchmarking views and, surprisingly, we came away in a violent agreement.

He outlined the specific scenarios where Net Promoter benchmarking is useful and I agreed with every one of his points.

[Read more...]

[Guest Post] 4 Insights Into Building a Better Organisational Structure With Customer Feedback

Auto_mechanic_tools

Auto_mechanic_toolsThe only reason we collect customer feedback, including Net Promoter ® is to understand how we can improve the customer experience and lift profits. Often this impacts the organisational structure, but driving change in this area can be difficult.

Beatrice Hofmeyr having identified this issue and is doing something about it. She is currently collecting practical, best practice techniques from real Australian organisations. In today’s guest post Beatrice provides some great early findings from the project.

Please welcome Beatrice Hofmeyr…

[Read more...]

The Practicalities of Giving Frontline Staff Net Promoter Targets

linking-nps-sale

bigstock-Dart-in-bulls-eye-of-dartboard-16555100“If you don’t give us a 9 or 10 on the survey you receive it will mean we have failed”. On the surface it was an odd way to end my check out process at a well known hotel chain but one I suspect that many of us have experienced. It’s called score begging and it’s an indication of poorly set front-line customer satisfaction targets.

One of the critical success factors for NPS or customer feedback success is ensuring that everyone in the organisation has the score in their personal goals. But applying that idea to front line staff is difficult and if done poorly, as you can see, it drives the wrong behaviours.

To me it is clear that you must link NPS/CSAT to performance review outcomes at least as strongly as you link other hard metrics: revenue, average handle time (AHT), etc. If you don’t staff, quite rightly, deduce that NPS is nice but what you really care about is AHT and sales at all costs.

Not having a strong CSAT/NPS goal is at the heart of many issues in the customer experience. If you’ve ever been relentlessly handed off between operators in a contact centre you have felt the effects. You know they have a tight AHT goal and it’s more important to keep their personal AHT down by shuffling you to another operator than to solve your problem. [Read more...]

[Guest Post] Can Technology Delight Customers? Yes and No.

Bob Thompson

bigstock-Social-media-on-smartphone-60749657Like it or not, the world is going digital. Websites were just the beginning. Now we have social media, smartphones and so on. The only conclusion I can reach is that more of our lives will be spent interacting with technology of one kind or another.

The conundrum: People are more likely to create memorable experiences, which is essential to drive genuine loyalty. In CustomerThink’s research, we find that friendly, well-trained, and knowledgeable employees are key factors what customers perceive to be excellent experiences. Only humans can express empathy and solve problems creatively, so it’s not surprising that interactions with people are more memorable.

New technology can be a positive surprise — I still think it’s amazing that I can buy and receive a Kindle e-book from Amazon in less than one minute. But as Jon Picoult of Watermark Consulting notes: “When technology helps fuel a delightful customer experience, it usually doesn’t take much for other firms to copy that technology.”

More typically, automation is used to make customer interactions more efficient, consistent, and less memorable. So if you’re not careful, you can automate your way to a faceless company with no personality whatsoever!

The solution is to understand the three important roles that technology can play in delightful customer experiences. Let’s use the analogy of a great movie, where the audience experience is delivered by the star, supporting actors, and a “key grip” to manage lighting and camera movements. Obviously only the actors are seen on screen, but without great support behind the camera, a film won’t be delightful to watch. [Read more...]

You Unconsciously Resent Your Customers

bigstock-Close-up-portrait-of-a-serious-54544025

bigstock-Close-up-portrait-of-a-serious-54544025Recently, while pondering the surge in discussion about customer centricity and customer focus over the last 10 years, I wondered: do employees in most organisations subtly resent their customers?

It has always struck me as just a little odd that commercial enterprises must explicitly say that they are customer focused and create programs and approaches to instil employees with a zeal for serving customers.

After all the only function of a commercial enterprise is to serve customers. It has literally no other reason for existing.

Perhaps I’m being naïve but stating that we are “customer focused” or “we put the customer first” seem just a little absurd in this context. If you are not putting the customer first then who is first?

That truly innovative product that R&D has just developed is worthless without customers to buy it. No matter how great your supply chain management process runs, if it’s not supplying customers it’s just a bunch of idle trucks and conveyer belts.

So why are organisations not more naturally customer focused? Many big (and small) corporations seem to spend most of their time ignoring customers and even actively doing the reverse of what customers really want when it’s customers that they are there to serve.

[Read more...]

[Guest Post] 6 Ways to Improve Your Customer Testimonials Today

6 Ways to Improve Your Testimonials Today Image 3

Today we welcome a guest post by Sam Johnson. Sam is the Founder at Feedback Loop – An Australian Software Company that helps businesses get the best testimonials on their website.

Extending your customer feedback process to also capture testimonials is a relatively simple task. You already know which customers are advocates of your brand from the feedback. Now all you need to do is ask them for a testimonial.

I’ll let Sam take it from here on some great ways to improve your customer testimonials.

Customer testimonials are one of the most powerful forms of marketing yet there are a few common mistakes that you can fix to quickly increase sales on your website.

[Read more...]

The Perfect Net Promoter® Survey Design

Blueprints and drawing tools concept for construction or development

Blueprints and drawing tools concept for construction or development“The one number you need to grow”: who’d have thought that such a simple statement would cause such a fuss.

Some people actually believe that you only need one question in your Net Promoter Survey. But of course that doesn’t work: so let’s review the anatomy of a perfect Net Promoter Survey.

Your Survey Design Goals

When you design anything you need to have goals to focus you on what you are trying to achieve. A Formula 1 car and an SUV are both cars but they have very different design goals.

So it is with surveys. Get your design goals wrong and your Formula 1 survey could seat eight and have a maximum speed of 80 mph.

A lack of design goals is why so many surveys seem to have a split personality. They ask a few market research questions then a few customer feedback questions and even toss in a couple of marketing pushes.

So what should the design goals be for your Net Promoter Survey?

[Read more...]

10 Customer Experience Influencers You Really Should Follow

bigstock-Twitter-icon-on-smart-phone-57145373

bigstock-Twitter-icon-on-smart-phone-57145373There is no shortage of data on the interwebs but much of it is not worth reading. The trick is to find the information written by the people that add value and challenge your thinking. Then listen to what they say and hone your ideas against them.

These are some of the people I follow and find constructive. It is by no means an exhaustive list and I certainly follow others.

I don’t agree with everything they write, but when the people on this list have something to say I think it’s worthwhile listening.

As I look down the list I notice that while there is a good amount of traditional service based writers here, there is also a lot of systems thinking and process focus.
I think that mixing the two together is a very strong platform on which to build your customer experience success.

Here’s my list in alphabetical order:

[Read more...]

How to Demonstrate the ROI of your Customer Feedback Program

ROI (Return On Investment) highlighted in green with felt tip pen

ROI (Return On Investment) highlighted in green with felt tip penSometimes Voice of the Customer (VoC) practitioners frustrate me. In this recent research, customer feedback professionals identified that their biggest barriers to a more effective customer feedback process were engaging staff and management and understanding loyalty drivers.

But when asked what they would do with an unlimited budget the most common response, by far, was “collect more data”. Unbelievably, coming in 6th, and last, place was “make more changes to the business.”

These are the same people who complain about management not caring about or investing in VoC. The issue is that they themselves are the ones perpetuating the myth that customer feedback is nothing but data collection. They talk about increased sample sizes instead of the business value that informed change is bringing to the organisation.

[Read more...]