Biased Questions: How to Identify & Fix Them in Surveys

Biased Questions: How to Identify & Fix Them in Surveys

Download the Customer Feedback whitepaper

Designing a customer feedback survey in way that provides and concise data you can use in you business is not always simple.

Using Biased Questions is a common mistake among survey designers and this post outlines what they are, how to identify them and how to fix them.

Download the Customer Feedback whitepaper

What are Biased questions

Biased questions are ones that, intentionally or unintentionally, attempt to influence the way the respondent will answer. Take this example:

example of biased question
Example of an Intentionally Biased Survey Question

By prefacing the question (“Overall how satisfied…”) with the pejorative statement “… is continually striving to achieve 5-STAR customer satisfaction…” the company has subtly altered the respondents’ feelings towards their experience.

Question bias can also be caused through the options the respondent is offered.

Here’s a perfectly balanced question turned into a biased question through the use of limited response options:

Balanced Survey Question Turned into Biased Question Through Response Options

Of course no company wanting to really understand customer loyalty would write the question like this but it demonstrates what can happen.

Why Biased Questions are a Problem in Surveys

The main reasons such questions are a problem in surveys is they distort the data collected.

Most of the time surveys and questionnaires are trying to accurately understand a situation or perception of a customer or respondent. By using biased questions the organisation writing the survey is distorting the data collected making it less reliable and accurate.

Of course, when done intentionally that is the intended effect.

Interestingly not only can the data be distorted it can also be distorted in unknown ways.

Take the example question above.

example of biased question

Typically this style of distortion is done by organisation trying to maximise some feedback score but it’s a mistake because the respondent could go either way:

Reduce the score they were going to give: Some people will read the question and their expectations will rise, depressing the score they give: “well they were going for 5-star and that was only 3 star.”

Increase the score they were going to give: other people will subtly increase their score because they don’t want to disappoint the company by marking them down.

The problem is that you will never really know the effect.

Why Do Biased Questions Occur?

There are two main reasons biased questions turn up in surveys:

Unintentionally

Often it can slip into survey questions unintentionally, especially where the question writer has a strong affiliation with the organisation or subject of the survey and want it to be seen in a positive light, or not in a negative way.

Intentionally

However, just as often questions are written in a biased manner to attempt to alter the information collected so the results of the survey analysis support one or other of the writer’s agendas.

What are not Biased Questions

There are plenty of poor ways to word survey questions that are not necessarily biased including the following types:

Just because a question is not well written does not mean it is biased.

Biased Question Examples

Here are a few more biased question examples so you can get the feel for them.

Do you think there’s a danger in giving young people guns and teaching them how to kill? 
Example Biased Question

This question comes from the famous BBC comedy “Yes, Minister”. There are several pejorative words in this question: kill, danger, guns. Which biases people to answer in the affirmative – which was the intention.

What did you dislike about the service you received?
Example Biased Question

Unless this question occurs AFTER someone has already expressed dislike of the service, e.g. they have answered the Net Promoter question as a detractor, you’re inviting them to find fault where there might have been none.

Will you be voting for this disrespectable political party at the next election?
Example Biased Question

Adding “disrespectable “ to the question clearly biases the response.  

How to Spot a Biased Question

Unlike Double Barrelled Questions, which are easy to spot, it can sometimes be difficult to spot a biased question in a survey because the form bias can occur in so many different ways. 

In general if you see emotive or pejorative words of any type or limited response options in a survey question, there is a chance you have created a biased question.

Even if you only suspect the question to be a problem you should attempt to correct it.

How to fix Biased Questions

Fixing biased survey questions can sometimes be a little tricky.

The best approach is to simply re-write the question in a way that is as neutral as possible and eliminate emotional, contentious and pejorative wording.

For example to fix this biased question:

What did you dislike about the service you received?

You could rephrase it as:

Tell us about the service you received.
Corrected Biased Question

Biased Question FAQ

What are Leading Questions

Leading questions are a type of biased question intended to intentionally influences the respondent. For example this famous question from the BBC comedy “Yes, Minister”. 

“Do you think there’s a danger in giving young people guns and teaching them how to kill? “

Download the Customer Feedback whitepaper