Any survey-based research is only valuable if its results are valid. Still, many brands and companies are making important decisions based on unreliable data collection. There’s one factor they are not taking into account – different types of bias in research.
Bias is to surveys what kryptonite is to Superman – a weak spot. As a survey maker looking to get accurate data, you need to make sure you’ve done everything in your power to guard against bias. Often bias can creep into your research design and affect the quality of your data analysis without you even knowing it.
The risk of bias is present in almost all components of both quantitative and qualitative research and it can find its source in both the survey creator and the respondents. I’d even go as far as to say that it might be naïve to think that any survey could be 100% bias-free. But still, there are many things you can do to reduce and minimize bias in research.
Out of all the various types of bias in research, it’s easiest to deal with those coming directly from the survey creator. With so many different types of survey questions (ranging from demographic to personality questions), writing survey questions and answer options is an art form of its own.
Still, even the best-written surveys are susceptible to bias. That being said, we’ve identified the 5 most common types of bias in research and provided some actionable tips on how you can do your best to make your surveys bias-proof.
Let’s dig in.
5 Main Types of Research Bias to Avoid in Your Research Process
1. Sampling bias
In the world of market research and surveys, sampling bias is an error related to the way the survey respondents are selected. It happens when a survey sample is not completely random. In other words, if certain types of survey takers are more or less likely to be chosen as a sample for your research, chances are high you’re dealing with sample selection bias.
Let’s say you’re doing research on commuters. And you’ve decided to conduct your survey in-person on the streets. By surveying only the people you meet walking on the streets, you’re not likely to get a representative sample of all commuters, as people who drive or ride bicycles might be excluded from your sample.
That being said, you need to make sure your survey is distributed in such a way that all types of respondents get a chance to respond to it. Even though social media may seem like the way to go, do all of your customers have profiles on social media? Even if they do, have they liked or followed your page?
Often you’ll need to use several different distribution channels and collection methods in order to ensure proper allocation of different types of respondents. To make sure you avoid or at least minimize sampling bias in your surveys, we’ve prepared a detailed marketer’s guide to types of sampling (sample size calculator included).
2. Non-response bias
Even if you take utmost precautions to minimize sampling bias, it still doesn’t mean that you’re going to avoid other types of bias in research related to the structure of your survey respondents.
Equally distributing your surveys to all the relevant respondent groups doesn’t necessarily mean that you’re going to get an equal number of responses from all of them. You may get your sampling perfectly right, but still, there’ll always be people unwilling or unable to take the survey.
In many cases, groups of people who don’t respond at all to your surveys significantly and systematically differ from those who do. In cases where such discrepancy between the respondents and non-respondents occurs, you’re in danger of facing the so-called nonresponse bias.
Even though there’s no easy or foolproof way to avoid nonresponse bias, the safest and most effective thing to do is to get your overall response rate as high as possible. You’ll always have people who will simply ignore your survey.
However, having enough potential respondents will increase the chances of enough people in each of your target groups taking part in the survey, thus reducing the risk of nonresponse bias. What can you do to get your response rate up?
While certain factors are often outside of your control, properly communicating the message behind your surveys can still go a long way when it comes to your response rate. Along with the survey, you should consider sending out a pre-notification email (letting the general population learn more about the purpose and goals of the survey), a personalized and customized invite to take the survey, and a reminder in case they haven’t taken the survey yet.
3. Response bias
Having to persuade people to take your survey is hard enough. But getting them to merely respond doesn’t mean much to you if they don’t take the survey seriously. That being said, you also need to think about how to get the respondents to provide accurate and honest responses to your survey.
Which brings us to the second one on our list of types of bias in research – response bias. Less-than-truthful survey responses can come as a result of both conscious and subconscious cognitive factors. As a consequence, there are several types of response bias in research.
Acquiescence bias (also known as the friendliness bias, confirmation bias, or “yea-saying”) is one of the most common types of bias in research.
It manifests itself when a respondent shows a tendency to agree with whatever it is that you’re asking or stating. Experience has shown that whenever a survey contains the agree/disagree type of questions, the respondents are more likely to agree than disagree. There are several reasons for that.
Namely, due to a phenomenon inherently present in human nature, some people have acquiescent personalities and are more likely to agree with statements than disagree regardless of the content. Sometimes people perceive the question asker as an expert which causes them to be more likely to have a positive reaction to the question.
Acquiescence can also be an “easy way out” as it takes less time and effort than actually taking each of the options into careful consideration. Finally, some people will simply agree just so as to complete the survey as soon as possible.
How can you avoid the acquiescence response bias? It’s quite simple – never ask questions that may imply there’s a “right” answer. Instead, you need to focus on the respondent’s point of view by asking a real question rather than just asking for agreement or disagreement with an already predefined statement.
Let’s take a look at this in practice. Here are two ways you can ask your users about their customer support experience:
In the example above, we’re making a positive statement and asking if they agree or not. By phrasing your question this way, you’re increasing the risk of your respondents being affected by the acquiescence response bias.
The question above, on the other hand, doesn’t imply that there’s a “more right” answer, therefore enabling the respondents to choose their own answer without any bias.
Demand characteristics bias
For some respondents, the mere fact that they’re being a part of a survey may affect the answers they provide. That’s because when they are involved in a survey or any kind of examination, people often try to guess the purpose behind it and behave in accordance with the pre-set expectations.
This type of bias in research is known as the demand characteristics bias. The best way to avoid it is to make your survey as engaging and interactive as possible so as to make your respondents forget the fact that they’re being a part of research and just focus on providing as truthful responses as possible.
Extreme responding bias
When given a scale-type of a questionnaire (such as Likert scale surveys), the respondents are often biased to choose only the most extreme options on the scale. This kind of bias in research is commonly referred to as the extreme responding bias.
The opposite of the extreme responding bias is when the respondents only choose neutral responses. The tendency to choose only extreme or neutral responses is in many cases culturally specific.
Social desirability bias
Desirability bias in research occurs when the respondents are motivated to answer the survey questions so as to reinforce characteristics and behaviors that are generally socially desirable and deny those that aren’t.
In other words, people often answer questions in a way they think will lead other people to like or accept them. This innate desire to present oneself in the best possible light to the large social control group can be fatal to your survey results accuracy as it often leads to different types of bias in research.
This kind of research bias could be avoided or minimized by making the survey anonymous and distancing your brand from it. That way, the respondents wouldn’t have the fear of providing an answer that might be deemed undesirable.
Interviewer bias is related to the way interviewers ask questions or respond to answers. It is characteristic for in-person interviews.
In addition to body language, facial expressions (and other non-linguistic aspects of communication), interviewer bias may arise from the respondents’ perception of the interviewer’s demographic characteristics such as age, gender, ethnicity, social class, professional background, and more. Whenever seemingly related to the interview topic, those characteristics may influence the way participants respond to interview questions.
4. Question order bias
This is one of those types of bias in research many people don’t even pay attention to or realize it could cause bias. But the fact is that the order of both questions and answers could cause your survey respondents to provide biased answers.
In some cases, the initial questions of your survey could influence the answers your respondents give to the subsequent questions later on in the survey. For example, asking questions about the overall satisfaction with the quality of customer support at LeadQuizzes after the specific satisfaction questions may lead to biased responses.
Due to the so-called assimilation effect, the answer to a latter question resembles the answers to previous questions more closely than it would if it had preceded them or if it had been asked independently.
This form of bias in research can be caused not only by the order of questions in a survey but by the order of answer options as well. Our experience has shown that in online and print surveys, respondents usually prefer some of the first few answer options, while during phone and in-person surveys (where the interviewer is reading the answers) the respondents are more likely to opt for some of the latter options.
What can you do to minimize the question order research bias? Some of the things you could do are reduce the number of scale-type questions and make your questions as engaging as possible, randomize your questions and answer options, group your questions around a common topic, and so on.
5. Information bias
Information bias can refer to any misrepresentation of truthfulness that occurs during the collection, handling, or analysis of data in a research study, survey, or an experiment. Some of the most common forms of information bias include misclassification bias, recall bias, observer bias, and reporting bias.
As information bias can often stem from measurement and calculation errors, it is also sometimes referred to as measurement bias.
Reporting bias (also known as outcome reporting bias or publication bias) originates from academic research and happens when the outcome of a research study affects the decision whether to publish the results or not. Often, authors and researches have the tendency to publish only the research that has brought upon significant results.
This type of bias can be found in marketing as well. For example, if a brand or a company runs a survey to check what their target population thinks of their new product and the results show that customer satisfaction is on a low level – they are highly unlikely to brag about it, aren’t they?
Above, we’ve identified the 5 main types of bias in research – sampling bias, nonresponse bias, response bias, question order bias, and information bias – that are most likely to find their way into your surveys and tamper with your research methodology and results.
In addition, I’ve shared several tips on how you may fight each of the biases and what you can do to prevent those from occurring in the first place. Basically, if you make sure you’re serving quality questions to the right people while being aware of the potential sources of bias, you have a much higher chance of conducting a bias-free survey and obtaining reliable results.