How to make sure your survey respondents are really paying attention? And not just clicking on random answers for the sake of completing the survey as quickly as possible.
There’s a neat trick experienced survey makers use to ensure better validity of their surveys – attention check questions.
In this post, I’m going to teach you more about how you can perform attention checks in your surveys and whether you should do it.
Let’s dig in.
What are the attention check questions?
Attention checks are a simple way to find out who does or doesn’t pay attention to your survey questions. Such questions are designed to make sure that survey takers’ attention is at a high level throughout the entire survey – or else, they’re disqualified.
In most cases, attention questions seem almost deceptively easy to answer. But there’s a catch – usually there’s an instruction somewhere in the middle of the question copy that tells the taker to take a particular action or give a particular answer.
To make it more clear, I’ve used our survey maker to create this example of an attention check question:
If you’re just skimming the questions, your impulse would be to answer “green”, right? Still, the question seems deceptively easy, doesn’t it?
That’s because the survey makers often “hide” instructions inside such questions telling you what to answer. The easiest question ever – IF you read it in entirety and with attention.
In the example above, the attention check question itself contains the answer – “select purple as an answer so that we know you are paying attention”. So, if you don’t answer “purple”, you’re disqualified as an inattentive survey taker.
3 Types of Attention Checks in Surveys
1. Attention filters
An attention filter is usually found in large blocks of text. For example, job posters on Upwork often use attention filters to make sure job candidates have read the entire job description and know exactly what is expected from them if they get the job. (It’s not uncommon to find job posts that say something like “to apply for this job, start your cover letter with the word ‘pineapple’”, for example.)
An attention check question example I shared in the previous section might be an example of an attention filter. In surveys, you will often find directions telling you to do the thing that does not correspond to what you would normally reply to the question if you weren’t given the additional instructions.
2. Trap questions
Unlike attention filters that “catch” respondents who fail to pay complete attention to their answers, trap questions explicitly target those who speed through or skip reading the questions entirely!
Researchers find these most effective in surveys with numerous questions or scale-based statements. You can insert these attention-check questions into the survey flow by requesting a specific answer.
Even though trap questions seem obvious, respondents who rush through the survey often struggle to spot them. Since lucky guessers might answer correctly, consider adding several trap questions to your larger surveys to ensure data quality.
3. Reverse wording
The reverse wording attention check changes the direction of the scale. So, when creating a survey, you should consider asking the same question twice – once using a positive voice and again using a negative voice.
When analyzing the results, you should compare the two answers and once you reverse the answer to the negative question, you should have two same answers. If not, chances are high your respondent wasn’t paying attention.
If you’re using Likert scale questions in your survey, you could consider reverting the answer options when asking one and the same question twice. For example, instead of starting with positive polarities and moving towards the negative ones, consider changing the direction.
Can attention check questions harm the validity of survey results?
There’s an entire science around surveys. In order to get valid research results, you need to take into account different methodologies, question types and topics, and all kinds of bias in research.
Researchers originally introduced attention checks to prevent respondents from speeding through surveys, giving low-effort answers, or skipping necessary questions. These tools also target incomplete responses to open-ended questions and other forms of undesirable survey-taking behavior.
In 2009, a group of researchers identified a growing trend: identifying survey takers who ignore instructions and questions (Oppenheimer, Meyvis, and Davidenko 2009).
The Pro-Choice Argument: Why Researchers Value Attention Checks
The original findings claimed that filtering out inattentive respondents could increase the efficiency of experiments. Consequently, many researchers began using attention check questions to eliminate participants who failed to follow instructions.
However, while these checks aimed to prevent invalid survey results, other studies show that inserting them can actually threaten the survey’s validity.
Another group of researchers now advises against eliminating respondents who fail an attention check (Anduiza and Galais 2016; Berinsky, Margolis, and Sances 2014; 2016; Hauser et al. n.d.; Miller and Officer 2009).
Instead, they suggest keeping these results and using them as a quality metric during the final data analysis.
The Hidden Risks: How Trap Questions Can Damage Your Data
Simply “eliminating” those who fail the check also creates another issue: demographic bias. What exactly does that mean?
When preparing serious research, researchers must ensure that all demographic and psychographic groups have equal representation among survey takers.
The problem lies in the fact that attention checks do not eliminate an equal number of members from each subset. Removing random respondents because they failed a check — without replacing them with someone from the same group — likely creates a disproportionate survey sample. This imbalance casts doubt on the validity of the entire study.
Other studies have found that attention check questions can trigger various biases, including socially desirable responding. In these cases, respondents edit or censor their answers to fit social expectations because the checks create a sense of being watched (Clifford and Jerit 2015).
Based on the findings above, there are three possible solutions:
- don’t use attention check questions at all
- replace the “eliminated” respondents with respondents from the corresponding demographic and psychographic group (not possible with anonymous surveys)
- use fair attention checks to ensure people ARE paying attention and not just for the sake of eliminating those who aren’t
What is a fair attention check?
The purpose of attention check question should be to make sure people actually pay attention when completing your survey and not to eliminate as many people as possible, right?
Also, an attention check should determine whether a respondent has carefully read the question itself, rather than instructions above/below/around it. That’s why you need to make sure you use fair attention checks. And use them only when it is vital for the valid completion of a task.
But what exactly makes attention check fair?
Let’s take a look at the two examples of attention check questions below.
And the second one, which could be considered a “fairer” attention check question.
The first example (“What is your favorite drink?”) is not a fair attention check question because respondents shouldn’t be obliged to pay close attention to long instructions or descriptions above the question to provide a valid answer.
Now that you know how (not) to check whether respondents are paying close attention to your survey questions, you’re ready to start creating your own surveys!