How to make sure your survey respondents are really paying attention? And not just clicking on random answers for the sake of completing the survey as quickly as possible.
There’s a neat trick experienced survey makers use to ensure better validity of their surveys – attention check questions.
In this post, I’m going to teach you more about how you can perform attention checks in your surveys and whether you should do it.
Let’s dig in.
What are the attention check questions?
Attention checks are a simple way to find out who does or doesn’t pay attention to your survey questions. Such questions are designed to make sure that survey takers’ attention is at a high level throughout the entire survey – or else, they’re disqualified.
In most cases, attention questions seem almost deceptively easy to answer. But there’s a catch – usually there’s an instruction somewhere in the middle of the question copy that tells the taker to take a particular action or give a particular answer.
To make it more clear, I’ve used our survey maker to create this example of an attention check question:
If you’re just skimming the questions, your impulse would be to answer “green”, right? Still, the question seems deceptively easy, doesn’t it?
That’s because the survey makers often “hide” instructions inside such questions telling you what to answer. The easiest question ever – IF you read it in entirety and with attention.
In the example above, the attention check question itself contains the answer – “select purple as an answer so that we know you are paying attention”. So, if you don’t answer “purple”, you’re disqualified as an inattentive survey taker.
3 Types of Attention Checks in Surveys
1. Attention filters
An attention filter is usually found in large blocks of text. For example, job posters on Upwork often use attention filters to make sure job candidates have read the entire job description and know exactly what is expected from them if they get the job. (It’s not uncommon to find job posts that say something like “to apply for this job, start your cover letter with the word ‘pineapple’”, for example.)
An attention check question example I shared in the previous section might be an example of an attention filter. In surveys, you will often find directions telling you to do the thing that does not correspond to what you would normally reply to the question if you weren’t given the additional instructions.
2. Trap questions
Unlike attention filters that are designed to “catch” respondents who are not paying complete attention to the answers they provide, trap questions are obviously intended to catch respondents who are speeding or not reading the questions at all!
They are best used with surveys that contain a large number of questions or scale-based statements. Such attention check questions are usually inserted into the survey flow asking for a specific answer.
Even though it’s pretty obvious that it’s a trap question, it might be difficult to spot for those respondents who’re just speeding through the survey. If they’re lucky, they might get it right, so in large surveys, you can consider adding several trap questions.
3. Reverse wording
The reverse wording attention check changes the direction of the scale. So, when creating a survey, you should consider asking the same question twice – once using a positive voice and again using a negative voice.
When analyzing the results, you should compare the two answers and once you reverse the answer to the negative question, you should have two same answers. If not, chances are high your respondent wasn’t paying attention.
If you’re using Likert scale questions in your survey, you could consider reverting the answer options when asking one and the same question twice. For example, instead of starting with positive polarities and moving towards the negative ones, consider changing the direction.
Can attention check questions harm the validity of survey results?
There’s an entire science around surveys. In order to get valid research results, you need to take into account different methodologies, question types and topics, and all kinds of bias in research.
Attention checks were originally introduced to prevent speeding through the survey or giving incomplete and low-effort answers, skipping necessary questions, not fully responding to open-ended questions, and other forms of undesirable survey-taking behavior.
In 2009, a group of researchers has acknowledged a growing trend aimed at identifying those survey takers that are not paying full attention to survey instructions and questions (Oppenheimer, Meyvis, and Davidenko 2009).
The original findings claimed that filtering the respondents who didn’t pay attention could increase the efficiency of experiments. So, many researchers started using attention check questions to eliminate respondents who don’t follow instructions.
But even though attention checks were originally introduced to prevent invalid survey results, other studies have shown that inserting them into a survey can be a threat to the survey validity itself.
Another group of researchers started advising against eliminating the respondents who didn’t fully follow the instructions and failed some kind of an attention check (Anduiza and Galais 2016; Berinsky, Margolis, and Sances 2014; 2016; Hauser et al. n.d.; Miller and Officer 2009).
Instead, they suggested that such respondents’ results be kept and used as one of the quality metrics when analyzing the final survey results after collecting all the data.
Another issue with simply “eliminating” the takers who fail the attention check is that it might lead to a demographic bias. What exactly does that mean?
When preparing any kind of serious research, researchers need to make sure that all the demographic and psychographic groups are equally represented among the survey takers.
But the problem with attention checks is that they do not eliminate an equal number of members of each subset – eliminating random respondents on the basis that they have failed the check (and not replacing them with another respondent from the same demographic and psychographic group) is likely to lead to disproportion among the survey sample and put a question mark over the validity of survey results.
Other studies have found that attention check questions can lead to all kinds of biases, including socially desirable responding, where respondents edit, adjust or censor their answers to fit what’s socially acceptable and expected because the attention checks create a sense of being watched (Clifford and Jerit 2015).
Based on the findings above, there are three possible solutions:
- don’t use attention check questions at all
- replace the “eliminated” respondents with respondents from the corresponding demographic and psychographic group (not possible with anonymous surveys)
- use fair attention checks to ensure people ARE paying attention and not just for the sake of eliminating those who aren’t
What is a fair attention check?
The purpose of attention check question should be to make sure people actually pay attention when completing your survey and not to eliminate as many people as possible, right?
Also, an attention check should determine whether a respondent has carefully read the question itself, rather than instructions above/below/around it. That’s why you need to make sure you use fair attention checks. And use them only when it is vital for the valid completion of a task.
But what exactly makes attention check fair?
Let’s take a look at the two examples of attention check questions below.
And the second one, which could be considered a “fairer” attention check question.
The first example (“What is your favorite drink?”) is not a fair attention check question because respondents shouldn’t be obliged to pay close attention to long instructions or descriptions above the question to provide a valid answer.
Now that you know how (not) to check whether respondents are paying close attention to your survey questions, you’re ready to start creating your own surveys!
