Year: 2019
Type: article
Abstract: Researchers across academic disciplines have applied Instructional Manipulation Checks (IMCs) to detect inattentive survey participants (Anduiza & Galais, 2017; Motta, Callaghan, & Smith, 2016; Oppenheimer, Meyvis, & Davidenko, 2009; Paas & Morren, 2018; Peer, Brandimarte, Samat, & Acquisti, 2017; Persson & Solevid, 2013; Van Vaerenbergh & Thomas, 2013). IMCs typically instruct participants to endorse a specific category, such as “strongly agree,” or to skip answering the question altogether. Inattention is detected when the respondent fails to follow these instructions. IMC-failers have been found to have different values on marketing variables such as attitudes toward advertisements and brands (Paas & Morren, 2018). IMC-failers can also reduce the magnitude of negative correlations between oppositely scored scale-items (Paas, Dolnicar, & Karlsson, 2018), their responses can reduce test power (Berinsky, Margolis, & Sances, 2014; Oppenheimer et al., 2009) and their response patterns may obscure analyses and manipulations (Maniaci & Rogge, 2014). Their inattention therefore has important implications for survey researchers. Currently, consensus is lacking regarding the measurement of respondent inattention. As well as short Likert-type IMCs that are embedded among other items in the same format, IMCs may consist of longer texts that include instructions not to answer the question or to endorse a specific response option (Liu & Wronski, 2018; Oppenheimer et al., 2009; Peer et al., 2017). Anduiza and Galais (2017) proposed that passing longer IMCs requires more attention by respondents due to task difficulty. In addition, various other inattention indicators have been proposed, such as speeding and straight-lining (Kaminska, McCutcheon, & Billiet, 2010; Meade & Craig, 2012; Paas & Morren, 2018; Zhang & Conrad, 2014). Authors of previously published papers have suggested combining alternative indicators for detecting inattentive respondents (Kaminska et al, 2010; Meade & Craig, 2012); however, weak correlations between these indicators suggest that various forms of respondent inattention may occur (Meade & Craig, 2012). Our research aims to combine IMCs and other inattention indicators, such as speeding and straight-lining, in order to distinguish between attentive respondents and alternative groups of inattentive respondents. In addition, we assess the differences between attentive and inattentive respondents, and between different groups of inattentive respondents, in terms of the resulting response patterns. Our research complements existing knowledge about respondents’ inattentiveness and leads to recommendations that researchers can use to handle different types of inattentive respondents. In our empirical study, the 1,581 participants were exposed to both a short and a long IMC (condition 1), a short IMC only (condition 2), a long IMC only (condition 3), or were in a control group without an IMC (condition 4). Before reporting the study, we present a literature review and hypotheses. At times inattentive respondents accidentally select inappropriate answer categories, skip or misread questions, or fail to follow survey instructions. Under such circumstances random error may occur. When participants perform this inattentive behavior throughout the survey, their systematic tendency to respond based on factors other than item content affects accurate measurement (Paulhus, 1991). Attentive participants are assumed to (a) aim to understand the question, (b) retrieve information from their memory, (c) form a judgment, (d) map the judgment to the response scale, and (e) edit their answer (Sudman, Bradburn, & Schwarz, 1996). Conversely, inattentive respondents follow satisficing strategies, reducing their cognitive burden by skipping or superficially completing (some of) these stages (Krosnick, 1991). Weak satisficing implies that they follow some or all of the five stages less thoroughly, whereas strong satisficers skip multiple stages altogether (Krosnick, 1991; Galesic, Tourangeau, Couper, & Conrad 2008). Holbrook, Green, and Krosnick (2003) define weak satisficing as a minor effort cutback, while strong satisficing implies an absence of memory search and information integration. In an eye-tracking study, Galesic, Tourangeau, Couper, and Conrad (2008) found that satisficing respondents either skim quickly through later answer options depicted vertically on a screen, that is, weak satisficing, or skip later response options, that is, strong satisficing. These findings provide empirical support for the distinction between weak and strong satisficing. Satisficing has been measured in various ways, but previous research has not distinguished between indicators for detecting weak versus strong satisficing. For example, Kaminska et al. (2010) used the occurrence of “don’t know” answers, straight-lining, inconsistent answers, and extreme and middle responses as indicators of satisficing. They reported two groups of responders: optimizers scored low and satisficers scored higher on these indicators. Other researchers have found that respondents failing IMCs are likely to straight-line, speed, and display lower response consistencies (Huang, Curran, Keeney, Poposki, & DeShon 2012; Meade & Craig, 2012). However, such relationships between IMC-failure and other inattention indicators are not particularly strong, possibly as a result of the occurrence of various forms of satisficing (Meade & Craig, 2012). Another complication is that different researchers have used different IMCs. The short IMCs in Table 1 provide single-sentence instructions to select a specific category, or request respondents to skip the question (Anduiza & Galais, 2017; Liu & Wronski, 2018), possibly by clicking somewhere else on a page (Oppenheimer et al., 2009). Other short IMCs present statements that have an obviously correct answer (Peer et al., 2017). The long IMCs present multiple-lined texts followed by an instruction to not select an answer-category but instead to click somewhere else (Anduiza & Galais, 2017; Oppenheimer et al., 2009). Other long IMCs instruct respondents to select a category and add a number to this category (Peer et al., 2017) or to select multiple categories (Liu & Wronski, 2018). We adopted the terms “short” and “long” IMCs from Anduiza and Galais (2017) who found that the number of words strongly determines IMC-failure rates, more so than respondents’ characteristics, survey length, or the place of the IMC in the survey. In all the papers that are included in Table 1, higher failure rates were reported on the longer IMCs. Note that Table 1 only includes research in which both short and long IMCs were assessed. An Overview of IMC Studies That Compared Short With Long IMC Note. IMC = Instructional Manipulation Check. These percentages are obtained using the data made available by Peer et al. (2017) at https://osf.io/murdt/. Note that this Table only includes papers reporting empirical studies that compare the failure rates on both short and long IMCs. An Overview of IMC Studies That Compared Short With Long IMC Note. IMC = Instructional Manipulation Check. These percentages are obtained using the data made available by Peer et al. (2017) at https://osf.io/murdt/. Note that this Table only includes papers reporting empirical studies that compare the failure rates on both short and long IMCs. Based on the preceding discussion, we expect that strong satisficers will perform poorly on many indicators of inattention: they will often straight-line, speed, and fail both short and long IMCs. We conjecture that weak satisficers are likely to pass easy tasks, such as short IMCs, but will probably fail tasks that require more effort, such as long IMCs (Anduiza & Galais, 2017; Holbrook et al., 2003). Weak satisficers may possibly speed through the survey, because they only superficially fulfil some stages of the response process, but they do not straight-line, which would imply no effort whatsoever. The proposed distinction between weak and strong satisficers implies that different antecedents are likely to be relevant for the two groups. Moreover, the consequences of weak satisficing in terms of response patterns are probably smaller. Based on the discussion in this section of the article, we hypothesize: Hypothesis 1: Among inattentive respondents, weak satisficers can be distinguished from strong satisficers Hypothesis 2.1: Compared to non-satisficers, weak satisficers fail the long IMC and speed to a higher extent Hypothesis 2.2: Compared to non-satisficers, strong satisficers fail both IMCs, speed and straight-line to a higher extent Hypothesis 3.1: Weak versus strong satisficing occur as a result of different causes Hypothesis 3.2: Weak versus strong satisficing result in different consequences The LISSpanel, a household panel that is operated by CenterData (Tilburg University), is based on a representative register-drawn sample of the population in The Netherlands who are 16 years and older. Its lack of bias is maintained by sampling people using a stratified design based on the differential drop-out rates of subpopulations. We requested the LISSpanel organizers to randomly draw from this panel, so that every panel-member has an equal selection probability. Hence, the obtained sample does not require weighting to ensure representativeness. Panel members receive a financial incentive for their participation. Our study was conducted in July 2014 among 1,581 LISSpanel members. After an introduction on a separate screen, the survey contained five screens each consisting of eight Likert questions about alexithymia, that is, the inability to describe and identify emotions in the self. The answer options were labelled totally applicable (1) to not at all applicable (5). On the next 12 screens, 43 items on health attitudes, exercise, and survey attitudes were presented, so that there were 18 screens in total, including the introduction. The short IMC, which was embedded on the third screen, requested respondents not to answer a Likert-format question. This IMC has been included in the Supplementary Appendix and is a procedure akin to the short IMCs used by, for example, Anduiza and Galais (2017) and Liu and Wronski (2018). The long IMC, on the 10th screen, instructed respondents to click on the title, “sport activities,” and to ignore the question underneath a 10-line text, a procedure that is akin to the long IMC of Oppenheimer et al. (2009), and is also described in the Supplementary Appendix. In the study’s two-by-two design, 221 respondents were exposed to both IMC-versions (condition 1), 224 only to the short IMC (condition 2), 247 respondents only to the long IMC (condition 3), and the remaining 889 respondents completed the standard questionnaire without IMCs (condition 4; control condition). Based on the G*Power 3.1.7 calculation (Faul, Erdfelder, Buchner, & Lang, 2007), sample sizes of at least 200 per condition ensure a power of at least 80% to detect a 0.1 difference in IMC-failure rates. The nonresponse rate was 24.3%, and no item nonresponse was possible in the programmed online questionnaire. Note that the data resulted from a larger study of IMCs, in which the remaining four conditions were not relevant to the current paper. The survey data provided the values of respondents on the binary speeding and straight-lining indicators. Respondents dedicating less than 0.3 seconds on average to each survey-word were coded as speeders (Paas et al., 2018; Paas & Morren, 2018; Zhang & Conrad, 2014). Straight-lining respondents were detected using the Longstring variable that was developed by Meade and Craig (2012). Longstring values are calculated by determining the maximum number of times a respondent consecutively selects the same response category on the specified survey-screen. Next, Longstring values were averaged across the five screens with the 40 alexithymia items, which all had the same Likert-type format. On the resulting Avg-Longstring variable, scores of at least four were considered high, based on the elbow criterion suggested by Johnson (2005) and on analysis of boxplots. Dividing four by the number of Alexithymia questions per screen, that is, eight, yields a proportion of 50% as the binary cut-off for detecting straight-lining respondents. We first report model-free descriptive statistics. As in previous studies, failure occurred more often on the long IMC (65.2%) than on the short IMC, (22.6%), p < 0.01, in the within-subjects design of condition 1. A between-subjects test supports that the failure rate on the short IMC, 17.4% (condition 2), was lower than on the long IMC, 69.2% (condition 3), p < 0.01. Table 2 reports that in condition 1, the correlation between failing the short and long IMC was insignificant. Furthermore, failing the short IMC related most strongly to straight-lining, while failing the longer IMC related to both straight-lining and speeding. Men did not differ from women on any of the satisficing variables. In condition 2, highly educated people were relatively unlikely to fail the short IMC in condition 2, but this was not the case in condition 1. In all conditions, younger people are more likely to speed. Relationships Among Satisficing Variables and Socio-demographics Note. Table 2 presents phi-values, except for age, with * indicating significance at the 0.05 level and ** at the 0.01 level. The long and short IMCs, respectively, are not presented for the tests on conditions 2 and 3 as a result of the between-subjects design. Relationships between the variable age and other variables in Table 2 concern mean differences in years between the two levels of the binary IMCs, speeding and straight-lining. IMC = Instructional Manipulation Check. Relationships Among Satisficing Variables and Socio-demographics Note. Table 2 presents phi-values, except for age, with * indicating significance at the 0.05 level and ** at the 0.01 level. The long and short IMCs, respectively, are not presented for the tests on conditions 2 and 3 as a result of the between-subjects design. Relationships between the variable age and other variables in Table 2 concern mean differences in years between the two levels of the binary IMCs, speeding and straight-lining. IMC = Instructional Manipulation Check. We applied Latent Class Analysis (LCA) (Vermunt & Paas, 2018), using LatentGold (Vermunt & Magidson, 2013), to the condition 1 data, which included both IMC-versions. In general, LCA results in segments of respondents who displayed similarities on a set of pre-selected variables, which are called segmentation indicators. The resulting segments can then be labeled based on specific traits or behaviors. In our application of LCA, we aimed to segment the respondents according to satisficing behavior, and we included as the segmentation indicators: speeding, straight-lining and failure of the short and long IMC. The demographics, age (in years), gender (f/m), and tertiary education (yes/no) were included as covariates for explaining segment membership. In the LCA, the minimum AIC criterion shows that a three- or four-segment solution is appropriate, that is, for one segment the AIC is 838, for two segments, 795, for three and four segments, 792 and for five segments, 796. The minimum BIC criterion suggests that a two-segment solution is appropriate, with a value of 835. For the one-segment solution BIC is 852, for three segments, 858, for four segments, 887, and for five segments 918. Thus, the information criteria point to a two- to four-segment solution. Information criteria are often inconsistent (Vrieze, 2012) and therefore face validity is an important consideration (Vermunt & Paas, 2017). We find that in terms of face validity the three-segment solution outperformed other solutions. Table 3 shows that the individuals in segment 1, “attentive respondents,” neither speeded nor straight-lined (both 0.00%). They had a 17.82% failure rate on the short IMC and 56.36% failed the long IMC. Segment 2, “weak satisficers,” usually failed the long IMC (93.35%), had a relatively low failure rate on the short IMC (20.01%), often speeded (89.09%), but hardly ever straight-lined (2.89%). Segment 3, “strong satisficers,” had relatively high failure rates on both the short (71.61%) and long IMC (92.71%) and often straight-lined (64.98%). They were less likely to speed (47.99%) than weak satisficers, but more likely to speed than attentive respondents. LCA Results Note. Table 3 reports the statistics that are based on proportional classification, i.e. respondents are as to all by the When we use the LCA, we use the classification, i.e. every respondent is to the with the (Vermunt & Magidson, 2013). IMC = Instructional Manipulation Check. LCA Results Note. Table 3 reports the statistics that are based on proportional classification, i.e. respondents are as to all by the When we use the LCA, we use the classification, i.e. every respondent is to the with the (Vermunt & Magidson, 2013). IMC = Instructional Manipulation Check. We find support for and a proportion of respondents were strong satisficers, who poorly on all indicators, while weak satisficers often speeded and failed the long IMC, Table We suggest that the of weak and strong satisficers may across for example, longer may result in higher levels of inattentiveness & 2008). the causes of weak and strong satisficing, we included age, and education The in the LCA that age membership. assess the consequences of satisficing we the three with to response across the five alexithymia in our For four the was lower for strong satisficers as to weak satisficers and attentive respondents, Supplementary Table For example, the for was for strong satisficers 3), for weak satisficers 2), and for attentive respondents This result suggests that strong satisficing affects the relationships between variables. In support of we found differences on the mean of one alexithymia across the three LCA Furthermore, satisficing on two other alexithymia For all five we found that the were higher for attentive respondents, Supplementary Table Next, we the relationships between failing the short and long IMC, on the one and straight-lining and speeding, on the other may have resulted from to the IMCs. has previously been suggested that to long IMCs may satisficing behavior (Anduiza & Galais, 2017). In condition 3 IMC the average proportion of respondents who straight-lined on the two screens with alexithymia items the short IMC did not differ from the proportion of straight-lining respondents on the three alexithymia screens, = = p = In condition 2 IMC straight-lining on the five alexithymia screens the long IMC was lower than straight-lining on the four screens with and survey questions = = p < However, in the control condition without IMCs, we found that straight-lining was also higher on these four exercise, and survey screens = = p < This is to be as the exercise, and survey question had less than five answer options, as was the case for the alexithymia answer options result in a higher probably of the same answer category across Note that we the occurrence of speeding and the IMCs, as only the that respondents to the survey was speeding did not differ across the four conditions in our study, that is, the respondents were speeders in condition 1 with both IMCs, to in condition 2 with only the long IMC, in condition 3 with only the short IMC and in the condition = 3, = p = the of IMCs in a survey did not to speeding Weak satisficing has been distinguished from strong satisficing in one previously published et al., but to our these alternative forms of satisficing have not been related to satisficing indicators, such as short IMCs, long IMCs, speeding and straight-lining. In previous studies, LCA has been applied to combine indicators for detecting satisficers (Maniaci & Rogge, 2014; Meade & Craig, 2012). Based on an LCA using multiple indicators, Kaminska et al. (2010) found that response was for respondents who these distinguished between attentive and inattentive respondents they did not distinguish between weak and strong satisficing. We found and empirical support for the occurrence of weak and strong satisficing, which is with Moreover, we found that strong satisficers are likely to speed and straight-line and that they relatively often fail both IMCs, whereas weak satisficers often speed and fail the long IMC. These results provide support for and In support of in the LCA 3), we found that younger respondents were more likely to be weak We also found support for differences in the consequences of weak and strong satisficing the responses of strong satisficers the relationships between variables, as by the differences in Weak and strong satisficers, when to attentive respondents, had different mean values and on the survey variables. Our research has knowledge about weak and strong satisficing and this has implications for researchers would the attention levels of respondents to be as high as the distinction we found between weak and strong satisficers implies that a be made between attention levels that respondents display and the number of respondents who be considered as respondents from an analysis implies higher test power and may reduce selection as satisficers may differ from attentive respondents on various (Anduiza & Galais, such as age Based on the reported we suggest that researchers use multiple satisficing indicators, with for example, short and long IMCs, to distinguish between weak satisficers, strong satisficers, and attentive respondents. Furthermore, studies aim to weak and strong satisficing In our study, we found that researchers relationships between variables, they strong satisficers, but that when they on of they both weak and strong satisficers for researchers who aim to detect both weak and strong satisficing assess the use of indicators speeding and straight-lining, for example, the “don’t know” option or no response at all (Kaminska et al., indicators may be for researchers who are not in satisficing, but who do require of correlations between variables the and of the in their survey. A research would be to assess the of various factors to question as by et al. We labelled the alternative IMCs “short” and but IMCs can also differ on other characteristics, for example, question and the that is assess the in which respondents alternative IMC-versions on such researchers which has previously been used to survey responses et al., et al., In addition, may consist of short questions, long questions or a In studies, short long IMCs be embedded in such different Supplementary are available at is an at the of at are data and variable has published in and of Paas is a at the of of The of research concern survey data and various marketing research has published in such as of the of and of in
DOI: 10.1093/ijpor/edz046
Authors: Meike Morren, Leonard J Paas
Institutions: Vrije Universiteit Amsterdam, University of Auckland
Language: English
Subfield: Sociology and Political Science
Field: Social Sciences
Domain: Social Sciences
Open Access status: hybrid
Estimated APC paid: $3,121
OpenAlex ID: W2995979067