Let Them Guess: The Sensitivity of Political Information Effects to Variations of Question Format
Year: 2011
Type: preprint
Abstract: Political information is generally measured by counting the correct answers given by respondents to factual survey questions related to politics. Some of the weaknesses of the additive approach to measuring skills have already been discussed in the psychometric literature, but only recently have they started to arouse the interest of political scientists. To mention only a few of these shortcomings (discussed in this article), the questions that are used for constructing ability scales are not of equal difficulty; questions of different format yield different rates of correct responses by allowing the respondents to guess at unequal rates; some items perform better than others at reliably estimating the knowledge of respondents. The question that we address here is whether the format related inflations of the averages of knowledge scales affect the reliability and validity of inferential statistical findings obtained with models in which knowledge is used as an independent or dependent variable in a multivariate fashion. In other words, assuming that we construct a political information variable by adding up or averaging the number of correct answers to quiz-like questions; do different question formats yield different slopes or error terms for knowledge or for its predictors in regression-based models? We start by constructing an individual-level format-inflation variable that accounts for the possibility and plausibility that correct answers were obtained by random guessing using data from a survey experiment carried out in Hungary in 2009. Then, we compare the results of regression models in which format-inflation is not accounted for with the results of nested models in which it is controlled for. Political knowledge is alternatively used as independent and as dependent. Using the count of multiple choice and binary response items as controls does not seem to alter the results obtained with these models. Finally, in order to provide additional empirical support to our findings and to rule out the possibility of having results that only reflect the idiosyncrasies of the Hungarian public opinion, we use similar regression models as the ones described previously for the countries in the second module of the CSES database.
Source: SSRN Electronic Journal
Author: Paul T. Weith
Institution: Central European University
Language: English
Field: Decision Sciences
Domain: Social Sciences
Open Access status: green
OpenAlex ID: W1591301901