Harmonization Still Failing? Inconsistency of Education Variables in Cross-National Public Opinion Surveys
Year: 2015
Type: article
Abstract: During recent decades, cross-national comparative research in public opinion has grown tremendously, both in quantity and quality. Through the increased availability of various types of international public opinion survey data, many research questions can today be tackled from a comparative point of view. This allows researchers to test the generality of hypotheses, as well as contextual effects that may explain why countries differ the way they do (Przeworski & Teune, 1970). The credibility of comparative studies, however, hinges on the cross-national comparability of the data they are based on. This is a matter of continuous debate among comparative survey researchers and methodologists (e.g., Heath, Martin, & Spreckelsen, 2009; Hoffmeyer-Zlotnik & Warner, 2014). Consistency across data sources is a necessary condition of comparability. If data are not coded consistently across time points and surveys, they are not comparable across countries. Consistency across data sources is important because it allows researchers to compare results from different studies and pool different data sources for analysis. The aim of this study is to evaluate the consistency of four cross-national public opinion survey data sets for a large number of European countries. We chose the variable highest level of education for this purpose. This measure is of special interest for two reasons: Firstly, it is one of the most widely used variables in public opinion research (see Smith, 1995) because it is a “core” variable reflecting socialization, social stratification, and individual life chances. From numerous studies we know that, across countries, educational attainment substantially correlates with attitudes, beliefs, values, and behaviors (e.g., Bekhuis, Lubbers, & Verkuyten, 2014; Kalmijn, 2003; Weakliem, 2002). Secondly, educational attainment is one of the most difficult background variables to measure and code in a comparable and coherent way across countries and studies (see Braun & Müller, 1997; Kerckhoff & Dylan, 1999; Schneider, 2009; Schröder & Ganzeboom, 2013). Until recently, little research involving only small numbers of countries and surveys was available with respect to the comparable measurement of educational attainment (e.g., Braun & Müller, 1997; Kerckhoff, Ezell, & Brown, 2002; Smith, 1995). Regarding the more specific issue of incoherence of educational attainment data across data sources, the distribution of education coded using the International Standard Classification of Education (ISCED) was shown to be inconsistent across time and surveys. Hoffmeyer-Zlotnik (2008) compares the distributions of ISCED in the European Social Survey (ESS) 2002 with (unspecified) Eurostat data for Austria, Denmark, France, and Spain. He finds discrepancies, and he explains them by inconsistent coding of country-specific education categories into ISCED across data sources. Kieffer (2010) compares data from the French ESS 2002–2008 with the French Labour Force Survey from corresponding years, confirming the results of Hoffmeyer-Zlotnik, while adding detailed explanations for these inconsistencies for France. Schneider (2009) compares official data from the European Labour Force Survey and the European Survey of Income and Living Conditions with data from the ESS for the years 2002–2006 for 26 European countries. In this most comprehensive analysis of education data inconsistencies to date, she reveals inconsistencies over time, surveys, and countries. The research question of this study is: Do we find the same inconsistencies of educational attainment data across time and surveys for widely used, recent, cross-national public opinion surveys that have not yet been evaluated in this respect? This study thus adds to existing research by analyzing the ESS, the International Social Survey Programme (ISSP), the European Values Study (EVS), and the Eurobarometer. We compare data from 2008 to 2012. In the next part of the article, the methodological background is summarized by reviewing the process of ex ante output harmonization and the ISCED, which is widely used for the harmonization of educational attainment data. In the third section of the article, the data sources, analysis strategy, education coding, and the indicator for coherence across data sets, are introduced. The results are then presented, followed by a discussion of the findings and conclusions. Educational systems, with their idiosyncratic institutions and certificates, differ substantially across countries. Response categories for the question “what is your highest level of education?” include many proper names rather than generic descriptions that are universally understood. Thus, they cannot be translated. Educational attainment can only be measured using country-specific response categories in survey questionnaires. Respondents choose their attainment from a list of country-specific educational qualifications or levels. For cross-national comparisons, these qualifications have to be coded into an internationally comparable education variable, a process called output harmonization. Comparative survey designers normally plan how to make variables comparable across countries in advance of the survey. This process is called ex ante output harmonization (Ehling, 2003). It requires specifying a cross-national coding framework and the mapping of country-specific survey responses to this framework in the survey design phase. The country-specific data collection instrument has to be developed with the cross-national coding framework in mind because the latter implies the kinds of distinctions required for cross-national comparison. While most comparative surveys aim at ex ante output harmonization for the education variable, they do not coordinate this process with each other. Country-specific questionnaire items and cross-national coding frameworks, as well as the relationship between the two and their documentation, differ across surveys. The most commonly used cross-national coding framework for ex ante harmonization of education data in surveys is the ISCED. ISCED was designed by UNESCO in the 1970s and revised in 1997 and 2011. The aim of ISCED is “to serve as an instrument suitable for assembling, compiling and presenting comparable indicators and statistics of education both within individual countries and internationally” (UNESCO-UIS, 1997 [2006], p. 7). It defines comparable education categories applicable around the world. Because the newest ISCED version has not yet been widely implemented, this study refers to ISCED 1997. ISCED 1997 consists of seven main levels: ISCED 0: Pre-primary education (or not completed primary education) ISCED 1: Primary education or first stage of basic education ISCED 2: Lower secondary or second stage of basic education ISCED 3: Upper secondary education ISCED 4: Post-secondary non-tertiary education ISCED 5: First stage of tertiary education ISCED 6: Second stage of tertiary education Dimensions of differentiation within education levels such as vocational, pre-vocational, and general education, as well as whether a qualification allows access to a higher level of education, are not implemented in most surveys and thus are not used in this study, despite their known importance for predicting, for example, labor market outcomes (e.g., Schneider, 2010). We analyze data from four cross-national public opinion surveys that have not yet been examined with respect to education variable consistency. In the early 1970s, the Eurobarometer program was launched by the European Commission. ISCED 1997 main levels were implemented in three Standard and Special Eurobarometer studies in 2010 and 2011. From the late 1970s onward, the academically driven public opinion research programs EVS and ISSP were established. The EVS 2008 contains three ISCED variables. One reflects the seven main ISCED 1997 levels and is used in this study. The ISSP changed its harmonized education variable in 2011 from a nonstandard education scheme to a scheme closely related to but not identical with ISCED 1997. The ESS was launched in the early 2000s with an ISCED main level variable that was later corrected to a five-level version and introduced a detailed cross-national education variable closely related to ISCED 2011 in 2010. Respondents aged 25–64 are selected so that samples are as comparable as possible. We check the consistency of European public opinion data with respect to two dimensions: consistency over time and consistency across surveys. Firstly, we check whether education distributions differ within surveys and countries over time. This can be done for ESS, ISSP, and Eurobarometer data because, for these studies at least two rounds are available for this time period. Especially with respect to the education measurement changes in ESS 2010 and ISSP 2011, a detailed look at those data over time is useful for identifying and evaluating the effects of those changes. For comparison over time in the ESS, we also include ESS 2002, 2004, and 2006 because these data have been corrected since the inconsistencies were first presented (Hoffmeyer-Zlotnik, 2008; Kieffer, 2010; Schneider, 2009). Secondly, we compare the distribution of the harmonized education variable within time points and countries across surveys. For these comparisons, we can also include the EVS. ESS data from 2010 or 2012 are used as a benchmark because, owing to the large-scale revision effort to improve cross-country consistency, as well as the validity of its education measurement, it can be expected to provide high-quality education variables. Because ISSP did not use ISCED for education coding before 2011, we only use years 2011 and 2012 for the comparison of ISSP and ESS. The education variables of EVS, Eurobarometer, and ESS 2010–2012 are, or can be, coded into the seven main ISCED 1997 levels for comparisons over time and across surveys (see Table 1). The variables for ESS rounds 2002, 2004, 2006, and 2008 as well as the ISSP education variable, however, have to be treated separately. For the ESS, we aggregate ISCED categories 0 and 1 and categories 5 and 6 for the analysis over time. In the ISSP, we create a four-category measure to render the data before and after the changes in 2011 comparable (see Table 2). For the cross-survey comparison, the detailed education variable in ESS 2010 and 2012 is coded to fit the new ISSP education variable with seven categories. Categories and Recodes of the Education Variables Into ISCED 1997 for Cross-survey Comparison Sources:Eurobarometer 73.2 (February–March 2010) files from Eurostat, data file versions 2.0.1, variable v362. Eurobarometer 73.3 (March–April 2010) files from Eurostat, data file versions 2.0.1, variable v362. ESS 2010–2012, data file versions: 3.0 (2010), 2.0 (2012), variable edulvlb. EVS 2008, data file version 3.0.0, variable v336. Categories and Recodes of the Education Variables Into ISCED 1997 for Cross-survey Comparison Sources:Eurobarometer 73.2 (February–March 2010) files from Eurostat, data file versions 2.0.1, variable v362. Eurobarometer 73.3 (March–April 2010) files from Eurostat, data file versions 2.0.1, variable v362. ESS 2010–2012, data file versions: 3.0 (2010), 2.0 (2012), variable edulvlb. EVS 2008, data file version 3.0.0, variable v336. Categories and Relationship Between the Education Variables in ISSP 2008–2012 and ESS 2010 for Cross-time and Cross-survey and Time Comparison Sources:ESS 2010–2012, data file versions: 3.0. (2010), 2.0 (2012), variable edulvlb. ISSP 2008–2012, data file versions: 2.2.0 (2008), 3.0.0 (2009), 2.0.0 (2010), 2.0.0 (2011), 2.0.0 (2012), variable DEGREE. Categories and Relationship Between the Education Variables in ISSP 2008–2012 and ESS 2010 for Cross-time and Cross-survey and Time Comparison Sources:ESS 2010–2012, data file versions: 3.0. (2010), 2.0 (2012), variable edulvlb. ISSP 2008–2012, data file versions: 2.2.0 (2008), 3.0.0 (2009), 2.0.0 (2010), 2.0.0 (2011), 2.0.0 (2012), variable DEGREE. For comparing the education distributions and measuring inconsistencies, Duncan’s dissimilarity index is used (Duncan & Duncan, 1955) as a summary measure consistent with Schneider (2009). The index was originally developed for measuring residential segregation, but it can be generalized to measure differences in the distributions of categorical variables. The index is rescaled to range from 0 to 100. The dissimilarity index can then be interpreted as the percentage of cases that would have to change categories to achieve equal distributions across two data sources. Formally, if xi denotes the size of category i of k ISCED categories for country A in year B in survey S in percent and yi denotes the same for country A in year B in survey T, the dissimilarity index is defined as D = ½ ∑i=1k|xi−yi|?>. The equivalent holds when survey rounds are compared rather than surveys. An example of the calculation is given in Table 3: When comparing the distribution of the harmonized education variable for Spain in EVS 2008 and ESS 2010, we first calculate the absolute difference between data sources for each ISCED category. These differences are summed across ISCED categories and divided by two. The larger the differences between individual ISCED categories across surveys, the larger Duncan’s index. Example of Calculation Duncan’s Dissimilarity Index for Spain Sources:ESS 2010, data file version: 3.0, variable edulvlb, weighted using dweight. N = 1276. EVS 2008, data file version 3.0.0, variable v336. N = 974. Sample selection. Only respondents aged 25–64. Example of Calculation Duncan’s Dissimilarity Index for Spain Sources:ESS 2010, data file version: 3.0, variable edulvlb, weighted using dweight. N = 1276. EVS 2008, data file version 3.0.0, variable v336. N = 974. Sample selection. Only respondents aged 25–64. Duncan’s dissimilarity index for educational attainment distributions over time within ESS, ISSP, and Eurobarometer data averaged across countries. Sources:ESS 2002–2008, data file versions: 6.2 (2002), 3.3 (2004), 3.4 (2006), 4.1 (2008), variable edulvla, weighted using dweight. ESS 2010–2012, data file versions: 3.0. (2010), 2.0 (2012), variable edulvlb, weighted using dweight. Eurobarometer 73.2 (February to March 2010) files from Eurostat, data file versions 2.0.1, variable v362. Eurobarometer 73.3 (March to April 2010) files from Eurostat, data file versions 2.0.1, variable v362. Eurobarometer 75.4 (2011) files from Eurostat, data file; version 3.0.1, variable v105. All Eurobarometer data weighted to correct regional oversampling for Germany and the U.K. ISSP 2008–2012, data file versions: 2.2.0 (2008), 3.0.0 (2009), 2.0.0 (2010), 2.0.0 (2011), 2.0.0 (2012), variable DEGREE, data weighted to correct regional oversampling for Germany. Sample selection. Only respondents aged 25–64. Countries included. ESS: BE, BG, DE, DK, ES, FI, GB, IE, NL, NO, PL, PT, SE, SI; Eurobarometer: all 27 EU-member states; ISSP: CH, CZ, DE, DK, FI, FR, GB (excluding Northern Ireland), HR, NO, RU, SE, SI, SK, TR Duncan’s Dissimilarity Index for Educational Attainment Distributions Over Time for Eurobarometer, ISSP, and ESS Data Sources:ESS 2002–2008, data file versions: 6.2 (2002), 3.3 (2004), 3.4 (2006), 4.1 (2008), variable edulvla, weighted using dweight. ESS 2010–2012, data file versions: 3.0 (2010), 2.0 (2012), variable edulvlb, weighted using dweight. Eurobarometer 73.2 (February to March 2010) files from Eurostat, data file versions 2.0.1, variable v362. Eurobarometer 73.3 (March–April 2010) files from Eurostat, data file versions 2.0.1, variable v362. Eurobarometer 75.4 (2011) files from Eurostat, data file; version 3.0.1, variable v105. All Eurobarometer data weighted to correct regional oversampling for Germany and the U.K. ISSP 2008–2012, data file versions: 2.2.0 (2008), 3.0.0 (2009), 2.0.0 (2010), 2.0.0 (2011), 2.0.0 (2012), variable DEGREE, data weighted to correct regional oversampling for Germany; Wallonia for BE, Northern is for Sample selection. Only respondents aged 25–64. Duncan’s Dissimilarity Index for Educational Attainment Distributions Over Time for Eurobarometer, ISSP, and ESS Data Sources:ESS 2002–2008, data file versions: 6.2 (2002), 3.3 (2004), 3.4 (2006), 4.1 (2008), variable edulvla, weighted using dweight. ESS 2010–2012, data file versions: 3.0 (2010), 2.0 (2012), variable edulvlb, weighted using dweight. Eurobarometer 73.2 (February to March 2010) files from Eurostat, data file versions 2.0.1, variable v362. Eurobarometer 73.3 (March–April 2010) files from Eurostat, data file versions 2.0.1, variable v362. Eurobarometer 75.4 (2011) files from Eurostat, data file; version 3.0.1, variable v105. All Eurobarometer data weighted to correct regional oversampling for Germany and the U.K. ISSP 2008–2012, data file versions: 2.2.0 (2008), 3.0.0 (2009), 2.0.0 (2010), 2.0.0 (2011), 2.0.0 (2012), variable DEGREE, data weighted to correct regional oversampling for Germany; Wallonia for BE, Northern is for Sample selection. Only respondents aged 25–64. In the distributions over time are in the Eurobarometer and the ESS with of cases to categories to achieve equal distributions across survey In both ESS and ISSP, with measurement changes and in the year the measurement changes. they more in the ESS than in the The across all comparisons over time is D = that we compare a variable with seven categories for Eurobarometer, in the ESS and ISSP, the variable consists of and four The larger the number of the more can be so the in the Eurobarometer and the of the ISSP are more also that the of countries is the same but not surveys. Dissimilarity Index for Educational Attainment Distributions EVS, ISSP, and Eurobarometer ESS Sources:ESS data file versions: 3.0 (2010), 2.0 variable edulvlb, weighted using dweight. For comparison with ISSP, Wallonia in and Northern in GB are EVS 2008, data file version variable data weighted to correct regional oversampling for and the U.K. Eurobarometer 73.2 (February–March 2010) files from Eurostat, data file versions 2.0.1, variable v362. Eurobarometer 73.3 (March–April 2010) files from Eurostat, data file versions 2.0.1, variable v362. Eurobarometer 75.4 (2011) files from Eurostat, data file; version 3.0.1, variable v105. All Eurobarometer data weighted to correct regional oversampling for Germany and the U.K. ISSP data file versions: 2.0.0 (2011), 2.0.0 variable DEGREE, data weighted to correct regional oversampling for Germany; Wallonia for BE, Northern is for Sample selection. Only respondents aged 25–64. Dissimilarity Index for Educational Attainment Distributions EVS, ISSP, and Eurobarometer ESS Sources:ESS data file versions: 3.0 (2010), 2.0 variable edulvlb, weighted using dweight. For comparison with ISSP, Wallonia in and Northern in GB are EVS 2008, data file version variable data weighted to correct regional oversampling for and the U.K. Eurobarometer 73.2 (February–March 2010) files from Eurostat, data file versions 2.0.1, variable v362. Eurobarometer 73.3 (March–April 2010) files from Eurostat, data file versions 2.0.1, variable v362. Eurobarometer 75.4 (2011) files from Eurostat, data file; version 3.0.1, variable v105. All Eurobarometer data weighted to correct regional oversampling for Germany and the U.K. ISSP data file versions: 2.0.0 (2011), 2.0.0 variable DEGREE, data weighted to correct regional oversampling for Germany; Wallonia for BE, Northern is for Sample selection. Only respondents aged 25–64. Duncan’s dissimilarity index for educational attainment distributions comparing EVS, ISSP, and Eurobarometer with ESS averaged across countries. Sources:ESS 2010–2012, data file versions: 3.0 (2010), 2.0 (2012), variable edulvlb, weighted using dweight. EVS 2008, data file version 3.0.0, variable data weighted to correct regional oversampling for and the U.K. Eurobarometer 73.2 (February–March 2010) files from Eurostat, data file versions 2.0.1, variable Eurobarometer 73.3 (March–April 2010) files from Eurostat, data file versions 2.0.1, variable Eurobarometer 75.4 (2011) files from Eurostat, data file; version 3.0.1, variable v105. All Eurobarometer data weighted to correct regional oversampling for Germany and the U.K. ISSP data file versions: 2.0.0 (2011), 2.0.0 variable DEGREE, data weighted to correct regional oversampling for Germany. Sample selection. Only respondents aged 25–64. Countries included. BG, CZ, DE, DK, FI, FR, GB (excluding Northern in ISSP PL, SE, SI, all surveys different education distributions than the ESS. The dissimilarity can be between the ESS and EVS and between the ESS and Eurobarometer A level of is between ESS and ISSP, despite the that the coding between the two education variables was for this study. The across the countries is D = across surveys are thus more than as as inconsistencies over time is the also when only at the countries in all the distributions of educational attainment in four large cross-national public opinion surveys over a inconsistencies are The first are inconsistencies across rounds within individual survey The second are inconsistencies within years across surveys. The size of the across surveys on the comparability of educational attainment variables across these data It that more than a of cases on have to change categories to achieve equal a the comparability of results across studies using these data is in It also not look for these data These inconsistencies cannot differences in education differences in these distributions over time such as the we look be because they change For identical time points and countries, differences in distributions are expected because the samples of the different surveys all have and were harmonized for analysis. The inconsistencies have methodological by the Survey are at least four explanations for these instrument measurement and Firstly, on the is one for differences across surveys and survey The surveys we look at all have of which be by education across surveys and however, way to check how this has on Secondly, on the measurement the education question is not across surveys or countries, and is different of this question and the response categories across surveys survey This may in differences in validity across surveys, countries, and time points and explain of the inconsistencies For example, in the ISSP for the question the of the the rather than the highest educational In the ISSP, are on the and the level of education of is to be compared with data (see also Schneider, 2009). also related to the of different questionnaire items across surveys and time measurement differ across data sets because of social or of of the surveys we use We cannot the more closely inconsistencies in harmonization that the coding of country-specific education variables over time and surveys into the cross-national education may also explain the results in this study. While all surveys use ISCED as the harmonization framework from ISSP before 2011, which explains why the methodological changes in the ISSP in 2011 to higher over time than the change in the ESS they to it We can two kinds of this because of a of on how to country-specific education categories to ISCED and from the official For years of basic education is coded as ISCED 1 of ISCED and basic education is coded as ISCED of in the ISSP to use the coding used in ISSP with the education variable before 2011. such as these are and have been & Dylan, 1999; Kieffer, 2010; Schneider, 2009). from official explain the inconsistencies between the ESS and the surveys for the U.K. and For the it was to the of Education to ISCED rather than it is which not make when comparing this with European countries Because this is a the ESS substantially from the ISSP and the Eurobarometer, which use the official For basic education completed before is in ISCED rather than at ISCED in the ESS because the were increased in from official aim at comparability across countries and time, but they are not across different public opinion surveys. we for the measurement, coding, and harmonization of educational attainment in comparative public opinion surveys. We that cross-national survey and cross-survey can make a difference for data consistency by measurement and harmonization Firstly, ex ante output harmonization of variables educational attainment is a and to be and & for which it would be across surveys. Secondly, surveys more from each other. While the Eurobarometer is designed and by opinion in the ESS, ISSP, and EVS of rather country that and from the and (e.g., ISSP 2010). The ESS a of country than EVS and the of the new education in 2010, a process with in and of the and was so that only changes were required in 2012. The education measurement change in the ISSP 2011, in was to many in 2012. These differences in survey in the of the ESS, may explain the over time in the Eurobarometer and ESS compared with the ISCED can be for to which is why public opinion survey for the of from official ISCED they do so in an on an version of ISCED for public opinion surveys ESS, EVS, and ISSP may be way This was by the the number We would to the Data and the Social Data for the data sets for this study and for their We would also to the for their and versions of this were presented at the European on in in and at the of the for in
DOI: 10.1093/ijpor/edv025
Authors: Verena Ortmanns, Silke L. Schneider
Institution: GESIS - Leibniz Institute for the Social Sciences
Language: English
Subfield: Sociology and Political Science
Field: Social Sciences
Domain: Social Sciences
Open Access status: hybrid
Estimated APC paid: $3,121
Funders: European Commission, Leibniz-Gemeinschaft
Award: G4183413470
OpenAlex ID: W2405723709
