When the adjustment fails: population-denominator sensitivity in coverage-adjusted PISA trends
Year: 2026
Type: preprint
Abstract: Programme for International Student Assessment (PISA) cycles sample different shares of each economy’s 15-year-old population, and the OECD’s coverage-adjusted top-quarter statistic depends on POP15, the reported population denominator. We analyse 2018–2022 Coverage Index 3 (CI3) changes for 73 economies and top-quarter trends for 69 under two named denominator scenarios: PISA POP15 and World Population Prospects (WPP). Estimates use 10 plausible values, 80 Fay balanced-repeated-replication replicates and a weighted-percentile convention calibrated to OECD’s published percentiles. CI3 changed by a mean 1.82 percentage points (SD 8.64; range − 45.19 to 27.05), with 24 absolute changes at least 5 points; an exact decomposition assigned the larger component to the survey channel in 50 economies. Mean absolute PISA–WPP trend contrasts were 3.4342, 3.4526 and 3.4290 points in mathematics, reading and science, exceeding the published link-error reference value for 38, 52 and 51 of 69 economies; link errors serve as descriptive references, not denominator-error standard errors. Aligning the external series to each economy’s observed PISA age-eligibility window and removing systems with documented territorial, vintage or consistency problems leaves contrasts of 3.29, 3.55 and 3.44 points in 58 economies. Annex A7’s documented synthetic-weight and zero-replacement rule was triggered in 6 of 159 economy–cycle cells: final-weight zeroing switched off the adjustment for Korea and Ireland; four further cases affected replicates only. Ratios to an explicitly invalid negative-weight continuation do not estimate bias, and neither population source is treated as ground truth. We recommend transparent POP15 provenance, named-scenario sensitivity reporting and non-reporting when synthetic weights are negative.
Source: Research Square
Language: English
Field: Social Sciences
Domain: Social Sciences
SDG: Reduced inequalities
Open Access status: green
OpenAlex ID: W7204489396