Wrong field, wrong baseline, wrong performance rankings: Why publication-level normalization matters in research assessment exercises
Year: 2026
Type: article
Abstract: In selected-output research assessment, a fixed submission quota draws on eligible publication portfolios that differ in size across fields. These unequal selection opportunities motivate normalizing selected-output scores and make the choice of reference class consequential. Italy’s 2020–2024 Valutazione della Qualità della Ricerca (VQR) explicitly normalizes selected-output scores by the researcher’s administrative area. This paper examines whether, within that framework, STEMM outputs should instead be normalized by the publication’s scientific field. We simulate a VQR-style assessment of Italian universities using Web of Science articles and reviews produced during 2020–2024 by 37,366 professors at 96 universities. We select an average of 2.5 outputs per professor under VQR-like constraints, assign VQR-like scores from citation percentiles, and compare university-area rankings under author-area and publication-field normalization. Scores are highly correlated, but rankings are not equivalent: Physics and Industrial and Information Engineering display substantial rank movements and quartile changes. Conditional on the VQR’s decision to normalize scores for cross-field institutional aggregation, a publication-field baseline is conceptually more coherent because it follows the object evaluated. It does not, however, by itself remove portfolio-size and selection effects.
Source: Journal of Informetrics
Authors: Giovanni Abramo, Ciriaco Andrea D’Angelo
Institutions: Mercatorum University, University of Rome Tor Vergata
Language: English
Subfield: Library and Information Sciences
Field: Social Sciences
Domain: Social Sciences
Open Access status: hybrid
Estimated APC paid: $3,220
OpenAlex ID: W7235341623
