Work

Wrong field, wrong baseline, wrong performance rankings: Why publication-level normalization matters in research assessment exercises

Year: 2026
Type: article
Abstract: In selected-output research assessment, a fixed submission quota draws on eligible publication portfolios that differ in size across fields. These unequal selection opportunities motivate normalizing selected-output scores and make the choice of reference class consequential. Italy’s 2020–2024 Valutazione della Qualità della Ricerca (VQR) explicitly normalizes selected-output scores by the researcher’s administrative area. This paper examines whether, within that framework, STEMM outputs should instead be normalized by the publication’s scientific field. We simulate a VQR-style assessment of Italian universities using Web of Science articles and reviews produced during 2020–2024 by 37,366 professors at 96 universities. We select an average of 2.5 outputs per professor under VQR-like constraints, assign VQR-like scores from citation percentiles, and compare university-area rankings under author-area and publication-field normalization. Scores are highly correlated, but rankings are not equivalent: Physics and Industrial and Information Engineering display substantial rank movements and quartile changes. Conditional on the VQR’s decision to normalize scores for cross-field institutional aggregation, a publication-field baseline is conceptually more coherent because it follows the object evaluated. It does not, however, by itself remove portfolio-size and selection effects.
Language: English
Open Access status: hybrid
Estimated APC paid: $3,220
OpenAlex ID: W7235341623
FWCI:
Cites: 29
Cited by:
Related to: 10