Get started
Work

From Document Classification to Scholar Shortlists: An Audit of Shortlist Sensitivity to OpenAlex's Topic-Classifier Update

Year: 2026
Type: preprint
Abstract: Open databases assign topics to works; their case rests on work-level agreement, institutional mix and citation-impact ranking. A list of people adds an ordering rule and a cutoff; none of the three checks covers an author profile or a fixed-length list. In October 2026 OpenAlex replaced its topic assignment. We scored a frozen set under both assignments, holding works, authors, pool and cutoff fixed and re-estimating weights per label world. At work level, 47.0% of 429,985 works changed their primary topic. In top-50 lists we build for six keyword-defined query domains, 80 of 300 seats changed hands (26.7%), 18% to 42% by domain. Across the two specifications built from the archived label tables (B, C) the pooled figure stays between 23.7% and 26.7%; a crossed bracket (D) gives 24.7%, a counting variant (E) 23.3%, and a separate endpoint bracket (A), read on 16 October, 29.3%; three to eight of each top ten kept their seats. Pooled, 42 of the 80 dropped seats fall outside the narrow band, so enrichment at the cut is partial; shorter lists turn over faster. Labels, text and weights moved together, so this is an association between two versions of one database, not an isolated classifier effect. For research evaluation, the question is whether the list, at its length, still names the same people. A shortlist is a reading of one label version at one length; its stability is separate from label accuracy. We state what it should report: the label version, the date it was scored, the length, and the width of the cutoff band.
DOI: 10.31222/osf.io/syfkn_v1
Author: Yuan Gao
Language: English
Open Access status: green
OpenAlex ID: W7225730374
Cites:
Cited by:
Related to: