Work

From Screening Automation to Review Prioritization: Leakage-Aware Evaluation of LLM-Augmented Administrative Selection

Type: article
Abstract: Organizations increasingly consider large language models (LLMs) for administrative selection tasks that combine formal criteria with human judgment. We evaluate four LLM scoring designs in a real university scholarship program (1,352 applications, 250 recipients), treating realized human selections as an operational benchmark rather than ground truth under a leakage-aware protocol. Rubric-based codification provides the strongest broad review-prioritization performance, reaching 80% recall of human-selected applicants within a review pool of 800 of 1,336 applications, while stronger few-shot calibration improves final-scale alignment. We add deployment policies for using such rankings under imperfect recall, together with qualitative divergence cases, positioning LLM-augmented selection as review prioritization rather than screening automation.
Language: English
Open Access status: green
OpenAlex ID: W7222907044
Cites: 12
Cited by:
Related to: 10