From Screening Automation to Review Prioritization: Leakage-Aware Evaluation of LLM-Augmented Administrative Selection
Type: article
Abstract: Organizations increasingly consider large language models (LLMs) for administrative selection tasks that combine formal criteria with human judgment. We evaluate four LLM scoring designs in a real university scholarship program (1,352 applications, 250 recipients), treating realized human selections as an operational benchmark rather than ground truth under a leakage-aware protocol. Rubric-based codification provides the strongest broad review-prioritization performance, reaching 80% recall of human-selected applicants within a review pool of 800 of 1,336 applications, while stronger few-shot calibration improves final-scale alignment. We add deployment policies for using such rankings under imperfect recall, together with qualitative divergence cases, positioning LLM-augmented selection as review prioritization rather than screening automation.
Source: AIS Electronic Library (AISeL)
Authors: Hyun, Sibaek, Yoo, Byungjoon
Institution: Seoul National University
Language: English
Topic: AI and HR Technologies
Domain: Social Sciences
Open Access status: green
OpenAlex ID: W7222907044
