RATE: Reviewer Profiling and Annotation-free Training for Expertise Ranking in Peer Review Systems
Year: 2026
Type: conference-paper
Abstract: Reviewer assignment is increasingly critical yet challenging in the LLM era, where rapid topic shifts render many pre-2023 benchmarks outdated and where proxy signals poorly reflect true reviewer familiarity.We address this evaluation bottleneck by introducing LR-bench, a high-fidelity, up-to-date benchmark curated from 2024-2025 AI/NLP manuscripts with five-level self-assessed familiarity ratings collected via a large-scale email survey, yielding 1,055 expert-annotated paper-reviewer-score annotations.We further propose a reviewercentric ranking framework that distills each reviewer's recent publications into compact keyword-based profiles and fine-tunes an embedding model with weak preference supervision constructed from heuristic retrieval signals, enabling to match each manuscript against a reviewer profile directly.Across the LRbench and the CMU gold-standard dataset, our approach consistently achieves state-ofthe-art performance, outperforming strong embedding baselines by a clear margin.
Authors: Weicong Liu, Zixuan Yang, Yibo Zhao, Xiang Li
Institution: East China Normal University
Language: English
Subfield: Information Systems
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W7166853926