Year: 2026
Type: preprint
Abstract: As Large Language Models (LLMs) exhibit plateauing performance on conventional benchmarks, a pivotal challenge persists: evaluating their proficiency in complex, open-ended tasks characterizing genuine expert-level cognition. Existing frameworks suffer from narrow domain coverage, reliance on generalist tasks, or self-evaluation biases. To bridge this gap, we present XpertBench, a high-fidelity benchmark engineered to assess LLMs across authentic professional domains. XpertBench consists of 1,346 meticulously curated tasks across 80 categories, spanning finance, healthcare, legal services, education, and dual-track research (STEM and Humanities). These tasks are derived from over 1,000 submissions by domain experts--including researchers from elite institutions and practitioners with extensive clinical or industrial experience--ensuring superior ecological validity. Each task uses detailed rubrics with mostly 15-40 weighted checkpoints to assess professional rigor. To facilitate scalable yet human-aligned assessment, we introduce ShotJudge, a novel evaluation paradigm that employs LLM judges calibrated with expert few-shot exemplars to mitigate self-rewarding biases. Our empirical evaluation of state-of-the-art LLMs reveals a pronounced performance ceiling: even leading models achieve a peak success rate of only ~66%, with a mean score around 55%. Models also exhibit domain-specific divergence, showing non-overlapping strengths in quantitative reasoning versus linguistic synthesis.. These findings underscore a significant "expert-gap" in current AI systems and establish XpertBench as a critical instrument for navigating the transition from general-purpose assistants to specialized professional collaborators.
Source: arXiv (Cornell University)
Authors: Liu, Xue, Ma, Xin, Ma, Yuxin, Peng, Yongchang, Wang, Duo, Wen, Zhoufutu, Zhang, Ge, Zhang, Kaiyuan, Chen, Xinyu, Ding, Yida, He, Tianci, Hou, Jiani, Hu, Liang, Huang, Ziyun, Hui, Yongzhe, Jiao, Jianpeng, Ju, Chennan, Kong, Yingru, Li, Yiran, Liu, Jiashuo, Liu, Mengyun, Ma, Luyao, Ni, Fei, Ni, Yiqing, Niu, Pengbo, Qiu, Yueyan, Ren, Yanle, Shen, Xinyu, Shi, Zilin, Wang, Zaiyuan, Yue, Wenjie, Zhang, Chun, Zhang, Shiyu, Zhang, Xinyi, Zhao, Kaiwen, Zhu, Zhenwei, Wu, Shanshan, Zhao, Qi, Huang, Wenhao
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7150752175