Year: 2026
Type: conference-paper
Abstract: As LLM capabilities advance rapidly, the evaluation methods used to assess them increasingly lag behind.Traditional benchmarks rely on programmatic verification of narrow, surfacelevel constraints, yet real-world instruction following and agentic tasks demand assessment of nuanced, context-dependent behaviors that resist simple scripted checks.We present a systematic analysis of expert-curated rubricbased evaluation as an alternative paradigm, drawing on empirical evidence from two domains: complex instruction following and enterprise agentic tasks.We first articulate five design principles for constructing high-quality rubrics, including Maximum Viable Atomicity, intent-aware criterion design, and iterative LLM-judge calibration.To validate these principles, we introduce COMPLEX-IF, a new expert-curated instruction-following dataset in which each prompt is paired with 10-40 atomic rubric criteria.We demonstrate that these expert rubrics are not only better evaluation instruments but also highly effective training signals: training on approximately 1,000 COMPLEX-IF examples yields +15.5 pp improvement for a 4B-parameter model and +12.2 pp for a 235B-parameter model on instruction following, while single-epoch RL training on a rubricgraded enterprise environment produces gains that transfer to out-of-distribution benchmarks the model was never trained on (+4.5 pp BFCL, +7.4 pp τ 2 -Bench, +6.8 pp Toolathlon).Our findings establish that expert-authored rubrics improve both the measurement and the development of frontier LLM capabilities, serving as effective evaluation and RL training signals.
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W7166857223