Year: 2026
Type: conference-paper
Abstract: Large Language Model (LLM) evaluation is frequently challenged by prompt sensitivity, which limits the reliability of benchmarks relying on a single template.To address this, we systematically investigate a multi-prompt evaluation framework that aggregates performance across diverse templates, leveraging a Quality-Diversity (QD) prompt selection mechanism to ensure both variety and fidelity.Through extensive empirical analysis, we uncover a novel Evaluation Scaling Law: within the range of prompt counts and model families we tested, evaluation reliability tends to improve with template diversity.Specifically, our experiments demonstrate that evaluation variance diminishes as the scale of prompts increases, while ranking monotonicity exhibits a log-linear improvement relative to the number of templates.By decoupling a model's intrinsic capability from template-specific fluctuations, our framework facilitates more reliable assessments of LLMs, thereby providing a solid foundation for benchmarking that enables more effective monitoring of training dynamics and rigorous comparisons among state-of-the-art models.
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: bronze
OpenAlex ID: W7224601685