Year: 2026
Type: conference-paper
Abstract: Recent years have seen rapid growth in evaluation and benchmarking in NLP, driven by advances in large language models (LLMs).This growth has shifted evaluation from measuring generalization to tracking capability, often without reference to training assumptions.We argue that this creates a conceptual gap: results are frequently interpreted without considering what models could plausibly have learned, rendering many conclusions scientifically underdetermined.We propose an expectation-aware view, where the informativeness of evaluation depends on its relationship to training data, model design, and tasks.We further distinguish between evaluation for scientific understanding and capability tracking, and provide recommendations for aligning evaluation with its intended purpose in the LLM era.
Author: Jiawei Zhou
Institution: Stony Brook University
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W7166884377