Are LLM Benchmarks Already Contaminated? A Systematic Review of Contamination Detection Methods
Year: 2026
Type: review
Study design: Systematic Review
Abstract: Large Language Models (LLMs) are trained on web-scale corpora, increasing the risk that benchmark test data appears in training sets and inflates reported performance.We present a systematic literature review of 55 studies on LLM benchmark contamination through late 2025.Our contributions are: (1) a four-tier contamination taxonomy (Exact, Syntactic, Semantic, Task-Level; T1-T4); (2) a comparative analysis of five detection families (stringmatching, likelihood-based, membership inference, LLM-prompted detection, and benchmark auditing), including access assumptions and failure modes; (3) a synthesis of contamination evidence on MMLU, GSM8K, HU-MANEVAL, and HELLASWAG by measurement construct; (4) a comparative evaluation of mitigation strategies across lifecycle points, access assumptions, and evidence maturity; and (5) a Contamination Transparency Card (CTC) framework for future releases.Across studies, no detection method is consistently reliable across contamination tiers, model-access settings, and training stages.We identify instruction tuning as a persistent blind spot, note that RL/post-training contamination auditing is only beginning to mature, and report inflation estimates spanning roughly 6%-40% under benchmark-and setting-dependent assumptions.
Institution: The University of Texas at San Antonio
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W7166876690