Year: 2025
Type: conference-paper
Abstract: Multiple-choice question (MCQ) datasets like Massive Multitask Language Understanding (MMLU) are widely used to evaluate the commonsense, understanding, and problem-solving abilities of large language models (LLMs).However, the open-source nature of these benchmarks and the broad sources of training data for LLMs have inevitably led to benchmark contamination, which is studied in our contamination evaluation experiment, resulting in unreliable evaluation.To alleviate this issue, we propose the contamination-free MCQ benchmark called MMLU-CF, which reassesses LLMs' understanding of world knowledge by averting both unintentional and deliberate data contamination.To mitigate unintentional data contamination, we source questions from a broader domain of over 200 billion webpages and apply three specifically designed decontamination rules.To prevent deliberate data contamination, we divide the benchmark into validation and test sets with similar difficulty and subject distributions.The test set remains closed-source to ensure reliable results, while the validation set is publicly available to promote transparency and facilitate independent evaluation.We evaluated over 40 mainstream LLMs on the MMLU-CF.Compared to the original MMLU, not only LLMs' performances significantly dropped but also the performance rankings of them changed considerably.This indicates the effectiveness of our approach in establishing a contaminationfree and fairer evaluation standard.
Authors: Qihao Zhao, Yangyu Huang, Tengchao Lv, Lei Cui, Qinzheng Sun, Shaoguang Mao, Xin Zhang, Ying Xin, Qiufeng Yin, Scarlett Li, Furu Wei
Institution: Microsoft (United States)
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: bronze
OpenAlex ID: W4412945366