TripleFact: Defending Data Contamination in the Evaluation of LLM-driven Fake News Detection
Year: 2025
Type: conference-paper
Abstract: The proliferation of large language models (LLMs) has introduced unprecedented challenges in fake news detection due to benchmark data contamination (BDC), where evaluation benchmarks are inadvertently memorized during the pre-training, leading to the inflated performance metrics.Traditional evaluation paradigms, reliant on static datasets and closedworld assumptions, fail to account the BDC risk in large-scale pre-training of current LLMs.This paper introduces TripleFact 1 , a novel evaluation framework for fake news detection task, which designed to mitigate BDC risk while prioritizing real-world applicability.Triple-Fact integrates three components: (1) Human-Adversarial Preference Testing (HAPT) to assess robustness against human-crafted misinformation, (2) Real-Time Web Agent with Asynchronous Validation (RTW-AV) to evaluate temporal generalization using dynamically sourced claims, and (3) Entity-Controlled Virtual Environment (ECVE) to eliminate entityspecific biases.Through experiments on 17 state-of-the-art LLMs, including GPT, LLaMA, and DeepSeek variants, TripleFact demonstrates superior contamination resistance compared to traditional benchmarks.Results reveal that BDC artificially inflates performance by up to 23% in conventional evaluations, while TripleFact Score (TFS) remain stable within 4% absolute error under controlled contamination.The framework's ability to disentangle genuine detection capabilities from memorization artifacts underscores its potential as a fake news detection benchmark for the LLM era. Aspect Traditional Approach LLM-Driven Approach Data Scope Curated, domain-specific datasets Massive, open-domain corpora Training Supervised fine-tuning pre-training on diverse text EvaluationClosed-world train-test splits Zero/few-shot prompting Key Limitation Limited generalization Data contamination risks
Institutions: University College Dublin, Georgia Institute of Technology
Language: English
Subfield: Sociology and Political Science
Field: Social Sciences
Domain: Social Sciences
Open Access status: bronze
OpenAlex ID: W4412945605