Auditing the Shadows: A Review of Methods to Detect Shared Training Data in Large Language Models
Year: 2025
Type: article
Abstract: Large language models (LLMs) are often trained on undisclosed data. This practice has intensified debates about transparency, copyright compliance, and reproducibility. From this lens, this article systematically reviews methodologies to detect shared training data across LLMs. More specifically, our review spans five methodological families: (1) lexical/semantic overlap metrics, which compare output similarity but struggle with knowledge convergence; (2) memorization analysis, which identifies verbatim regurgitation of rare training examples yet risks extracting copyrighted content; (3) temporal alignment by leveraging models’ knowledge cutoffs to infer shared data timelines; (4) adversarial susceptibility correlation to measure shared failure modes under attack; and (5) synthetic fingerprinting by embedding detectable artifacts in training data. Our review reveals several critical gaps: existing methods are highly siloed, with little cross-community dialogue, evaluation inconsistency epidemics, and ethical risks are understudied and potentially violate data privacy laws. We propose the taxonomy of auditing techniques, an approach to conduct the same, and conduct a case study to showcase its merit. Finally, this review highlights an under-explored knowledge gap and sets a roadmap for future research directions.
Source: ACM Computing Surveys
DOI: 10.1145/3774897
Author: M.Z. Naser
Institution: Clemson University
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W4415958484