Year: 2026
Type: article
Abstract: This paper provides a survey on memorization of training data of large language models (LLMs), summarizingaround 180 studies across natural language processing, security, and other research fields. With the rapid proliferationof LLMs, concerns about memorization of training data have become increasingly critical. The issues involved areextensive, encompassing security, copyright, and sound model evaluation. Recent studies have sought to defineand categorize memorization, to quantify its prevalence across different model scales, and to explore mitigationstrategies such as deduplication, unlearning, and knowledge editing. These efforts have revealed that memorizationis not merely a byproduct of overfitting but a nuanced phenomenon dependent on string duplication, model size, andcontext length. Looking forward, promising directions include (i) developing principled frameworks to balance utilityand safety by distinguishing harmful memorization from legitimate knowledge retention, (ii) establishing unifiedtaxonomies and benchmarks to evaluate memorization across domains and languages, and (iii) extending the analysisof memorization into multimodal models, where text, image, and audio data interact in complex ways. Through thissurvey, we synthesize the fragmented literature along the axes of training sets, models, and outputs and highlightemerging research avenues that will shape the responsible development of LLMs.
Authors: Shotaro Ishihara, Hiromu Takahashi
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: diamond
OpenAlex ID: W7166868568
FWCI:
Cites:
Cited by:
Related to: