Year: 2026
Type: conference-paper
Abstract: Memorization in language models is widely studied but remains difficult to isolate and control.Understanding when and what models memorize is essential for explaining their predictions, yet existing approaches are post-hoc: they can detect memorization in trained models, but cannot disentangle its effects from architecture, data, or optimization.We introduce MEMORY DIAL, a training framework that makes memorization pressure an explicit, controllable variable.MEMORY DIAL interpolates between standard cross-entropy and a temperature-sharpened objective via a single parameter α, producing a family of models identical in architecture and training setup (within each sweep), differing only in memorization pressure.Experiments across six architectures and five benchmarks demonstrate that: (1) α reliably controls memorization pressure, with seen-example accuracy increasing monotonically while unseen accuracy remains stable; (2) larger models are more responsive to memorization pressure; and (3) frequent sequences are easier to memorize than rare ones.Additional analyses show that the effect is robust across a range of sharpening temperatures, differs qualitatively from singletemperature cross-entropy, transfers to multilingual settings, and is detectable even on naturally occurring single-occurrence sequences.MEMORY DIAL provides a controlled experimental framework for studying how memorization behavior emerges and interacts with generalization in language models.
Authors: Xiangbo Zhang, Ali Emami
Institutions: Georgia Institute of Technology, Emory University
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W7166838861