Year: 2025
Type: conference-paper
Abstract: Catastrophic forgetting remains a formidable obstacle to building an omniscient model in large language models (LLMs).Despite the pioneering research on task-level forgetting in LLM fine-tuning, there is scant focus on forgetting during pre-training.We systematically explored the existence and measurement of forgetting in pre-training, questioning traditional metrics such as perplexity (PPL) and introducing new metrics to better detect entity memory retention.Based on our revised assessment of forgetting metrics, we explored low-cost, straightforward methods to mitigate forgetting during the pre-training phase.In addition, we carefully analyzed the learning curves, offering insights into the dynamics of forgetting.Extensive evaluations and analyses on forgetting of pre-training could facilitate future research on LLMs.
Authors: Chonghua Liao, Ruobing Xie, Xingwu Sun, Haowen Sun, Zhanhui Kang
Institutions: Tsinghua University, Tencent (China), University of Macau
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W4412886772