Year: 2025
Type: conference-paper
Abstract: Large language models (LLMs) are known to memorize parts of their training data, raising important concerns around privacy and security.While previous research has focused on studying memorization in pre-trained models, much less is known about how knowledge distillation (KD) affects memorization.In this study, we explore how different KD methods influence the memorization of fine-tuned task data when a large teacher model is distilled into smaller student variants.This study demonstrates that distilling a larger teacher model, fine-tuned on a dataset, into a smaller variant not only lowers computational costs and model size but also significantly reduces the memorization risks compared to standard fine-tuning approaches.
Author: Simardeep Singh
Institution: Indian Institute of Technology Roorkee
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
SDG: Quality education
Open Access status: gold
OpenAlex ID: W4412944284