Year: 2025
Type: conference-paper
Abstract: Large Language Models (LLMs) have achieved impressive results across numerous NLP tasks, and fine-tuning them for Machine Translation (MT) has improved their performance.However, vanilla fine-tuning often leads to catastrophic forgetting, compromising the broad general abilities of LLMs and introducing potential security risks.These abilities, which are developed using proprietary and unavailable training data, make simple data replay methods ineffective.To overcome this issue, we propose a novel approach called RaDis (Rationale Distillation).RaDis harnesses the strong generative capabilities of LLMs to create rationales for training data, which are then "replayed" to prevent forgetting.These rationales connect prior knowledge with new tasks, acting as selfdistillation targets to regulate the training process.By jointly training on reference translations and self-generated rationales, the model can learn new translation skills while preserving its general abilities across other tasks.Additionally, RaDis provides a fresh perspective on using rationales in the CL field and has the potential to serve as a general continual learning method for a variety of tasks.
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
OpenAlex ID: W4412888237