Year: 2026
Type: conference-paper
Abstract: Knowledge distillation (KD) transfers capabilities from large language models (LLMs) to smaller students, yet it can fail unpredictably and also underpins model leakage risks.Our analysis revealed several distillation traps: tail noise, off-policy instability, and, most fundamentally, the teacher-student gap, that distort training signals.These traps manifest as overconfident hallucinations, self-correction collapse, and local decoding degradation, causing distillation to fail.Motivated by these findings, we propose a post-hoc calibration method that, to the best of our knowledge, for the first time enables control over a teacher's distillability via reinforcement fine-tuning (RFT).Our objective combines task utility, KL anchor, and acrosstokenizer calibration reward.This makes distillability a practical safety lever for foundation models, connecting robust teacher-student transfer with deployment-aware model protection.Experiments across math, knowledge QA, and instruction-following tasks show that students distilled from distillable calibrated teachers outperform SFT and KD baselines, while undistillable calibrated teachers retain their task performance but cause distilled students to collapse, offering a practical knob for both better KD and model IP protection.
Institution: Nanyang Technological University
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
SDG: Quality education
Open Access status: hybrid
OpenAlex ID: W7166842531