Logitwise Distillation Network: Improving Knowledge Distillation via Introducing Sample Confidence
Year: 2025
Type: article
Abstract: While existing knowledge distillation (KD) methods typically force students to mimic teacher features without considering prediction reliability, this practice risks propagating the teacher’s erroneous supervision to the student. To address this, we propose the Logitwise Distillation Network (LDN), a novel framework that dynamically quantifies sample-wise confidence through the ranking of ground truth labels in teacher logits. Specifically, LDN introduces three key innovations: (1) weighted class means that prioritize high-confidence samples, (2) adaptive feature selection based on logit ranking, and (3) positive–negative sample adjustment (PNSA) to reverse error-prone supervision. These components are unified into a feature direction (FD) loss, which guides students to selectively emulate trustworthy teacher features. Experiments on CIFAR-100 and ImageNet demonstrate that LDN achieves state-of-the-art performance, improving accuracy by 0.3–0.5% over SOTA methods. Notably, LDN exhibits stronger compatibility with homogeneous networks (2.4% gain over baselines) and requires no additional training costs when integrated into existing KD pipelines. This work advances feature distillation by addressing error propagation, offering a plug-and-play solution for reliable knowledge transfer.
Source: Applied Sciences
DOI: 10.3390/app15052285
Authors: Teng Shen, Zhenchao Cui, Jing Qi
Institution: Hebei University
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
Estimated APC paid: $2,896
OpenAlex ID: W4407765948