Work

Fully logits guided distillation with intermediate decision learning for deep model compression

Year: 2025

Type: article

Abstract: Knowledge distillation, as a model compression technique , has been widely applied in artificial intelligence to improve the efficiency of deep learning models, especially in resource-constrained environments. Considering that logit contains more decision information compared to the intermediate feature maps, fully logits guided distillation is proposed, which allows student networks to have better access to the guidance from both the intermediate and decision levels of the teacher’s network. Intermediate feature logicalization is designed, which perform logit transformations on the intermediate feature maps to obtain intermediate decision information. A logit matrisation strategy is proposed, which aim to capture inter-class information of the logits. Furthermore, cross layer distillation is presented in order to enable the final logit of the teacher to provide guidance to the intermediate layers of the student. The proposed mechanism can be embedded into State-of-the-art distillation frameworks to further improve the accuracy. Experiments conducted on the CIFAR-10, CIFAR-100, and ImageNet datasets demonstrate the effectiveness of the proposed method. Image classification accuracy was used as the evaluation metric, and the results show that the proposed method improves accuracy by an average of 1.84%, with a best improvement of 3.19% over the baseline models. Code is available at https://github.com/YiqinWang-JN/FLGD .

Source: Engineering Applications of Artificial Intelligence

DOI: 10.1016/j.engappai.2025.110850

Authors: Yiqin Wang, Ying Chen

Institution: Jiangnan University

Language: English

Topic: Advanced Neural Network Applications

Subfield: Computer Vision and Pattern Recognition

Field: Computer Science

Domain: Physical Sciences

Open Access status: hybrid

Estimated APC paid: $3,200

Funder: National Natural Science Foundation of China