Multi-assistant Dynamic Setting Method for Knowledge Distillation
Year: 2025
Type: article
Abstract: Knowledge distillation is increasingly gaining attention in key areas such as model compression for object recognition.Through in-depth research into the efficiency of knowledge distillation and an analysis of the characteristics of knowledge transfer between the teacher and student models,it is found that the reasonable setting of an assistant model can significantly reduce the performance gap between the teacher and student.However,the unreasonable choice of the scale and number of assistant models can have a negative impact on the student.Therefore,this paper proposes an innovative multi-assistant knowledge distillation training framework,which optimizes the process of knowledge transfer from the teacher to the student by dynamically adjusting the number and scale of assistant models,thereby improving the training accuracy of the student model.In addition,this paper also designs a dynamic stopping strategy for knowledge distillation,sets student models with different training methods as a control group,and achieves personalized design of the stopping rounds for knowledge distillation,further improving the training efficiency of the student model and constructing a more streamlined and efficient multi-assistant knowledge distillation framework.Experiments on public datasets prove the effectiveness of the proposed multi-assistant dynamic setting method for knowledge distillation.
Author: SI Yuehang, CHENG Qing, HUANG Jincai
Institution: National University of Defense Technology
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
SDG: Quality education
Open Access status: green
OpenAlex ID: W7084152387
FWCI:
Cites:
Cited by:
Related to: