Self-Feature Regularization: Self-Feature Distillation Without Teacher Models
Year: 2021
Type: preprint
Abstract: Knowledge distillation is the process of transferring the knowledge from alarge model to a small model. In this process, the small model learns the
generalization ability of the large model and retains the performance close to
that of the large model. Knowledge distillation provides a training means to
migrate the knowledge of models, facilitating model deployment and speeding up
inference. However, previous distillation methods require pre-trained teacher
models, which still bring computational and storage overheads. In this paper, a
novel general training framework called Self-Feature Regularization~(SFR) is
proposed, which uses features in the deep layers to supervise feature learning
in the shallow layers, retains more semantic information. Specifically, we
firstly use EMD-l2 loss to match local features and a many-to-one approach to
distill features more intensively in the channel dimension. Then dynamic label
smoothing is used in the output layer to achieve better performance.
Experiments further show the effectiveness of our proposed framework.
Source: arXiv (Cornell University)
Authors: Wenxuan Fan, Zhenyan Hou
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W3138829374