Work

A coded knowledge distillation framework for image classification based on adaptive JPEG encoding

Year: 2024

Type: article

Abstract: In knowledge distillation (KD), a lightweight student model yields enhanced test accuracy by mimicking the behaviour of a pre-trained large model (teacher). However, the cumbersome teacher model often makes over-confident responses, resulting in poor generalization when presented with unseen data. Consequently, a student trained by such a teacher also inherits this problem. To mitigate this issue, in this paper, we present a new framework of KD dubbed coded knowledge distillation (CKD) in which the student is trained to mimic instead the behaviour of a coded teacher. Compared to the teacher in KD, the coded teacher in CKD has an additional adaptive encoding layer in the front, which adaptively encodes an input image into a compressed version (using JPEG encoding for instance) and then feeds the compressed input image to the pre-trained teacher. Comprehensive experimental results show the effectiveness of CKD over KD. In addition, we extend the deployment of a coded teacher to other knowledge transfer methods, showcasing its ability to enhance test accuracy across these methods.

Source: Pattern Recognition

DOI: 10.1016/j.patcog.2024.110966

Authors: Ahmed H. Salamah, Shayan Mohajer Hamidi, En-Hui Yang

Institution: University of Waterloo

Language: English

Topic: Advanced Neural Network Applications

Subfield: Computer Vision and Pattern Recognition

Field: Computer Science

Domain: Physical Sciences

SDG: Quality education

Open Access status: hybrid

Estimated APC paid: $2,710