Work

Knowledge distillation in deep learning and its applications

Year: 2021

Type: article

Abstract: Deep learning based models are relatively large, and it is hard to deploy such models on resource-limited devices such as mobile phones and embedded devices. One possible solution is knowledge distillation whereby a smaller model (student model) is trained by utilizing the information from a larger model (teacher model). In this paper, we present an outlook of knowledge distillation techniques applied to deep learning models. To compare the performances of different techniques, we propose a new metric called distillation metric which compares different knowledge distillation solutions based on models' sizes and accuracy scores. Based on the survey, some interesting conclusions are drawn and presented in this paper including the current challenges and possible research directions.

Source: PeerJ Computer Science

DOI: 10.7717/peerj-cs.474

Authors: Abdolmaged Alkhulaifi, Fahad Alsahli, Irfan Ahmad

Institution: King Fahd University of Petroleum and Minerals

Language: English

Topic: Advanced Neural Network Applications

Subfield: Computer Vision and Pattern Recognition

Field: Computer Science

Domain: Physical Sciences

Open Access status: gold

Estimated APC paid: $1,395

Funder: King Fahd University of Petroleum and Minerals