Work

Trustworthy knowledge distillation via anchor-guided distribution learning

Year: 2025

Type: article

Abstract: Existing knowledge distillation methods typically do not adequately account for the representativeness of sampled training data or the adverse effects of missing classes within mini-batches, both of which can lead to suboptimal knowledge transfer. In this paper, we introduce A nchor-based K nowledge D istillation (AKD), a method that leverages the most informative and representative samples, referred to as anchors , during the transfer process, and matches the representation distribution of the data in the feature space rather than their actual representations. Anchors are strategically chosen based on their ability to encapsulate critical features of the data distribution, ensuring that the student model focuses on the most informative aspects of the teacher’s knowledge. The proposed method enables the introduction of information from a wider variety of classes at each mini-batch iteration, ensuring a more balanced data distribution and thereby improving the knowledge transfer process. Furthermore, by leveraging the static nature of anchors, we enhance the student model’s representation learning through attention maps, improving both convergence and generalization. The proposed approach is extensively evaluated across various datasets and tasks, demonstrating that the incorporation of anchors into the knowledge distillation process improving the accuracy and trustworthiness of the resulting models.

Source: Knowledge-Based Systems

DOI: 10.1016/j.knosys.2025.114297

Authors: Dimitrios Spanos, Nikolaos Passalis, Anastasios Tefas

Institution: Aristotle University of Thessaloniki

Language: English

Topic: Advanced Neural Network Applications

Subfield: Computer Vision and Pattern Recognition

Field: Computer Science

Domain: Physical Sciences

SDG: Quality education

Open Access status: hybrid

Estimated APC paid: $3,430