Work

Optimizing Knowledge Transfer Graph for Deep Collaborative Learning

Year: 2025

Type: article

Abstract: Knowledge transfer among multiple networks, using predicted probabilities or intermediate-layer activations, has evolved significantly through extensive manual design, ranging from simple teacher–student approaches (for example, knowledge distillation) to bidirectional cohort methods (for example, deep mutual learning). However, key factors such as network size, the number of networks, transfer direction, and loss function design interact in complexways and limit conventional methods to exploring only a narrow range of possible combinations. In this paper, we propose a novel training method that enables more flexible and diverse combinations of knowledge transfer strategies. Specifically, we introduce a graph-based representation called the knowledge transfer graph, which provides a unified framework for representing a wide variety of knowledge transfer strategies. Furthermore, we propose specialized loss functions that incorporate five distinct gate functions to dynamically control gradient propagation and thereby enable a richer set of knowledge transfer strategies. By systematically searching the structure of the knowledge transfer graph, our method automatically discovers more effective knowledge transfer strategies than those designed manually. Experimental results on eleven datasets demonstrate that our approach significantly enhances performance and reliably identifies effective graph structures.

Source: IEEE Access

DOI: 10.1109/access.2025.3579344

Authors: Soma Minami, Naoki Okamoto, Tsubasa Hirakawa, Takayoshi Yamashita, Hironobu Fujiyoshi

Institution: Chubu University

Language: English

Topic: Domain Adaptation and Few-Shot Learning

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: gold

Estimated APC paid: $2,075

Funder: New Energy and Industrial Technology Development Organization