Work

Random Dense Knowledge Distillation for Continual Learning

Year: 2026

Type: article

Abstract: Continual Learning (CL), involving sequential training on diverse tasks, often faces catastrophic forgetting. While knowledge distillation–based approaches exhibit notable success in preventing forgetting, we pinpoint a limitation in their ability to distill the cumulative knowledge of all the previous tasks. To remedy this, we propose Random Dense Knowledge Distillation (RDKD). RDKD uses a task pool to track the model’s capabilities. It partitions the output logits of the model into dense groups, each corresponding to a task in the task pool. It then distills all tasks’ knowledge using all groups. However, using all the groups can be computationally expensive, so we also suggest random group selection in each optimization step. Moreover, we propose an adaptive weighting scheme, which balances the learning of new classes and the retention of old classes, based on the count and similarity of the classes. Our RDKD outperforms recent state-of-the-art baselines across diverse benchmarks and scenarios. Empirical analysis underscores RDKD’s ability to enhance model stability, promotes flatter minima for improved generalization, and remains robust across various memory budgets and task orders. Moreover, it seamlessly integrates with other CL methods to boost performance and proves versatile in offline scenarios like model compression.

Source: ACM Transactions on Multimedia Computing Communications and Applications

DOI: 10.1145/3816731

Authors: Jie Chu, Pei Liu, Tong Su, Yunpeng Wu, Yu Song, Zenglin Shi

Institutions: Hefei University of Technology, Zhengzhou University

Language: English

Topic: Domain Adaptation and Few-Shot Learning

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: hybrid

Funder: National Natural Science Foundation of China

Award: G3071399522