Work

Revisiting Supervision for Continual Representation Learning

Year: 2024

Type: conference-paper

Abstract: In the field of continual learning, models are designed to learn tasks one after the other. While most research has centered on supervised continual learning, there is a growing interest in unsupervised continual learning, which makes use of the vast amounts of unlabeled data. Recent studies have highlighted the strengths of unsupervised methods, particularly self-supervised learning, in providing robust representations.The improved transferability of those representations built with selfsupervised methods is often associated with the role played by the multilayer perceptron projector. In this work, we depart from this observationand reexamine the role of supervision in continual representation learning. We reckon that additional information, such as human annotations, should not deteriorate the quality of representations. Our findings show that supervised models when enhanced with a multi-layer perceptron head, can outperform self-supervised models in continual representation learning. This highlights the importance of the multi-layer perceptron projector in shaping feature transferability across a sequence of tasks in continual learning.

Source: Zenodo (CERN European Organization for Nuclear Research)

DOI: 10.5281/zenodo.13841790

Authors: Marczak, Daniel, Cygert, Sebastian, Trzciński, Tomasz, Twardowski, Bartłomiej

Institutions: Warsaw University of Technology, Gdańsk University of Technology, Universitat Autònoma de Barcelona, Computer Vision Center

Language: English

Topic: Domain Adaptation and Few-Shot Learning

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: green