Year: 2021
Type: conference-paper
Abstract: Continual learning aims to build intelligent agents that can continuously learn new tasks over time while preserving the old learned knowledge. Ideally, the agent should continually learn without adding a huge computational and memory overhead to learn a new task or remember the old ones. The main challenge in this paradigm is catastrophically forgetting previous tasks when the model is optimized for a new one. Existing methods mitigate this problem at the expense of increasing the model capacity or replaying the old samples. This hinders its applicability to real-world applications where the old data might not be available and computation and memory efficiency is required. To address these limitations, we proposed SpaceNet a new architectural-based strategy that utilizes the available fixed-capacity of the model efficiently. We harness the significant redundancy of deep neural networks and learn each task in a compact space using dynamic sparse training. SpaceNet learns semi- distributed sparse representation for each task. This representation has two key advantages: (1) it reduces the interference between tasks. (2) It leaves free neurons for future tasks without adding extra computation and memory overhead.
Authors: Sokar, Ghada A.Z.N., Mocanu, Decebal Constantin; id_orcid 0000-0002-5636-7683, Pechenizkiy, Mykola
Institutions: Eindhoven University of Technology, University of Jyväskylä
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7062385893