Year: 2020
Type: article
Abstract: Deep neural networks suffer from the inability to preserve the learned data representation (i.e., catastrophic forgetting) in domains where the input data distribution is non-stationary, and it changes during training. Various selective synaptic\nplasticity approaches have been recently proposed to preserve network parameters, which are crucial for previously learned tasks while learning new tasks.\nWe explore such selective synaptic plasticity approaches through a unifying lens\nof memory replay and show the close relationship between methods like Elastic\nWeight Consolidation (EWC) and Memory-Aware-Synapses (MAS). We then propose a fundamentally different class of preservation methods that aim at preserving the distribution of the network’s output at an arbitrary layer for previous tasks\nwhile learning a new one. We propose the sliced Cramer distance as a suitable ´\nchoice for such preservation and evaluate our Sliced Cramer Preservation (SCP) ´\nalgorithm through extensive empirical investigations on various network architectures in both supervised and unsupervised learning settings. We show that SCP\nconsistently utilizes the learning capacity of the network better than online-EWC\nand MAS methods on various incremental learning tasks.
Institutions: HRL Laboratories (United States), Loughborough University
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W2994851835