Work

Sparse Distributed Memory is a Continual Learner

Year: 2023

Type: preprint

Abstract: Continual learning is a problem for artificial neural networks that their biological counterparts are adept at solving. Building on work using Sparse Distributed Memory (SDM) to connect a core neural circuit with the powerful Transformer model, we create a modified Multi-Layered Perceptron (MLP) that is a strong continual learner. We find that every component of our MLP variant translated from biology is necessary for continual learning. Our solution is also free from any memory replay or task information, and introduces novel methods to train sparse networks that may be broadly applicable.

Source: arXiv (Cornell University)

DOI: 10.48550/arxiv.2303.11934

Authors: Bricken, Trenton, Davies, Xander, Singh, Deepak, Krotov, Dmitry, Kreiman, Gabriel

Institutions: Harvard University, IBM (United States), Massachusetts Institute of Technology

Language: English

Topic: Domain Adaptation and Few-Shot Learning

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: green

Funders: National Science Foundation, National Institutes of Health, National Eye Institute

Awards: Neural dynamics underlying spatiotemporal cognitive integration, A Center for Brains, Minds and Machines: the Science and the Technology of Intelligence

OpenAlex ID: W4360103626

Cites:

Cited by: 9

Related to: 10