Editor's evaluation: Learning predictive cognitive maps with spiking neurons during behavior and replays
Year: 2022
Type: peer-review
Abstract: The hippocampus has been proposed to encode environments using a representation that contains predictive information about likely future states, called the successor representation.However, it is not clear how such a representation could be learned in the hippocampal circuit.Here, we propose a plasticity rule that can learn this predictive map of the environment using a spiking neural network.We connect this biologically plausible plasticity rule to reinforcement learning, mathematically and numerically showing that it implements the TD-lambda algorithm.By spanning these different levels, we show how our framework naturally encompasses behavioral activity and replays, smoothly moving from rate to temporal coding, and allows learning over behavioral timescales with a plasticity rule acting on a timescale of milliseconds.We discuss how biological parameters such as dwelling times at states, neuronal firing rates and neuromodulation relate to the delay discounting parameter of the TD algorithm, and how they influence the learned representation.We also find that, in agreement with psychological studies and contrary to reinforcement learning theory, the discount factor decreases hyperbolically with time.Finally, our framework suggests a role for replays, in both aiding learning in novel environments and finding shortcut trajectories that were not experienced during behavior, in agreement with experimental data. Editor's evaluationThis is an important article that leverages a spiking network model of the hippocampal circuit to show how spike-time-dependent plasticity can implement predictive reinforcement learning and form a predictive map of the environment.The authors provide a convincing and solid framework for understanding the prediction based learning rules that may be employed by the hippocampus to optimize an animal's behavior.This paper will be of interest to theoretical and experimental neuroscientists working on learning and memory as it provides new ways to connect computational models to experimental data that has yet to be fully explored from a reinforcement learning perspective.
Author: Lisa M Giocomo
Institutions: Stanford Medicine, Stanford University
Language: English
Topic: Memory and Neural Mechanisms
Subfield: Cognitive Neuroscience
Field: Neuroscience
Domain: Life Sciences
Open Access status: gold
OpenAlex ID: W4327660751