Biologically grounded locality priors close the data gap for vision transformers in neural prediction
Year: 2026
Type: preprint
Abstract: In computer vision, transformers have largely replaced convolutional networks, but not in neuroscience: vision transformers predict mouse V1 responses well only when trained on thousands of neurons. With no built-in locality, they must learn from data where each neuron looks. Electrophysiological experiments now commonly record from tens of neurons, with a few thousand stimuli, two orders of magnitude below the scale at which transformers are typically trained; with this little data, a transformer that sees the whole image tends to overfit, and convolutional neural networks and Gaussian processes perform better. We address this problem in the retina by restricting what the transformer sees: for each neuron, a differentiable crop keeps only the image features inside a small circle centered on its receptive field, O ( r 2 ) tokens for a radius r rather than the whole image, and a transformer, shared by all neurons, processes them. We call this model circular receptive-field vision transformer (CiRF-ViT) and test it on multi-electrode-array recordings of 41 mouse and 49 salamander retinal ganglion cells. Against CNN and Gaussian-process baselines, and a plain vision transformer with the same core that sees the whole image, CiRF-ViT reaches the highest mean noise-corrected correlation on both datasets (0.89 on mouse, 0.93 on salamander). We also tested the models with the local spike-triggered average (LSTA), the measured change of a cell’s response to small changes of a natural image, which for a model is the gradient of its response with respect to the image. For some cells, the LSTA shows a polarity inversion: around some images the cell responds like an ON cell and around others like an OFF cell. CiRF-ViT, never trained on this measure, reproduces this polarity inversion, which a linear-nonlinear model cannot capture by construction, and agrees with the measured LSTA as well as the baselines do. A biologically grounded, per-neuron locality prior is therefore enough to make transformers competitive for retinal response prediction well below their usual data scale.
Language: English
Subfield: Cognitive Neuroscience
Field: Neuroscience
Domain: Life Sciences
Open Access status: green
OpenAlex ID: W7211940616