What a Sparse Code Carries and What a Learner Can Use: Identity, Order and Magnitude in Sparse Distributed Memory
Year: 2026
Type: preprint
Abstract: We study which information in a sparse top-k code is actually used by a downstream learner: the identities of the active units, their rank order, or their activation magnitudes. Using the trainable Sparse Distributed Memory (SDM) of Bricken et al. as a controlled testbed, we hold the network and top-k selection fixed while varying only the code passed from the sparse address layer to the output layer. We compare graded, geometric rank-weighted, linear rank, and binary codes, together with shuffled, index-ordered, reversed-rank, fixed-address, surrogate-gradient, and random-address controls. On Split CIFAR-10 embeddings, geometric rank coding remains close to graded activations, although a pre-specified 1-point equivalence criterion is not met. Binary coding loses 9.5 percentage points at 50 epochs and about 34 points at 500 epochs when addresses are learned through the present straight-through training rule. Reversed rank, despite retaining the same identity and ordering information as the correct rank code, performs substantially worse, showing that the information contained in a code does not by itself determine what a learner can exploit. When good addresses are trained first and frozen, the binary deficit at 500 epochs shrinks to about 1 point. The results suggest an interaction between the sparse code and the way its addresses are learned. More broadly, they motivate separating representation information from learnability when studying sparse neural systems. The experiments are limited to one trainable SDM implementation and one benchmark family; they make no claim about language models.
Author: Bose, Joy
Institution: Ericsson (Sweden)
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7225659310
Cites:
Cited by:
Related to: