Transparent Language Models via Learned Pseudo-Riemannian Metrics
Year: 2026
Type: article
Abstract: We replace standard dot-product attention with a pseudo-Riemannian metric distance — a signed, input-dependent weighted squared distance over a shared embedding space. Unlike standard attention, where rotational invariance renders per-dimension analysis meaningless, the shared center space makes each metric dimension independently readable.Training a 152M-parameter model on 3B tokens of FineWeb-Edu at a cost of 3.5% perplexity, we find that the metric structures the entire model. In attention, the metric tensor spontaneously organizes a processing hierarchy from token classification through positional encoding to a multi-dimensional genre manifold. In the MLP, tracing the bigram "the light" through the final layer reveals two distinct computations at adjacent positions: sign-changing semantic disambiguation on the content word, and magnitude-varying predictive loading on the function word. Both attention and MLP internals are readable directly from the model's own values, without probing classifiers or sparse autoencoders.
Author: Smith, Richard
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7140192591
FWCI:
Cites:
Cited by:
Related to: