Internal State Geometry is Measurable in Small Open Language Models: A Cross-Family Survey
Year: 2026
Type: preprint
Abstract: I report a survey of dense activation observation on twelve open language models spanning five architecture families (BLOOM, BLOOMZ, Phi-2, Pythia, Qwen2.5, InternLM-2) and parameter counts from 124M to 7.6B. Every model in the qualified set produces an observable internal state structure under a fixed-budget probing protocol: coverage in the band 90.6% to 97.05%, total channel count from 523k to 1.54M, hooked-module count from 147 to 253. Three cross-family patterns are reported: module-count-to-parameter ratio varies by an order of magnitude between architectures (23 to 84 modules per billion parameters); channel count scales sub-linearly with parameter count C ~ N_p^0.63 on this sample; the linear-to-norm module ratio is roughly 5:1 across families. Small open LMs are not opaque substrates; they are dense, observable, and comparable.
Author: Chen, Ho Yiing
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7162782252
Cites:
Cited by:
Related to: