Small Language Models: A Survey of Architectures, Training, Efficiency, and Deployment
Year: 2026
Type: preprint
Abstract: Small language models (SLMs) — foundation models developed under an explicit efficiency constraint — have moved from a compromise category to a research program in their own right. Models between a few hundred million and a few billion parameters now run on phones and single consumer GPUs, anchor production agent systems, and, on a growing set of tasks, match systems ten to one hundred times their size. I survey this field as a foundation-model topic: its scaling foundations, its data and training recipes, its model families and architectures, its efficiency methods, its adaptation and reasoning-transfer results, its deployment systems, and its evaluation practice. I base the survey on a primary-verified corpus of 80 works, each checked against its publisher or archive record. Three findings organize the synthesis. First, at small scale the binding constraint is data policy, not architecture: the field's best-controlled evidence shows training-mixture decisions dominating architectural ones. Second, the efficiency literature — quantization, pruning, distillation, cache economics, speculative decoding — was validated mostly on large models and transfers to already-small models by extrapolation, not demonstration. Third, the field's open problems are now predominantly epistemic: it lacks a stable definition of its object, disclosed recipes for its strongest models, and an evaluation protocol that measures capability per unit resource comparably. I argue eight gaps in those terms and close with the research directions they imply.
Source: Figshare
Author: Vinuraj Devaraj
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7226017011
