Residual Stream Outliers as an Automatic Temperature Representation Mechanism for LLM Output Confidence
Year: 2026
Type: preprint
Abstract: The standard output pipeline of contemporary open-source large language models (LLMs) proceeds as follows: the final Transformer layer outputs a hidden state, which passes through Final RMSNorm, undergoes a linear projection to logits via the language modeling (LM) head, and is ultimately transformed into a next-token probability distribution by the softmax function. In this paper, we examine a frequently misunderstood mechanism within this pipeline: outliers in the residual stream are not numerical noise that ought to be “cleaned up,” but rather an active numerical representation of next-token certainty encoded within the numerical distribution. Its primary function is to elevate effective temperature. Outliers do not alter predictions by steering specific semantic directions; instead, by inflating the root-mean-square (RMS) statistic of the residual stream, they enlarge the denominator of the Final RMSNorm. This globally dampens the normalized magnitudes of all channels—suppressing overall logit scales, flattening the distribution, and thereby raising the effective temperature while decreasing output confidence. These outlierspikes predominantly originate at token boundaries characterized by high conditional entropy—most prominently following the sequence-initial <BOS> token (where predicting the very first content word relies entirely on broad corpus-level priors in the absence of context) and sentence-final punctuation (where subsequent clause-initiating tokens exhibit high branch entropy). Its auxiliary function is to concurrently and marginally increase the output probabilities of whitespace and punc-tuation characters: along the readout direction of the LM head, this channel predominantly elevates the logits of punctuation marks, whitespace, and delimiters, while depressing invalid byte fragments in the vocabulary. Crucially, neither effect alters the underlying semantic preference structure. Furthermore, intervention experiments demonstrate that artificially clipping or ablating these outlier channels induces spurious overconfidence: the collapsed denominator amplifies semantic logits and sharpens the predictive distribution, stripping the model of its entropy-mediated escape mechanism and trapping autoregressive generation in degenerate repetition loops. The outlier channel is typically persistent across intermediate layers (empirically, 54 out of 64 layers in a 27B-parameter model share the identical maximum outlier channel). However, it shifts to alternative channels in the initial and final layers, where the early-layer channel encodes corpus-level priors rather than punctuation or whitespace. Early layers establish a baseline confidence regime that passes directly through to the output, while subsequent and terminal layers revise and reconfigureit based on accumulated representations. As training progresses, the outlier magnitude in the final layer recedes from its peak in alignment with loss convergence. Final RMSNorm does not eliminate temperature; rather, it serves as the direct implementation substrate for this automatic temperature regulation mechanism. v2: Added Appendix: Mechanisms of Outlier Channels in Non-Terminal Layers
Author: Qiu, Haifeng
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7225457597