Work

Hallucinations as Orthogonal Noise: Inference-Time Manifold Alignment via Dynamic Contextual Orthogonalization

Year: 2026

Type: conference-paper

Abstract: Hallucination in Large Language Models (LLMs)-characterized by the generation of content inconsistent with contextual facts or logical constraints-remains a persistent challenge for reliable deployment.In this work, we address this issue through a geometric framework rooted in the linear representation hypothesis.We propose that hallucinations manifest as orthogonal noise relative to the semantic manifold of the residual stream.Specifically, we hypothesize that while attention heads ideally propagate information congruent with the context subspace, hallucinations arise when specific heads introduce components orthogonal to this subspace, disrupting the coherence of the latent representation.

Source: Findings of the Association for Computational Linguistics: ACL (Findings of ACL)

DOI: 10.18653/v1/2026.findings-acl.1822

Authors: Mingkuan Zhao, Wentao Hu, Tianchen Huang, Yuheng Min, Suquan Chen, Yide Gao, Yanbo Zhai, Shuangyong Song, Xuelong Li

Institutions: Xi'an Jiaotong University, China Telecom, University of Science and Technology of China, Tsinghua University, China Telecom (China)

Language: English

Topic: Topic Modeling

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: hybrid

OpenAlex ID: W7166859649

FWCI:

Cites:

Cited by:

Related to: 10