Year: 2026
Type: conference-paper
Abstract: This paper explores attention attractorstokens that draw significantly high attentionin large language models.We analyze them from three perspectives: (1) Functionality: We demonstrate their role in aggregating information from preceding contexts to facilitate future predictions.(2) Distribution: Through layer-wise and token-wise analysis, we reveal that attention attractors are widely distributed across layers but predominantly originate from low-semantic words like "_the".(3) Mechanism: We demonstrate the correlation between attention weights allocated to tokens with their specific activation dimension values.We hope these findings provide new insights into the attention mechanisms of large language models and inspire further exploration.Code will be released at https://github.com/luyouqi233/ AttentionAttractor.
Authors: Ziheng Wang, Zihao Yue, Wenxuan Wang, Qin Jin
Institution: Renmin University of China
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W7166880451