Relation-State Continuity in Qwen2.5-7B Base: Evidence for an Attention-Topology Path
Year: 2026
Type: preprint
Abstract: I probe Qwen2.5-7B base across eight relation-tracking cases and report four aggregate scores summarising the 29-layer profile: residual carry AUC 0.55, attention anchor mass AUC 0.089, formation peak 0.94, cross-layer circuit score 0.39. The formation profile is bimodal with peaks at layer 1 and layer 28, and no single intermediate layer dominates. The work of routing anchor information to query position is spread across layers 19 through 27, where residual carry rises from 0.60 to 0.81 while attention to anchors stays in the 0.12 to 0.23 range. I label this pattern attention topology path. The classification is observational; causal verification via activation patching is the next experiment.
Author: Chen, Ho Yiing
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7162764729
Cites:
Cited by:
Related to: