Work

Task-Dependent Intrinsic Dimensionality Across Transformer Layers: An Exploratory Study of Qwen2.5-1.5B

Year: 2026
Type: preprint
Abstract: We ask whether task categories differ in the dimensionality of a pretrained language model’s internal representations and whether their relationship changes with depth. Last-input-token representations are extracted from Qwen2.5-1.5B for 200 prompts spanning fac-tual recall, arithmetic, reasoning, and coding. Representation geometry is summarized using participation ratio, covariance-spectrum entropy effective rank, principal-component vari-ance thresholds, and the TWO-NN local intrinsic-dimension estimator. The primary held-out analysis contains 12 prompts from two template families per category. For raw representa-tions, the reasoning-minus-recall participation-ratio contrast is positive at blocks 1–20 and negative at blocks 21–28. At block 24, participation ratio is 2.869 for factual recall and 1.397 for reasoning; after the final block and normalization, it is 2.235 and 2.126, respectively. The observations establish a descriptive depth-dependent relationship in these prompt sets, rather than a uniform ordering of task categories. Limited held-out family counts prevent reliable primary family-bootstrap intervals, and insufficient length overlap prevents a joint matched-length comparison. We report a bounded exploratory experiment with its analysis code and limitations.
Institution: Kakatiya University
Language: English
Open Access status: green
OpenAlex ID: W7225878680
Cites: 7
Cited by:
Related to: 10