Dimensional Collapse and Expansion in Language Embeddings: Sign-Invariant Structural Dynamics Across Models
Year: 2026
Type: article
Abstract: We show that dimensional change in language embeddings is notpurely unidirectional collapse but a fundamentally bidirectional phenomenonthat includes expansion. While prior work has focused oncollapse as a loss of effective dimensionality, we demonstrate that semanticallydiverse inputs systematically induce dimensional expansion.Across four experimental axes—repetition, preprocessing, semanticpolarity/topic structure, and lexical diversity manipulation—weestablish that: (1) repetition induces near-linear dimensional collapse,(2) semantic convergence induces collapse without repetition, (3) semanticallydiverse domains induce expansion, and (4) the direction ofdimensional change relative to random baselines is preserved acrossembedding models.Cross-model validation using all-MiniLM-L6-v2 (D=384) and allmpnet-base-v2 (D=768) shows that while absolute effective rank andcondition ordering are model-dependent, the direction of dimensionalchange (collapse vs expansion) is invariant.These findings establish the sign of dimensional change as a modelagnosticstructural invariant, suggesting that relative contraction andexpansion constitute a fundamental property of embedding dynamicsbeyond model-specific geometry.
Author: Sakai, Jun
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7155105401
FWCI:
Cites:
Cited by:
Related to: