LATVEC: Stabilizing Iterative Latent Workspaces for Neural Sequence Models
Year: 2026
Type: preprint
Abstract: Iterative latent workspaces offer an alternative computational paradigm for maintaining, routing, and transforming structured relational information during neural sequence processing. However, repeatedly updating a shared continuous latent workspace across unconstrained recurrent iterations frequently leads to rapid representational collapse, wherein initially diverse slot representations contract into a degenerate low-rank subspace. We introduce **LATVEC**, an instrumented multi-slot latent workspace architecture designed to empirically investigate and govern this phenomenon. LATVEC combines a causal autoregressive Transformer perception backbone with an attention-free, gated pairwise communication workspace operating over $M=48$ discrete slots. We trace workspace stability via continuous effective-rank trajectories ($r_{\text{eff}}$) and systematically explore the interaction between recurrent update step sizes ($\alpha$) and off-diagonal gate sparsity regularization ($\lambda_{\text{gate}}$). Across rigorous benchmarks, unconstrained workspace dynamics exhibit rapid rank collapse ($r_{\text{eff}} \to 1.00$), whereas a jointly constrained regime ($\alpha \in [0.02, 0.05], \lambda_{\text{gate}} \in [0.01, 0.10]$) robustly stabilizes representation dimensionality ($r_{\text{eff}} \approx 8.45$ in small models, $r_{\text{eff}} \approx 4.93$ at 251.3M parameter scale). Multi-seed evaluations confirm this stability is reproducible across both pre-training and response-masked supervised fine-tuning (SFT). We then conduct extensive causal and functional ablations, evaluating active workspace dynamics against a zeroed workspace bypass (backbone-only control) and a norm-matched Gaussian scrambled negative control, alongside alternative readout interfaces (single-head vs. multi-head cross-attention). State randomization severely degrades task performance (3.1% accuracy), demonstrating that the learned workspace representation is functionally consumed by downstream decoding layers. However, under compute-matched conditions across 2 to 10-hop relational reasoning chains, the iterative workspace does not demonstrate a statistical advantage over the standard Transformer backbone control (35.7% vs. 39.3%). These empirical findings establish an essential conceptual distinction between **workspace stability** and **task-level superiority**: preventing recurrent rank collapse is a necessary engineering prerequisite for stable iterative computation, but does not inherently confer cognitive reasoning advantages over standard attention mechanisms. We present LATVEC as a foundational mechanistic study of latent-state dynamics, collapse prevention, and functional workspace utilization in neural sequence architectures.
Author: Murat Gürsoy
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7203873093
Cites:
Cited by:
Related to: