When Weights Collide with Latents: Direct Weight-Space Arithmetic Induces Asymmetric Degeneration in Sparse Autoencoder Monosemanticity
Year: 2026
Type: preprint
Abstract: : Model editing through weight-space task vector arithmetic (θedited = θbase + λτ) has emerged as an exceptionally compute-efficient paradigm for steering foundational language models without full parameter retraining. Concurrently, Sparse Autoencoders (SAEs) trained on intermediate residual activations have become the gold standard for mechanistic interpretability by decomposing polysemantic neural representations into interpretable, monosemantic feature dictionaries. In this monograph, we present the first empirical and mechanistic investigation into the intersection of weight-space programming and sparse dictionary learning. We formalize the perturbation of transformer parameters along directional task vectors and evaluate its downstream consequence on monosemantic feature preservation and reconstruction fidelity. We uncover a striking Asymmetric Latent Degeneration Phenomenon: positive weight-space interpolation (λ > 0) causes a non-linear collapse in latent preservation (cosine similarity drops from 1.00 to 0.72), accompanied by an exponential surge in reconstruction error. Furthermore, we reveal an empirical asymmetry ratio of ρ = 0.950247, proving that adding versus subtracting task vectors induces non-symmetric representational drift. Finally, we formulate the Orthogonal Latent Re-Projection Kernel (OLR-K), a closed-form geometric correction that restores up to 88.4% of dictionary alignment without retraining the autoencoder. Our findings expose fundamental trade-offs between parameter arithmetic and causal interpretability, establishing essential guardrails for model editing and neural reprogramming. Key Highlights: 19-Page Comprehensive Theoretical & Empirical Monograph Formal Mathematical Proofs (Latent Orthogonality Drift & Rank Collapse) Empirical Validation across Gemma-2, LLaMA, and GPT-2 models Interactive Visualizer & Reproducible Colab Notebook included
Author: Mushfiqur Rahim, Md
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7225622012