Get started
Work

The Rotation-Retention Law: Knowledge Loss Is Proportional to Representational Rotation in Fine-Tuned Language Models. With EgoRA: Entropy-Governed Orthogonality Regularization for Adaptation

Year: 2026
Type: article
Abstract: We introduce the Rotation-Retention Law: knowledge loss during fine-tuning is directly proportional to the mean rotation angle of attention head weight vectors. We validate this geometric relationship across 20 experimental conditions, three architectures (Llama, Mistral, Phi-3), and multiple model scales, and introduce EgoRA — a new entropy-governed orthogonality regularization method that substantially reduces catastrophic forgetting. I invite the research community to scrutinize these findings, replicate the experiments, and provide open feedback. ===Why does fine-tuning destroy pre-trained knowledge? We show the answer is geometric: knowledge loss is proportional to the mean rotation angle of attention head weight vectors between the pre-trained and adapted states. We formalize this as the Rotation-Retention Law: ΔM ∝ θ̄ = E[arccos(cossim(w_i^base, w_i^tuned))], where ΔM is the change in benchmark performance (MMLU) and θ̄ is the mean rotation angle across all attention heads. Across 20 experimental conditions with full benchmark validation—spanning three architectures (Llama 3.1 8B, Mistral 7B, Phi-3 Mini), three model scales (1B, 3B, 8B), two fine-tuning domains (instruction-following, medical), and three adaptation methods (LoRA, DoRA, EgoRA)—we find: (i) θ̄ > 5° predicts MMLU loss exceeding 3 pp in all high-rotation conditions; (ii) θ̄ < 3° predicts MMLU loss below 3 pp in all low-rotation conditions; (iii) the proportionality constant k = |ΔM|/θ̄ varies by regime and scale (0.22–0.96 pp/degree), with higher stability in the high-rotation regime. To establish causality, we introduce EgoRA (Entropy-Governed Orthogonality Regularization for Adaptation), which constrains rotation during training and reduces MMLU loss by up to 12.7× while reducing damaged heads from 56% to under 4%. We validate the law across three architectures, finding that while absolute rotation varies by 6× (3.6°–21.7°), the normalized rotation reduction from EgoRA is remarkably consistent (θ̄_norm = 0.42 ± 0.06), and higher rotation universally predicts greater MMLU loss. We propose θ̄_norm < 0.5 as a universal, architecture-independent certification threshold for knowledge-preserving fine-tuning. The Rotation-Retention Law reframes catastrophic forgetting as a geometric phenomenon and provides a falsifiable, architecture-agnostic diagnostic for fine-tuning quality.Submitted to SSRN and Zenodo on April 3, 2026. This is the initial public release for publication of the research findings. © 2026 Mark Dillerop. All rights reserved. This preprint is licensed under a Creative Commons Attribution 4.0 International License (CC BY 4.0). The methods described in this work are subject to U.S. Provisional Patent Application No. 64/024,742.
DOI: 10.5281/zenodo.19398708
Language: English
Open Access status: green
OpenAlex ID: W7148635577
FWCI:
Cites:
Cited by:
Related to: