Beyond Decorrelation: An Integration Dynamics Framework (IDF) for Neural Network Learning
Year: 2026
Type: preprint
Abstract: Understanding how neural networks reorganize their internal representations during learning remains an open problem in deep learning research. While prior work has examined representation compression and feature disentanglement, the evolution of global statistical dependency among hidden units measured by Total Correlation (TC) has received limited systematic attention as a function of both task geometry and neural architecture. This study investigates whether statistical integration dynamics during supervised learning constitute a universal property of gradient-based optimization or depend on the interaction between task structure and architectural design. We trained feed-forward (MLP) and recurrent (RNN) networks on three binary classification tasks with distinct geometric properties XOR, Two Moons, and Spiral while tracking Gaussian TC across training epochs. Four experimental conditions were evaluated: normal training, random labels,untrained networks, and frozen features. Unlike previous studies that focused on single tasks or universal decorrelation hypotheses, the present work systematically compares multiple task–architecture combinations and proposes a unifying conceptual framework for interpreting integration dynamics. Three reproducible integration dynamics emerged across the experimental conditions. Monotonic Decay a consistent reduction in TC during training was observed under XOR across both architectures and under Spiral with the RNN, with strong negative Pearson correlations between accuracy and TC (r = −0.851 and r = −0.811 for XOR/MLP and XOR/RNN respectively, FDR p < 10−47). Stable Dynamics approximately constant TC despite improving accuracy were observed under Two Moons and Spiral with the MLP. Monotonic Growth a substantial increase in TC during successful learning was observed exclusively under Two Moons with the RNN (r = +0.329, FDR p = 2.85 × 10−6, TC increase of 36.1%), constituting an empirical counterexample to a universal decorrelation hypothesis. The contrast between Two Moons/MLP (Stable) and Two Moons/RNN (Monotonic Growth) identical task, opposite integration trajectories demonstrate that integration dynamics arise from the interaction between task geometry and architecture rather than from either factor independently. To organize these findings, we propose the Integration Dynamics Framework (IDF), a descriptive conceptual mapping from task geometry and architecture to observed TC trajectories, potentially mediated by a latent, currently unobservable representation organization strategy. The IDF generates testable predictions for future work across broader tasks and architecture spaces. These results suggest that statistical integration is a structured and informative dimension of representational change during learning one that varies systematically across conditions and cannot be reduced to a simple trade-off with task performance. More broadly, these findings suggest that understanding integration dynamics may provide a complementary perspective on neural network learning beyond predictive performance alone.
Author: Mohammed, ElSayed A.
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7222880702
Cites:
Cited by:
Related to: