Foundations of Sparse Mixture-of-Experts Routing - Part I
Year: 2026
Type: report
Abstract: Modern Mixture-of-Experts (MoE) architectures achieve substantial parameter scaling without incurring proportional compute costs by sparsely routing tokens to specialized subnetworks. However, standard training regimes frequently suffer from structural imbalances and routing collapse, driven by greedy short-term utility maximization. This technical note serves as the foundational first installment in a series establishing a rigorous control-theoretic and optimization-based framework for MoE routing. Rather than relying on heuristic penalty terms, we formulate expert load balancing as a Stochastic Resource Allocation Program over an infinite horizon, bridging the ideal mathematical problem formulation with the empirical constraints of mini-batch deep learning training loops.
Author: Rehn, Carl Johan
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7234114498
