Work

Union-of-Experts: Neurons in Mixture-of-Experts are Secretly Routers

Year: 2026

Type: conference-paper

Abstract: Mixture-of-Experts (MoE) models rely on an external router to assign tokens to experts.This design inherently separates the routing decision from each expert's internal capabilities, leading to suboptimal performance.In this work, we address this limitation with Union-of-Experts (UoE), an MoE variant that performs "expertautonomous routing".The core mechanism of UoE is to pre-designate a minute fraction of neurons within each expert as routing neurons.Experts autonomously select relevant tokens by comparing the activation intensity of these neurons, aligning routing decisions with each expert's functional profile.To prevent the waste of activations from unselected experts, we aggregate all routing neuron outputs and sum them into the final layer output.This aggregation acts as a novel virtual shared expert whose parameters are distributed across the individual experts, and improves overall parameter efficiency.We pre-train UoE models with up to 3B parameters, demonstrating that they outperform traditional MoEs with matched efficiency.Furthermore, our analysis of the routing neurons provides valuable insights into expert-autonomous selection and advances the understanding of MoE routing.

Source: Proceedings of the conference - Association for Computational Linguistics. Meeting

DOI: 10.18653/v1/2026.acl-long.1675