Revenge of Monosemanticity: Neuron Specialization as a New Form of Feature Learning in MLPs
Year: 2026
Type: preprint
Abstract: Understanding how neural networks learn and organize features is central to understanding their behavior. Much existing theory of feature learning has focused on the emergence of a global low-dimensional representation. We show that this picture is incomplete. In regression problems with clustered data, we demonstrate that multilayer perceptrons (MLPs) develop specialized neurons: individual neurons become strongly aligned with specific features relevant to a particular region of the input space. A substantial fraction of these neurons exhibit monosemanticity: they activate predominantly on inputs from the corresponding cluster. Thus, rather than learning a single global low-dimensional representation, MLPs learn a collection of local low-dimensional representations. Furthermore, we show that this ability to specialize gives MLPs a provable data-efficiency advantage over feature-learning methods based on a global low-dimensional representation.
Source: arXiv (Cornell University)
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
Awards: G1453181110, Track 4: Advanced CI Coordination Ecosystem: Monitoring and Measurement Services, G3562386289, G3650812695, AI Institute for Learning-Enabled Optimization at Scale (TILOS), Track 1: ACCESS Resource Allocations Marketplace and Platform Services (RAMPS), G4010157413, ACO: An Open CI Ecosystem to Advance Scientific Discovery (OpenCI), Track 3: COre National Ecosystem for CyberinfrasTructure (CONECT), G5367127336, G5464919783, Collaborative Research: MFAI: Emergence of features in modern machine learning, G748459979, Track 2: Customized Multi-tier Assistance, Training, and Computational Help (MATCH) for End User ACCESS to CI
OpenAlex ID: W7204238824