Year: 2025
Type: conference-paper
Abstract: Large language models (LLMs) achieve strong performance but suffer from slow and costly inference.Existing acceleration methods often lead to noticeable performance degradation, while Mixture-of-Experts (MoE) models require extensive computational resources.In this paper, we propose L0-MoE, a lightweight MoE approach using L0-regularization to accelerate dense LLMs nearly without performance loss.Our method introduces a cluster confusion matrix for domain-aware dataset curation and applies dynamic batching for efficient training.Experiments show that L0-MoE achieves up to 2.5x speedup over dense models while maintaining competitive performance, outperforming existing LLM acceleration baselines.
Authors: Zhenyu Zhang, JiuDong Yang, Taozhaowen Taozhaowen, Meng Chen
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W4412889498