Year: 2025
Type: conference-paper
Abstract: The inference of large language models (LLMs) on distributed edge devices is crucial for privacy-preserving applications. However, its performance is severely degraded in a practically lossy edge network due to frequent synchronization. In this paper, we propose Coda, a novel context-aware distributed acceleration framework tailored for packet loss scenarios. We observe the consistency of sparse patterns of neuron group activations in the same context. By identifying sparsity patterns during the prefill phase and updating neuron groups mapping strategically, we enable a relaxed synchronization for decoding, without slowing down, while preserving accuracy. Our approach offers up to 5.62x speedup, significantly outperforming the state-of-the-art in various scenarios.
Authors: Peirong Zheng, Wenchao Xu, Haozhao Wang, Jinyu Chen
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: bronze
OpenAlex ID: W4416479290