Work

Serving Large Language Models on Huawei CloudMatrix384

Year: 2025
Type: preprint
Abstract: The rapid evolution of large language models (LLMs), driven by growing parameter scales, adoption of mixture-of-experts (MoE) architectures, and expanding context lengths, imposes unprecedented demands on AI infrastructure. Traditional AI clusters face limitations in compute intensity, memory bandwidth, inter-chip communication, and latency, compounded by variable workloads and strict service-level objectives. Addressing these issues requires fundamentally redesigned hardware-software integration. This paper introduces Huawei CloudMatrix, a next-generation AI datacenter architecture, realized in the production-grade CloudMatrix384 supernode. It integrates 384 Ascend 910 NPUs and 192 Kunpeng CPUs interconnected via an ultra-high-bandwidth Unified Bus (UB) network, enabling direct all-to-all communication and dynamic pooling of resources. These features optimize performance for communication-intensive operations, such as large-scale MoE expert parallelism and distributed key-value cache access. To fully leverage CloudMatrix384, we propose CloudMatrix-Infer, an advanced LLM serving solution incorporating three core innovations: a peer-to-peer serving architecture that independently scales prefill, decode, and caching; a large-scale expert parallelism strategy supporting EP320 via efficient UB-based token dispatch; and hardware-aware optimizations including specialized operators, microbatch-based pipelining, and INT8 quantization. Evaluation with the DeepSeek-R1 model shows CloudMatrix-Infer achieves state-of-the-art efficiency: prefill throughput of 6,688 tokens/s per NPU and decode throughput of 1,943 tokens/s per NPU (<50 ms TPOT). It effectively balances throughput and latency, sustaining 538 tokens/s per NPU even under stringent 15 ms latency constraints, while INT8 quantization maintains model accuracy across benchmarks.
Authors: Zuo, Pengfei, Lin, Huimin, Deng, Junbo, Zou, Nan, Yang, Xingkun, Diao, Yingyu, Gao, Weifeng, Xu, Ke, Chen, Zhangyu, Lu, Shirui, Qiu, Zhao, Li, Peiyang, Chang, Xianyu, Yu, Zhengzhong, Miao, Fangzheng, Zheng, Jia, Li, Ying, Feng, Yuan, Wang, Bei, Zong, Zaijian, Zhou, Mosong, Zhou, Wenli, Chen, Houjiang, Liao, Xingyu, Li, Yipeng, Zhang, Wenxiao, Zhu, Ping, Wang, Yinggang, Xiao, Chuanjie, Liang, Depeng, Cao, Dong, Liu, Juncheng, Yang, Yongqiang, Bai, Xiaolong, Li, Yi, Xie, Huaguo, Wu, Huatao, Yu, Zhibin, Chen, Lv, Liu, Hu, Ding, Yujun, Zhu, Haipei, Xia, Jing, Xiong, Yi, Yu, Zhou, Liao, Heng
Language: English
Open Access status: green
OpenAlex ID: W4417536562
Cites: 18
Cited by: 67
Related to: 10