PlanetServe: A Decentralized, Scalable, and Privacy-Preserving Overlay for Democratizing Large Language Model Serving
Year: 2025
Type: preprint
Abstract: While significant progress has been made in research and development on open-source and cost-efficient large-language models (LLMs), serving scalability remains a critical challenge, particularly for small organizations and individuals seeking to deploy and test their LLM innovations. Inspired by peer-to-peer networks that leverage decentralized overlay nodes to increase throughput and availability, we propose GenTorrent, an LLM serving overlay that harnesses computing resources from decentralized contributors. We identify four key research problems inherent to enabling such a decentralized infrastructure: 1) overlay network organization; 2) LLM communication privacy; 3) overlay forwarding for resource efficiency; and 4) verification of serving quality. This work presents the first systematic study of these fundamental problems in the context of decentralized LLM serving. Evaluation results from a prototype implemented on a set of decentralized nodes demonstrate that GenTorrent achieves a latency reduction of over 50% compared to the baseline design without overlay forwarding. Furthermore, the security features introduce minimal overhead to serving latency and throughput. We believe this work pioneers a new direction for democratizing and scaling future AI serving capabilities.
Source: arXiv (Cornell University)
Authors: Fang, Fei, Hua, Yifan, Wang, Shengze, Zhou, Ruilin, Liu, Yi, Qian, Chen, Zhang, Xiaoxue
Language: English
Subfield: Computer Networks and Communications
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
Funder: National Science Foundation
Awards: NeTS: SMALL: Develop Core Techniques and Applications of Learned and Disaggregated In-Network Lookups, CNS Core: Small: Core Scheduling Techniques and Programming Abstractions for Scalable Serverless Edge Computing Engine, Collaborative Research: NeTS: Small: Resource-efficient Consolidated Data Plane for Network Functions, ExpandQISE: Track 1: Education and Research of System and Network Supports for Quantum Cloud
OpenAlex ID: W4416679339