Scaling of Distributed Inference of BLOOMZ-560M Lightweight Large Language Models using Petals Tool : Analyzing Response Time, Resource Usage and Scaling Limits on CPU Infrastructure
Year: 2026
Type: dissertation
Abstract: Large Language Models (LLMs) increasingly serve as general-purpose inference services, yet their substantial resource requirements often preclude single-node deployment. Decentralised systems, such as Petals, address this by partitioning execution across peers. However, this converts a compute-bound workload into a distributed critical path constrained by computation, network communication, and orchestration overheads. This thesis characterises performance scaling and architectural bottle-necks in decentralised, RPC-Based inference. It quantifies how End-to-End Response Time, Generated Throughput, and Token-Normalised Efficiency evolve as active cluster size (N) scales from 1 to 10 nodes, determining how scaling differs across Sequential, Pipelined, and Microbatched client-side execution strategies. Experiments were conducted in a Local Area Network (LAN)-backed Azure Virtual Machine environment using Petals and the BLOOMZ-560M model. A fixed workload of 42,989 prompts was executed under a strict single-factor design, varying only N and the execution strategy. Performance was evaluated using client-side wall-clock measurements, system-level telemetry (Central Processing Unit (CPU), Random Access Memory (RAM), Network, Round Trip Time (RTT)), and validated Response Time decomposition to attribute bottlenecks to computation, communication, or orchestration. Scaling was strongly strategy-dependent. Pipelined execution achieved the highest gains, sustaining a mean speedup of ≈ 1.6× at N = 10, but with a 13× increase in network footprint. Sequential execution degraded monotonically withscale. Microbatched execution exhibited stability only at small cluster sizes (N ≤ 3). Critical path decomposition showed that model generation remained the dominant absolute cost; across N ∈ {1, . . . , 10} the marginal scaling penalty aligned consistently with communication growth, while orchestration overhead remained minimal. Decentralised inference on commodity CPUs is feasible, but scaling does not yield uniform improvements. The benefits of adding peers are diminished by communication costs unless high-occupancy strategies (e.g., Pipelined executionstrategy) are used. Deployment requires balancing throughput objectives against a rapid, multiplicative increase in network traffic, where communication overhead can grow faster than per-node compute relief.
Language: English
Subfield: Computer Networks and Communications
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7155475762