Coded Distributed Computing Scheme with Master-Aided Compression
Year: 2025
Type: conference-paper
Abstract: Li et al. proposed coded distributed computing (CDC) to reduce the communication load between servers by increasing the computational load of each server. They have shown that this scheme achieves the fundamental trade-off between computational and communication load. However, this scheme not only reduces communication load, but also creates the problem of requiring too many input files and output functions, In this paper, we propose a new scheme to solve this problem. In this scheme, we introduce compression technology into the main auxiliary encoding distributed computing, where edge nodes and main nodes use compression technology to compress the intermediate values of several identical functions into pre combined values, and then use encoding technology to encode the pre combined values of different functions and create encoded multicast messages. Finally, the encoded messages are multicast to the remaining edge nodes. Research has shown that our new approach significantly reduces the communication load in the Shuffle phase of distributed computing compared to the original approach.
Author: Shengpeng Ji
Institution: Shandong Xiehe University
Language: English
Subfield: Information Systems
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
OpenAlex ID: W7138911210