Work

Methods for Improving Computation and Communication Efficiency in Distributed Computing Algorithms

Year: 2024
Type: dissertation
Abstract: Distributed processing of data on large compute clusters has proven itself to be a practical and efficient approach for addressing today's massive computational tasks. Many challenges inherent to algorithms running on distributed systems are intricately linked to the limitations and deficiencies of compute nodes and the communication network that connects them. In this thesis, we develop techniques that address issues arising from such limitations. We investigate two important challenges that lead to many interesting research problems: variability of processing speeds of compute nodes and limited bandwidth in the communication networks. The former results in lags in task completion for some workers, while the latter leads to delays in transmitting large payloads between workers. Overall, both these factors contribute towards extending running times of algorithms. In the first half of the thesis, we focus on developing methods to cope with variability of processing speeds of compute nodes. In large compute clusters, the unreliability of compute hardware and the sharing of resources sometimes create nodes with slower processing speeds, also known as stragglers. While various techniques exist for mitigating the effects of stragglers, most methods are focused on increasing redundancy of compute tasks. These methods effectively take computations performed on slower nodes and repeat them on faster nodes. In contrast, we investigate methods that exploit the work performed on slower nodes by taking into account the structure and characteristics of the executed workloads. We develop straggler mitigation techniques tailored to statistical optimization workloads, as well as other general computations. In the second half of the thesis, we focus on the issues arising due to bandwidth limitations in communication networks that connect compute nodes. First, we investigate algorithms that reduce the communication payload in statistical optimization problems, e.g., optimizing large machine learning models with stochastic gradient descent. By exploiting the structure in messages exchanged between compute nodes, we develop compression methods that reduce the payload size, thus leading to more efficient use of network resources. Second, we concentrate on a source coding problem abstracted from a model motivated by statistical optimization. We develop information-theoretic bounds on this source coding problem, while connecting it to many classic information theory problems such as Wyner and Ziv's problem of source coding with side information.
Language: English
Open Access status: green
OpenAlex ID: W7132873515
Cites: 79
Cited by:
Related to: 9