Work

Straggler-Resilient Personalized Federated Learning

Year: 2022

Type: preprint

Abstract: Federated Learning is an emerging learning paradigm that allows training models from samples distributed across a large network of clients while respecting privacy and communication restrictions. Despite its success, federated learning faces several challenges related to its decentralized nature. In this work, we develop a novel algorithmic procedure with theoretical speedup guarantees that simultaneously handles two of these hurdles, namely (i) data heterogeneity, i.e., data distributions can vary substantially across clients, and (ii) system heterogeneity, i.e., the computational power of the clients could differ significantly. Our method relies on ideas from representation learning theory to find a global common representation using all clients' data and learn a user-specific set of parameters leading to a personalized solution for each client. Furthermore, our method mitigates the effects of stragglers by adaptively selecting clients based on their computational characteristics and statistical significance, thus achieving, for the first time, near optimal sample complexity and provable logarithmic speedup. Experimental results support our theoretical findings showing the superiority of our method over alternative personalized federated schemes in system and data heterogeneous environments.

Source: arXiv (Cornell University)

DOI: 10.48550/arxiv.2206.02078

Authors: Tziotis, Isidoros, Shen, Zebang, Pedarsani, Ramtin, Hassani, Hamed, Mokhtari, Aryan

Institutions: The University of Texas at Austin, University of Pennsylvania, University of California, Santa Barbara

Language: English

Topic: Privacy-Preserving Technologies in Data

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: green

Funders: National Science Foundation, University of Texas at Austin, Air Force Office of Scientific Research, Army Research Office

Awards: G1523888516, AI Institute for Future Edge Networks and Distributed Intelligence (AI-EDGE), G36446433, CPS: Medium: Rethinking Communication and Control for Low-Latency, High Reliability loT Devices, G459827528, CIF: Small: Computationally Efficient Second-Order Optimization Algorithms for Large-Scale Learning, G5005386172, G5729145195, CAREER: Submodular Optimization in Complex Environments: Theory, Algorithms, and Applications, AI Institute: Institute for Foundations of Machine Learning, MLWiNS: Optimization and Coding Theory for Fast and Robust Wireless Distributed Learning

OpenAlex ID: W4281708275

Cites:

Cited by: 3

Related to: 10