Coordinating Momenta for Cross-Silo Federated Learning
Year: 2022
Type: conference-paper
Abstract: Communication efficiency is crucial for federated learning (FL). Conducting local training steps in clients to reduce the communication frequency between clients and the server is a common method to address this issue. However, it leads to the client drift problem due to non-i.i.d. data distributions in different clients which severely deteriorates the performance. In this work, we propose a new method to improve the training performance in cross-silo FL via maintaining double momentum buffers. One momentum buffer tracks the server model updating direction, and the other tracks the local model updating direction. Moreover, we introduce a novel momentum fusion technique to coordinate the server and local momentum buffers. We also provide the first theoretical convergence analysis involving both the server and local standard momentum SGD. Extensive deep FL experimental results show a better training performance than FedAvg and existing standard momentum SGD variants.
Authors: An Xu, Heng Huang
Institution: University of Pittsburgh
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: diamond
Funder: National Science Foundation
Awards: BIGDATA: Collaborative Research: IA: Big Imaging-Omics Data Mining Framework for Precision Medicine, SCH: INT: New Machine Learning Framework to Conduct Anesthesia Risk Stratification and Decision Support for Precision Health, III: Small: Robust Large-Scale Data Mining for Knowledge Discovery in Depression Thought Records, BIGDATA: IA: Collaborative Research: Asynchronous Distributed Machine Learning Framework for Multi-Site Collaborative Brain Big Data Mining, III: Medium: Collaborative Research: Integrating Large-Scale Machine Learning and Edge Computing for Collaborative Autonomous Vehicles, NSF Convergence Accelerator Track D: A Trusted Integrative Model and Data Sharing Platform for Accelerating AI-Driven Health Innovation
OpenAlex ID: W4283817857