Hybrid Zeroth- and First-Order Split Federated Learning with Dimension-Free Convergence
Year: 2026
Type: conference-paper
Abstract: Split federated learning (SFL) trains models by exchanging smashed activations, so that resource-constrained clients may only need to train a small portion of a large neural network. However, conventional SFL still relies on client backpropagation computation and activation storage, which may exceed its capabilities. We develop a hybrid zeroth- and first-order training paradigm: clients perform forward-only zeroth-order updates for the encoder, while the server updates the remaining modules using first-order gradients. Perturbations are generated locally without additional smashed-activation transmissions, and client peak memory is reduced to constant order beyond model parameters. Leveraging the low effective rank assumption, we theoretically prove that HERON-SFL’s convergence rate is independent of model dimensionality, addressing a key scalability concern common to ZO algorithms.
Authors: Zhoubin Kou, Zihan Chen, Jing Yang, Cong Shen
Institution: University of Virginia
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W7204193159
