Work

Hybrid Zeroth- and First-Order Split Federated Learning with Dimension-Free Convergence

Year: 2026
Type: conference-paper
Abstract: Split federated learning (SFL) trains models by exchanging smashed activations, so that resource-constrained clients may only need to train a small portion of a large neural network. However, conventional SFL still relies on client backpropagation computation and activation storage, which may exceed its capabilities. We develop a hybrid zeroth- and first-order training paradigm: clients perform forward-only zeroth-order updates for the encoder, while the server updates the remaining modules using first-order gradients. Perturbations are generated locally without additional smashed-activation transmissions, and client peak memory is reduced to constant order beyond model parameters. Leveraging the low effective rank assumption, we theoretically prove that HERON-SFL’s convergence rate is independent of model dimensionality, addressing a key scalability concern common to ZO algorithms.
Language: English
Open Access status: hybrid
OpenAlex ID: W7204193159
FWCI:
Cites: 17
Cited by:
Related to: 10