Work

Federated Low-Rank Adaptation of Foundation Models: Representation, Ownership, Transformation, and Cross-Round Continuity

Year: 2026

Type: article

Abstract: Low-rank adaptation (LoRA) reduces training and communication costs in federated fine-tuning of foundation models, including large language models (LLMs). However, a given effective weight update generally admits multiple low-rank factorizations. Shared states, private states, residuals, backbone models, and optimizer states may also be transformed during aggregation, refactorization, merging, and client re-entry. These transformations create state-consistency challenges across clients, rounds, and model versions that cannot be fully characterized by parameter averaging or communication volume alone. This survey reviews federated LoRA from a state-lifecycle perspective. It maps existing methods across state representation, ownership, transformation, cross-round continuity, and correctness, while comparing factor-based, effective weight-update, subspace, and composite representations, together with states used to support training and recovery. It also derives exact first-moment transport under fixed invertible factor reparameterization, analyzes the limits of diagonal second-moment migration, and distinguishes four outcomes of version changes: direct reuse, migration, reset, and rejection. The synthesis shows that trainable, uploaded, aggregated, and persistent states must be defined separately; aggregation in factor space is generally not equivalent to aggregation in weight-update space; and refactorization, backbone merging, and structural changes require explicit version management and state-migration rules. Building on these findings, this survey presents an evaluation checklist and research agenda for transformation fidelity, offline recovery, optimizer migration, and long-term storage, providing guidance for federated foundation-model adaptation across diverse deployment settings, including future sixth-generation (6G) edge-intelligence systems.

Source: Information

DOI: 10.3390/info17111127

Authors: Rongping Zhang, Kai Wu, Xin Yuan, Wei Ni, J. Andrew Zhang, Ren Ping Liu, Y. Jay Guo

Institutions: University of Technology Sydney, Commonwealth Scientific and Industrial Research Organisation, UNSW Sydney, Data61, Edith Cowan University

Language: English

Topic: Privacy-Preserving Technologies in Data

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: gold

Estimated APC paid: $1,930

OpenAlex ID: W7225733914

FWCI:

Cites:

Cited by:

Related to: