A Survey on Side-Tuning for Foundation Models
Year: 2026
Type: preprint
Abstract: The adaptation cost of parameter-efficient fine-tuning (PEFT) depends on trainable parameters, activation storage, and the computations required for optimization. Methods such as LoRA and adapters may still require backward computation through frozen backbone operations and retention or recomputation of intermediate activations. Side-tuning places trainable adaptation modules outside the backbone, creating opportunities to reduce this cost through different connections to the backbone gradient graph. This survey presents a topology- and deployment-aware study of side-tuning for foundation models and proposes a graph-topological definition with three categories: graph-coupled side-tuning, in which an external side branch remains coupled to backbone gradients; mixed side-tuning, in which a detached side path re-enters a later layer of the same backbone; and detached side-tuning, in which an external trainable network consumes frozen backbone representations through a stop-gradient interface and computes task-specific gradients entirely outside the backbone. We review a curated cross-domain corpus spanning natural language processing, computer vision, multimodal learning, recommendation, and relevant methodological precursors, while examining information retrieval as an emerging deployment setting. The literature is organized along three axes: gradient topology, feature interface, and deployment regime. We derive a resource model that accounts for backward activations, online feature production, and cached representations, and use it to develop design guidelines and reporting recommendations. This organization connects the training graph of each method to its resource requirements and the conditions under which its features can be reused.
Source: Preprints.org
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7223115960
Cites:
Cited by:
Related to: