All is Not Lost: LLM Recovery without Checkpoints
Year: 2026
Type: conference-paper
Abstract: Training Large Language Models on decentralized nodes, e.g., on-spot instances, lowers the training cost and enables model democratization. The inevitable challenge here is the transient churns of nodes due to failures and the operator's scheduling policies, leading to losing parts of the model (some layers). The conventional approaches to recover from failures is to either use checkpointing, where periodically a copy of the entire model is sent to an additional storage, or redundant computation. These approaches yield significant communication and/or computation overhead even in non-failure cases and scale poorly in settings with large models.
Institution: University of Neuchâtel
Language: English
Subfield: Computer Networks and Communications
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
OpenAlex ID: W4415333939