Work

Deterministic Logical Time for Fault-Tolerant Distributed LLM Training

Year: 2026
Type: article
Abstract: Physical clock synchronization is an under-examined infrastructure constraint in large-scale distributed LLM training. Both synchronous and asynchronous training paradigms rely on physical wall-clock alignment to decide gradient aggregation timing, parameter-version acceptance, and failed-node replacement. This creates an inherent tension: synchronous training stalls on the slowest worker, while asynchronous training trades convergence guarantees for throughput and lacks a cluster-wide, independently verifiable timing baseline. This working paper proposes a deterministic logical time axis decoupled from physical clocks, in two layers. The single-domain design anchors a training step to the logical tick boundary g_step_start mod N_train = 0, defines the aggregation deadline as a fixed logical offset g_step_start + Δn_deadline, derives a parameter-version window from the logical staleness bound to filter stale gradients, and achieves fault recovery via deterministic election and candidate-window realignment. The multi-domain extension, presented as a design draft, maintains an independent local logical axis and DSF/PSSM instance for each heterogeneous training domain, aligns domains to a global reference axis through mapping rules, and provides cross-domain aggregation anchors, domain-weight scaling, placeholder updates, QoS-driven window configuration, cross-domain quorum, and degradation mechanisms. Scope and evidence. The contribution is problem formalization, single-domain timing-correctness simulation, and an open validation protocol, not established training throughput or convergence advantages. Single-domain results come from a single-machine discrete-event simulator and cryptographic microbenchmarks. The simulation verifies timing logic only; it does not include GPU training or neural-network convergence. The multi-domain part is a design draft, argued for self-consistency only and not simulation-verified. Open call for validation. The paper defines six validation challenges, each with a failure criterion so that negative results are equally informative. Challenge A: placeholder-induced convergence deviation, executable on 8 to 16 GPUs. Challenge B: recovery bound under real fault injection, executable on 8 to 16 nodes. Challenge C: behavior-chain auditability, with third-party verification. Challenge D: cross-region beacon overhead, using two regions with measured RTT. Challenge E: cross-domain mapping and Token closure, executable on a single machine with multiple processes. Challenge F: cross-domain aggregation semantics, requiring real WAN and multi-party cooperation. Independent replication, falsification, and critique are welcome, including negative results. A suggested validation report template is included in the appendix. Licensing and patents. The document is released under CC BY 4.0. This license applies to the document copyright only and grants no patent license; patent application rights, patent rights, and all related rights are reserved by the rights holders. Academic reproduction, critique, and open-source implementation are permitted for research and validation purposes and do not constitute a patent license. Two pending patent applications cover mechanisms described in this work; the authors are primary inventors. Companion documents. Zenodo 23293794, Cryptographic-Driven Deterministic Logical Timeline: A Temporal Baseline for Machine Collaboration and AI Auditing, v2.0. Zenodo 23271917, Dual-timestamp binding and trigger-mode baseline, v3.0. This version adds a Chinese-language PDF of the same working paper. Both language versions share the same concept DOI. Correspondence: [[email protected]]
Language: English
Open Access status: green
OpenAlex ID: W7233580130
FWCI:
Cites:
Cited by:
Related to: