Work

Quaternion-Tucker ZigZag Networks: Halving Transformer Backbone Parameters via Shared Hypercomplex Tensor Cores and Braided Top-Down Routing

Year: 2026
Type: preprint
Abstract: Standard Transformer architectures allocate independent, dense two-dimensional weight matrices to every feed-forward network (FFN) layer, coupling parameter count linearly with depth O(L). Conversely, weight-tied Looped Transformers decouple depth from parameters O(1) but enforce time-homogeneous updates across iterations. In this preprint, we formulate the L-layer FFN stack as a unified 3D Tensor Cuboid and introduce QuatTucker-ZigZagGPT, unifying three orthogonal mechanisms:1. 4D Quaternion Hamilton Algebra (H) for hypercomplex feature projections compiled into a single fused bfloat16 Tensor-Core GEMM (4x FFN compression).2. Block-Diagonal Tucker SwiGLU Cores enabling each depth slice to execute distinct 64x64 subspace rotations over a single shared high-rank feature basis.3. Braided ZigZag Diamond Routing (L0 -> L2 -> L1(h0 + h2) -> L3(h1 + h2)), introducing zero-FLOP, zero-parameter coarse-to-fine top-down refinement. We also introduce Annealed Gated Bounce-Back Scaffolding (GatedBounce), a sparsity-taxed training-time feedback path that self-distills to 0.0% inference overhead. Key Empirical Results:- FineWeb-Edu (GPT-2 BPE, NVIDIA L4, 3,000 steps): QuatTucker-ZigZagGPT-8 (26.5M total / 7.2M backbone params) achieves 5.1339 validation loss (169.68 PPL), outperforming a 33.5M (14.3M backbone) FlashAttention StandardGPT-8 baseline (5.2310 / 186.98 PPL) by -0.0971 loss (-17.30 PPL) with half the backbone parameters.- TinyStories (Tesla T4, 1.84M iso-params): CP-ZigZagLM achieves 2.48 PPL vs. 2.54 PPL for StandardTransformer-4.- Multi-Domain Compression: CP-CuboidNet achieves 92.2% on 8-bit Parity (3.2x smaller than ResNet4), 100.0% monotonic grokking on Mod-53 Addition (7.6x smaller than ResNet-8), 96.94% on Handwritten Digits (3.1x smaller), and 96.00% on 3-Turn Double Spirals (3.5x smaller).- Negative Results: We empirically establish that while FFNs thrive under Quaternion-Tucker sharing, attention projections must remain dense and independent per layer.
Language: English
Open Access status: green
OpenAlex ID: W7237347696
Cites:
Cited by:
Related to: