Mixture of Scars (MoS) and SCache O(1): Breaking the Dual Memory Wall of Modern Transformers via Activation-Aware Low-Rank Scar Experts and Widrow-Hoff Associative Memory
Year: 2026
Type: preprint
Abstract: Autoregressive Large Language Models (LLMs) face a compounding Dual Memory Wall: (1) a static parameter wall O(N) requiring tens to hundreds of gigabytes of VRAM to host frontier weights, and (2) a dynamic context wall O(T) where key-value (KV) caches grow linearly with sequence length T. We introduce two synergistic architectural primitives that simultaneously break both walls without retraining from scratch: Mixture of Scars (MoS), which decomposes dense feed-forward and attention projections into INT4 quantized bases paired with activation-aware SVD (ASVD) low-rank residual "scar" experts and sequential layer-group streaming; and SCache (Scar-Cache), an exact O(1) constant-space dynamic memory combining attention sinks, a compact local KV window, and per-KV-head associative fast-weight slots updated via the orthogonal Widrow-Hoff Delta Rule with ReLU manifold gating. To prevent benchmark facades, we introduce a 7-vector Adversarial Red-Team Falsification Suite enforcing recursive PyTorch storage-pointer byte audits (0.00 KB discrepancy) and in-flight active causality ablations across real pretrained LLMs (Qwen2.5-0.5B-Instruct 24L, Qwen2.5-72B-Instruct 80L, and Mistral Large 2 123B). Empirically, MoS reduces single-model MLP RAM by 68.6%, cuts 8-specialist multi-tenant VRAM by 92.6% (13.58x), reduces out-of-domain logit KL divergence by 2.45x-4.22x over pure INT4, and streams 72B/123B backbones within < 3.8 GB peak RAM. Simultaneously, SCache bounds dynamic memory to strictly 0.00 marginal bytes beyond window saturation (-91.4% RAM at L=1,280), eliminates Hebbian contradictory ghosting (cosine 1.00000 vs. 0.467), achieves 22.58x higher exact probability on evicted needles than StreamingLLM with zero fine-tuning, and preserves full-stack QA accuracy.
Author: Reis Filho, Julio Cesar
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7225787920
Cites:
Cited by:
Related to: