Pyrite Phase 2: Decoupled Sparse Memory Hierarchy for 100B+ Mixture-of-Experts on Consumer Unified Memory
Year: 2026
Type: preprint
Abstract: Mixture-of-Experts (MoE) architectures such as Mixtral-8x22B (141B parameters) and DeepSeek-Coder-V2-Lite (15.7B parameters, 64 experts) achieve state-of-the-art capability, yet their massive footprint (70 GB to 280 GB) has restricted local deployment to high-end multi-GPU clusters or workstation-class hardware with 128 GB to 192 GB of unified memory. We present Pyrite Phase 2 (MoE-Stream), an execution architecture that decouples the model's static resident working set from dynamic routed feed-forward experts. By pinning the non-MoE working set (Multi-Head Latent Attention, LayerNorms, Router Gates, and Token Embeddings) into Unified Memory and streaming active Top-K routed experts on-demand from NVMe SSD storage via zero-copy linear byte-slicing (68 microseconds latency) and an in-memory Least-Recently-Used (LRU) cache, MoE-Stream achieves unprecedented memory efficiency on standard consumer hardware. On a single 16 GB Apple M2 Pro, DeepSeek-Coder-V2-Lite operates within 1.55 GB to 2.01 GB of RAM at 2.83 tokens/s (an 83% to 93% memory reduction). Furthermore, we demonstrate the successful 56-layer forward pass execution of the 141-Billion parameter Mixtral-8x22B within 2.84 GB to 3.61 GB of RAM, leaving over 12 GB of physical memory available for multitasking. Finally, we discuss the theoretical implications of this decoupled hierarchy, showing that discrete integer quantization is no longer mandatory for consumer deployment, unlocking true unquantized BF16 execution and lossless floating-point streaming.
Author: Nakatsugawa, Masahiro
Language: English
Subfield: Hardware and Architecture
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7224897913