Predictive Just in Time Telemetry Streaming (P-JITS) with Hybrid Hierarchical Storage for Frontier MoE Inference on Consumer Grade Hardware
Year: 2026
Type: article
Abstract: Large Language Models utilizing Mixture-of-Experts (MoE) architectures, such as Gemma4-26B-A4B, provide state-of-the-art reasoning capabilities but impose severe hardware constraints, demanding high VRAM capacities and substantial storage footprints. This paperintroduces Predictive Just-In-Time Telemetry Streaming (P-JITS), a novel, asymmetric, distributed inference paradigm designed for highly resource-constrained client environments,such as an NVIDIA RTX 3050 with 4GB of VRAM and standard NVMe or HDD storage.P-JITS partitions the neural network into an immutable local base skeleton and a hierarchicalexpert hierarchy spanning Cloudflare R2 cold-object storage, local NVMe raw blocks, pinnedhost RAM, and a static VRAM ring buffer. The architecture is extended with Pre-ExecutionSemantic Gating (PESG), or macro-routing, which classifies the entire user prompt beforedecoding and pre-loads the domain-specific expert cluster most likely to be used during the turn.This paper further introduces an optional Mixture of Optimized Experts (MOOE) mode:a hard-switching MoE variant that activates exactly one expert or adapter at a time, reducingthe live inference footprint to the base model plus the largest selected expert. Within that preloaded sandbox, an early-layer Speculative Linear Probe still forecasts fine-grained routingrequirements N + 2 tokens in advance. The updated system removes active cloud dependencyfrom the generation loop: cloud storage is used only for initialization, synchronization, anddelta updates, while inference relies on local Direct I/O from NVMe into locked host memoryand asynchronous PCIe DMA into VRAM. P-JITS aims to enable full-parameter frontier MoEexecution on a 4GB GPU while preserving offline operation and reducing cloud egress costs toa flat object-storage model.
Author: Shukla, Dwij
Language: English
Subfield: Hardware and Architecture
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7168916116
FWCI:
Cites:
Cited by:
Related to: