Pipit: A Placement-First Inference Engine for Frontier Mixture-of-Experts Models on Consumer Hardware
Year: 2026
Type: preprint
Abstract: Mixture-of-experts (MoE) language models with hundreds of billions of parameters activate only a few per cent of their weights for each token. That makes them runnable on machines with far less memory than the model, provided the right expert weights sit in the right memory tier at the right moment. Today's local engines each solve part of this. Colibri streams experts from SSD but relies on general-purpose kernels and has repeatedly patched memory-safety defects in its loaders; uzu reaches strong speculative-decoding speed on Apple Silicon but is not yet stable on other operating systems; Magnitude tunes kernels per device and connects to coding agents, though its headline speed-ups rest on comparisons with mismatched settings. This report presents the design of Pipit, a memory-safe Rust engine that treats decoding as a bandwidth problem across three tiers (GPU memory, RAM and NVMe). It combines a slot-slab expert store, a learned expert predictor, bandwidth-saturating CPU kernels for six instruction-set levels, union-aware speculative decoding, and validated loaders for GGUF, safetensors and a new streaming format, KXF. We pre-register ten falsifiable hypotheses against same-machine baselines, together with a protocol that scores every run against its physical roofline. The report contains no measured Pipit results: it fixes the claims in advance, so that forthcoming results can be judged against them.
Author: Chowdhury, Sayan
Language: English
Subfield: Hardware and Architecture
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7225820708
Cites:
Cited by:
Related to: