thox-edge-ops v2: fused decode kernels for small models on THOX edge hardware
Year: 2026
Type: preprint
Abstract: Small language models on edge hardware spend their decode step in launch overhead and memory traffic, not arithmetic. thox-edge-ops v2 is a set of six fused inference operators (RMSNorm with a weight offset that covers the Llama and Gemma conventions, fused residual-add RMSNorm, SiLU and tanh-GELU gated activations, rotary embedding, and a fused per-head QK-norm plus RoPE prologue) with one source tree and three backends: CUDA, Metal and CPU, distributed through the Hugging Face Kernel Hub as 23 ABI-checked x86_64 builds, with the full 438-test suite passing against 3 builds downloaded from the Hub. At decode shapes the GPU kernels run 6.90x faster than eager PyTorch for RMSNorm and 24.43x faster for the fused QK-norm and RoPE prologue on an NVIDIA RTX 4060 Ti; on the CPU, RMSNorm runs 2.42x and rotary embedding 6.75x faster. The revised public edition adds a deployment section covering the THOX stack (edge hardware, ThoxOS and ThoxInux, MeshStack, the kernels, the small models) and a hardware qualification matrix: the x86_64 workstation and hub class is MEASURED for v2; Jetson Orin, Apple silicon and the Arm device class (ThoxAir, ThoxMini, ThoxClip) are targets only, with no aarch64 Hub build published for v2 and no Arm measurement claimed. Preprint, not peer reviewed. THOX.ai Research.
Author: Xaypanya, Tommy
Language: English
Subfield: Hardware and Architecture
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7225326490