Get started
Work

Anudhi: A Fast Inference Engine for Edge Language Models

Year: 2026
Type: preprint
Abstract: Running a language model on a phone is constrained by four things that a datacenter runtime does not face together: decode speed is set by memory bandwidth and not by arithmetic, the operating system kills processes that exceed a memory budget, sustained throughput falls when the device heats up, and the platform forbids generating code at run time. Anudhi is an inference engine designed around those four constraints. A model is compiled offline into a single .kali container that carries its weights in consumption order, a flat execution plan with no control flow, a pre-computed memory plan and an Ed25519 signature. On the device a small executor replays that plan once per token with no allocation after session creation. Weight-bearing operations use fused kernels that multiply 4-bit weights against 32-bit floating-point activations without ever materializing dequantized weights. The key-value cache is typed, mutable state with ring and paged layouts and 8-bit and 4-bit block formats. Speculative decoding with exact rejection sampling raises tokens per weight pass, and a thermal governor degrades the configuration in a fixed, reversible order when the device throttles. Every backend segment falls back from neural accelerator to GPU to CPU at load time, and a load-time numerical check refuses any backend that disagrees with the reference. We describe the complete design, the C interface, and the verification harness that gates every kernel against a scalar reference and an independent oracle. On the reference host the plan executor reproduces oracle logits to 1.2 x 10^-6 maximum absolute error on both CPU tiers, greedy generation through the public interface matches an independent executor on every token tested, and the fused AVX2 kernel runs 8.0 times faster than the scalar tier. Device-class throughput is governed by the release gates stated in Section 12; we report no device throughput figures in this paper. Code is available at https://github.com/KalarisLabs/Anudhi.
DOI: 10.5281/zenodo.24144821
Language: English
Open Access status: green
OpenAlex ID: W7225823282
Cites:
Cited by:
Related to: