Token Selection Without Matrix Multiplication: Experimental Evidence for Retrieval-Based Inference
Year: 2026
Type: preprint
Abstract: We present experimental evidence that token selection in large language models can be performed wit hout computing the full transformer forward pass. By replacing softmax-based token selection with a retrieval-and-voting mechanism, we achieve 91-100% offline accuracy and a sustained 99% non-BOS to ken rate in standalone inference at 671 tokens per second on a consumer-grade Mac Studio. Our resul ts challenge the default assumption that inference quality is inseparable from matrix multiplicatio n.
Author: 韩, 文国
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7162785207
Cites:
Cited by:
Related to: