Parameter-Level Conditional Computation Does Not Buy Quality on Phones: Three Negative Results for On-Device LLMs
Year: 2026
Type: article
Abstract: Token generation on phones is bound by memory bandwidth, so the quality a phone can serve is setby how many parameter bytes it can afford to read per token. A tempting way around this limit isconditional computation at the parameter level: use a cheap signal about the request (its task) to keeponly the parameters that request needs in RAM, or to load a small task-specific module. We test thisidea three ways on an Apple M4 Max, each against a fair dense or shared baseline. (1) Expert usagein Qwen3-30B-A3B is genuinely task-local (a task-predicted resident set of 32 of 128 experts per layerserves 64% of decode-time expert reads, versus 49% for a task-agnostic set), but the remaining missesstill cost about 210 MB of flash reads per token at a 7 GB budget. (2) Routing only among resident expertsremoves those reads but collapses generation quality long before phone budgets (GSM8K 94% →10%at 7 GB of resident weights), while a dense 8B model in 4.6 GB scores 89%; seven training-free rescuevariants do not help. (3) A dense 4B core with per-task LoRA modules chosen per request (by oracle orby an embedding router) is within noise of a single shared multi-task LoRA on all four evaluations, atequal per-request cost and at equal storage. Along the way we find that fine-tuning on dataset referenceanswers damages a strong instruction-tuned core on every task, while self-distilled data does not, and thatsentence-embedding routing misroutes 27% of DROP questions. Per-request selection appears necessaryonly for state that a shared model cannot hold, such as an individual user’s data. All code, data, per-itemscores and generations are released.
Author: Ul Haq, Mohammed Wasif
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7221777841