PHOTON: Hierarchical Autoregressive Modeling for Lightspeed and Memory-Efficient Language Generation
Year: 2026
Type: conference-paper
Abstract: Transformers operate as horizontal token-bytoken scanners; at each generation step, attending to an ever-growing sequence of tokenlevel states.This access pattern increases prefill latency and makes long-context decoding more memory-bound, as KV-cache reads and writes dominate inference time over arithmetic operations.We propose Parallel Hierarchical Operation for TOp-down Networks (PHOTON), a hierarchical autoregressive model that replaces horizontal scanning with vertical, multi-resolution context scanning.PHOTON maintains a hierarchy of latent streams: a bottom-up encoder compresses tokens into low-rate contextual states, while lightweight top-down decoders reconstruct fine-grained token representations in parallel.We further introduce recursive generation that updates only the coarsest latent stream and eliminates bottom-up re-encoding.Experimental results show that PHOTON is superior to competitive Transformer-based language models regarding the throughput-quality tradeoff, providing advantages in long-context and multi-query tasks.In particular, this reduces decode-time KV-cache traffic, yielding up to 10 3 × higher throughput per unit memory.
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
Funder: RIKEN
OpenAlex ID: W7166809168