Evaluating Quantized Language Model Inference a Critical Review of Quality Memory and Workload Tradeoffs
Year: 2026
Type: preprint
Abstract: Quantization reduces the numerical precision of large language model weights, activations, or key-value caches, but its practical value depends on the interaction among numerical error, workload, and execution hardware. This review compares representative methods through those three dimensions. A purposive literature synthesis covers integer inference foundations, outlier-aware mixed precision, second-order weight reconstruction, activation-aware scaling, rotations, cache quantization, and training with low-bit representations. Rather than pool incompatible benchmark results, the review distinguishes the object being quantized, the required preparation, the retained precision, and the implementation needed for a benefit. An analytical memory model separates compressed weights from quantization metadata, caches, and runtime workspace. A recomputable worked example and concurrency sensitivity analysis show why a fourfold reduction in nominal weight bytes does not imply a fourfold reduction in total serving memory. The paper proposes a reproducible comparison protocol that separates prefill from decoding and evaluates quality, latency distributions, throughput, peak memory, and amortized preparation cost under a common workload. It also distinguishes quantized fine-tuning from inference optimization and native low-bit training from post-training conversion. No new model benchmark, speedup, or accuracy result is reported. The central conclusion is that quantization should be selected against an application-specific operating envelope, with quality constraints and hardware measurements, rather than ranked by bit width alone.
Source: Preprints.org
Authors: Bankatesh Choudhary, Meenakshi Sharma
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7224454142