Exploring Layer-wise Information Effectiveness for Post-Training Quantization in Small Language Models
Year: 2026
Type: conference-paper
Abstract: Large language models with billions of parameters are often over-provisioned: many layers contribute little unique information yet dominate the memory and energy footprint during inference.We present LieQ (Layerwise information effectiveness Quantization), a hardware-native, metric-driven post-training quantization framework that addresses the critical challenge of maintaining accuracy in sub-8B models, model parameters less than 8B, under extreme low-bit compression.LieQ keeps uniform bit-width within each layer while mixing precision across layers, preserving standard multiplication kernels and avoiding irregular memory access, codebooks, or irregular formats at inference time.Our method uncovers a strong correlation between layer-wise functional saliency and representational compactness, revealing that layers with higher traininginduced energy concentration are functionally irreplaceable.Leveraging this insight, we propose a purely geometry-driven sensitivity proxy that enables automatic bit-width allocation under a target average-bit budget without expensive gradient updates or inference-based perplexity probing.Under an average weight bit-width approaching two bits per parameter, LieQ consistently reduces the large accuracy gap typically observed for naive uniform 2bit baselines on Qwen3 and LLaMA3.xfamilies, while retaining standard-kernel efficiency.These properties make LieQ a practical path toward deploying small language models on resource-constrained edge devices.
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W7166868033