Empirical Evaluation of 1.58-Bit Ternary Large Language Models on Resource-Constrained Consumer Hardware
Year: 2026
Type: preprint
Abstract: The deployment of Large Language Models (LLMs) on local consumer-grade edge devices is severely constrained by strict hardware limitations, particularly physical VRAM capacity. While low-bit quantization offers a path forward, 1.58-bit ternary quantization theoretically eliminates matrix multiplications entirely, replacing floating-point calculations with integer additions and subtractions. In this work, we present a comprehensive systems-level empirical evaluation of a 7B parameter ternary model (Falcon3-7B-Instruct-1.58bit) running on a resource-constrained NVIDIA GeForce GTX 1650 Ti GPU (4GB VRAM) and a multi-core host CPU. We benchmark key operational metrics including prefill/decode throughput, dynamic active power draw (Watts), total energy consumption (Joules), context scaling memory envelopes, CPU multithreading constraints, and API-level caching efficiency. Our findings demonstrate that 1.58-bit models enable up to a 16,384- token context window within a 4GB VRAM envelope and achieve a 64.2% reduction in dynamic GPU active power draw (averaging 17.90 W under full load compared to 49.96 W for standard 2-bit models). Furthermore, we detail the speed-energy trade-offs resulting from unoptimized custom kernels and evaluate the effects of structured GBNF grammar constraints on inference latency. This study provides critical empirical baselines and highlights the optimization gaps remaining for ternary architecture deployment on the edge.
Author: Singh, Ichhabal
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7171493420
Cites:
Cited by:
Related to: