Work

Sizing GPU Memory for Large Language Model Inference: A Practical Model of Weights, KV Cache, and Runtime Overhead

Year: 2026

Type: preprint

Abstract: Technical report (preprint) — NOT peer-reviewed. This report presents calculations only; no new hardware benchmarks were conducted. Abstract. This report presents a practical model for sizing GPU memory (VRAM) for large language model inference, accounting for model weights, the key-value (KV) cache, and runtime overhead. For an 8B-parameter model in BF16 precision, the model estimates 20.46 GB of VRAM for a single sequence with 16,384 tokens, and 22.60 GB for 8 concurrent users with 4,096 tokens each. These figures are estimates intended for capacity planning only; actual memory usage varies with implementation, framework, and hardware.

Source: Zenodo (CERN European Organization for Nuclear Research)

DOI: 10.5281/zenodo.23904297

Author: Ismail, Ali

Language: English

Topic: Parallel Computing and Optimization Techniques

Subfield: Hardware and Architecture

Field: Computer Science

Domain: Physical Sciences

Open Access status: green

OpenAlex ID: W7225176090

Cites:

Cited by:

Related to: 10