continuous batching
Description: A serving technique that dynamically adds and removes requests from batches during model inference to improve accelerator utilization and throughput.
Primary topic: Parallel Computing and Optimization Techniques (27%)
OpenAlex ID: keywords/continuous-batching
