Work

InferLine: latency-aware provisioning and scaling for prediction serving pipelines

Year: 2020
Type: conference-paper
Abstract: Serving ML prediction pipelines spanning multiple models and hardware accelerators is a key challenge in production machine learning. Optimally configuring these pipelines to meet tight end-to-end latency goals is complicated by the interaction between model batch size, the choice of hardware accelerator, and variation in the query arrival process.
Language: English
Open Access status: hybrid
OpenAlex ID: W3095488153
FWCI: 9.131
Cites: 33
Cited by: 208
Related to: 10