Work

StreamBed: Capacity Planning for Stream Processing

Year: 2024

Type: conference-paper

Abstract: We present StreamBed, a capacity planning system for stream processing. StreamBed predicts, ahead of any production deployment, the resources that a query will require to process an incoming data rate sustainably, and the appropriate configuration of these resources. For this purpose, StreamBed builds a capacity planning model by piloting a series of runs of the target query in a small-scale, controlled testbed. We implement StreamBed for Apache Flink. Our evaluation with large-scale queries of the Nexmark benchmark demonstrates that StreamBed can accurately predict capacity requirements for jobs spanning more than 1,000 cores using a model built with a 48-core testbed.

Source: ACM International Conference on Distributed and Event-Based Systems (DEBS)

DOI: 10.1145/3629104.3666034

Authors: Guillaume Rosinosky, Donatien Schmitz, Etienne Rivière

Institutions: École Centrale de Nantes, Centre National de la Recherche Scientifique, Institut national de recherche en sciences et technologies du numérique, Laboratoire des Sciences du Numérique de Nantes, IMT Atlantique, UCLouvain, Nantes Université

Language: English

Topic: Advanced Database Systems and Queries

Subfield: Information Systems

Field: Computer Science

Domain: Physical Sciences

Open Access status: hybrid

Funder: Centre National de la Recherche Scientifique

OpenAlex ID: W4400880920

FWCI: 1.892

Cites: 17

Cited by: 9

Related to: 10