Experiment runner: A tool for the automatic orchestration of experiments targeting software systems
Year: 2025
Type: article
Abstract: Experiment Runner (ER) is a Python framework for the (semi-) automatic orchestration of measurement-based experiments targeting software systems. ER allows users to (i) programmatically define an experiment in terms of its factors, treatments, subjects, etc., (ii) start, stop, pause, and resume an experiment with no data loss, (iii) integrate third-party software- or hardware-profilers, (iv) execute an experiment either in automatic mode or semi-automatic mode, and (v) define custom callbacks that are automatically called by ER when needed. ER has been used in tens of empirical studies in the Software Engineering area. In this paper, we describe the main features of ER, its software architecture, a simple experiment executed by ER, and three scientific studies where it has been used successfully.
Source: Science of Computer Programming
Authors: Max Karsten, Andrei Calin Dragomir, Radu Apsan, Vincenzo Stoico, Ivano Malavolta
Institution: Vrije Universiteit Amsterdam
Language: English
Subfield: Information Systems
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
Estimated APC paid: $2,600
OpenAlex ID: W4416530204