Year: 2025
Type: preprint
Abstract: Benchmarking has driven scientific progress in Evolutionary Computation, yet current practices fall short of real-world needs. Widely used synthetic suites such as BBOB and CEC isolate algorithmic phenomena but poorly reflect the structure, constraints, and information limitations of continuous and mixed-integer optimization problems in practice. This disconnect leads to the misuse of benchmarking suites for competitions, automated algorithm selection, and industrial decision-making, despite these suites being designed for different purposes. We identify key gaps in current benchmarking practices and tooling, including limited availability of real-world-inspired problems, missing high-level features, and challenges in multi-objective and noisy settings. We propose a vision centered on curated real-world-inspired benchmarks, practitioner-accessible feature spaces and community-maintained performance databases. Real progress requires coordinated effort: A living benchmarking ecosystem that evolves with real-world insights and supports both scientific understanding and industrial use.
Source: arXiv (Cornell University)
Authors: Kononova, Anna V., van Stein, Niki, Mersmann, Olaf, Bäck, Thomas, Bartz-Beielstein, Thomas, Glasmachers, Tobias, Hellwig, Michael, Krey, Sebastian, Kůdela, Jakub, Naujoks, Boris, Papenmeier, Leonard, Raponi, Elena, Renau, Quentin, Rook, Jeroen, Schäpermeier, Lennart, Vermetten, Diederick, Zaharie, Daniela
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W4416458439
Cites:
Cited by:
Related to: