Work

Benchmarking Best Practices for LLM-driven Automated Algorithm Design in Heuristic Optimization

Year: 2026
Type: preprint
Abstract: Large Language Models (LLMs) are increasingly used to design optimization algorithms, in a paradigm referred to as LLM-driven Automated Algorithm Design (LLM4AAD). Such a shift challenges established benchmarking practice in optimization, which was built around evaluating a single fixed algorithm. In contrast, LLM4AAD systems produce a generated algorithm through a stochastic, LLM-driven generation process, and both the generation and the generated algorithms deserve attention in benchmarking. This position paper adapts and extends established benchmarking principles from evolutionary optimization to the specifics of LLM-driven algorithm design. We formalize the algorithm design problem, distinguishing the generated algorithm from the generation process that produces it, and argue that the two require distinct benchmarking targets and metrics. Using two representative examples, we systematically revisit the benchmarking goals established for classical meta-heuristic optimization, assess the current maturity of LLM4AAD practices against each, and introduce a further set of goals specific to the use of LLMs as the generative component. We further examine the reproducibility threats posed by stochastic, evolving, and often closed-source LLM backends, and outline minimum reporting practices to mitigate them. We conclude with a research agenda identifying the community infrastructure and evaluation protocols needed to establish shared benchmarking standards for the field.
Language: English
Open Access status: green
OpenAlex ID: W7234337542
Cites:
Cited by:
Related to: 10