Year: 2024
Type: article
Abstract: Six years ago, we performed the first large-scale comparison of automated test generators and software model checkers with respect to bug-finding capabilities on a benchmark set with 5693 C programs. Since then, the International Competition on Software Testing (Test-Comp) has established standardized formats and community-agreed rules for the experimental comparison of test generators. With this new context, it is time to revisit our initial question: Model checkers or test generators—which tools are more effective in finding bugs in software? To answer this, we perform a comparative analysis on the tools and existing data published by two competitions, the International Competition on Software Verification (SV-COMP) and Test-Comp. The results provide two insights: (1) Almost all test generators that participate in Test-Comp use hybrid approaches that include formal methods, and (2) although the considered model checkers are still highly competitive, they are now outperformed by the bug-finding capabilities of the considered test generators.
Authors: Dirk Beyer, Thomas Lemberger
Institution: Ludwig-Maximilians-Universität München
Language: English
Subfield: Software
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
Estimated APC paid: $3,190
OpenAlex ID: W4406473434