Do Smart Contract Auditing Results Transfer Across Datasets? A Two-Benchmark Empirical Study of Static and LLM-Based Security Tools
Year: 2025
Type: article
Abstract: Background. Implementing unit tests is an important yet time-consuming activity in software development. Therefore, Large Language Models (LLMs) are being used increasingly often for generating unit tests. Many previous work report promising results for open-source repositories written in common languages such as Python or Java. However, the performance of LLM-based test generation in large proprietary software projects written in C++ remains largely unknown. Aims. To reduce this gap, we investigate the performance of LLM-based test generation on a large closed-source C++ software project, which is not part of the training corpus of LLMs. Method. We study test generation on SAP HANA, a large C++ DBMS, and compare it to the open-source key-value store LevelDB. We analyze two LLM-based test generation approaches, each including an iterative repair loop for compilation failures, along five metrics: compilation success rate (CSR), execution success rate, line coverage, branch coverage, and mutation score (MS). Results. The metrics show a considerable gap between the two systems. The LLM-generated tests for LevelDB match the quality of the existing tests. On SAP HANA, the LLM-generated tests achieve only 25.2% MS and underwhelming coverage results. Iterative repair with at least 3 iterations is important, reaching 97.6% CSR on SAP HANA after 10 iterations. However, with more iterations, LLM may prioritize compilability over test quality, resulting in tests with weak assertions. Conclusions. Results reported on open-source projects, whose code is often part of LLM training data, may not generalize to unseen, industry-grade C++ codebases. The effectiveness of iterative repair saturates after several iterations. When applying LLMs to source code not in training data, practitioners should carefully select the context provided to the models.
Authors: Baltes, Sebastian, Angermeir, Florian, Arora, Chetan, Barón, Marvin Muñoz, Chen, Chunyang, Böhme, Lukas, Calefato, Fabio, Ernst, Neil, Falessi, Davide, Fitzgerald, Brian, Fucci, Davide, He, Junda, Treude, Christoph, Kalinowski, Marcos, Lambiase, Stefano, Russo, Daniel, Lungu, Mircea, Montes, Cristina Martinez, Prechelt, Lutz, Ralph, Paul, van Tonder, Rijnard, Wagner, Stefan
Institutions: University of Bayreuth, Monash University, University of Bari Aldo Moro, University of Victoria, University of Limerick, Lero, Blekinge Institute of Technology, Singapore Management University, Pontifícia Universidade Católica do Rio de Janeiro, Aalborg University, IT University of Copenhagen, Freie Universität Berlin, Dalhousie University, Technical University of Munich
Language: English
Subfield: Information Systems
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
Funder: Agence Nationale de la Recherche
OpenAlex ID: W7077142139