Work

Can LLMs Generate High-Quality Test Cases for Algorithm Problems? TestCase-Eval: A Systematic Evaluation of Fault Coverage and Exposure

Year: 2025

Type: conference-paper

Abstract: We introduce TestCase-Eval, a new benchmark for systematic evaluation of LLMs in test-case generation.TestCase-Eval includes 500 algorithm problems and 100,000 human-crafted solutions from the Codeforces platform.It focuses on two pivotal tasks: (1) Fault Coverage, which measures how well LLM-generated test sets probe diverse input scenarios and cover a wide range of potential failure modes.(2) Fault Exposure, which evaluates whether LLMs can craft a tailored test input that reveals a specific incorrect code implementation.We provide a comprehensive assessment of 19 stateof-the-art open-source and proprietary LLMs on TestCase-Eval, offering insights into their strengths and limitations in generating effective test cases for algorithm problems.

Source: Annual Meeting of the Association for Computational Linguistics (ACL)

DOI: 10.18653/v1/2025.acl-short.82

Authors: Zheyuan Yang, Zexi Kuang, Xue Xia, Yilun Zhao

Institutions: Tongji University, Yale University

Language: English

Topic: Software Testing and Debugging Techniques

Subfield: Software

Field: Computer Science

Domain: Physical Sciences

Open Access status: hybrid

OpenAlex ID: W4412889497

FWCI: 1.642

Cites:

Cited by: 2

Related to: 10