Multi-dimensional Constraint-based Test Case Generation and Evaluation Framework for Large Language Models
Year: 2025
Type: article
Abstract: Addressing the challenges of complex test case design and insufficient coverage in functional testing of large language models, this paper presents a multi-dimensional constraint-based test case generation framework. The framework defines constraint rules across four dimensions: syntactic correctness, semantic consistency, task relevance, and boundary conditions, employing reinforcement learning methods to optimize the test case generation process. Through the design of reward function-based generation strategies, the system can automatically produce high-quality functional test samples covering core tasks including text classification, sentiment analysis, and machine translation. Experimental results demonstrate that test cases generated by this method achieve a 42% improvement in functional coverage compared to random generation methods and a 28% increase in defect detection rate. Further ablation experiments validate the effectiveness of each dimensional constraint, providing a systematic solution for large language model quality assurance.
Author: Xuebing Wang
Institution: Yonyou (China)
Language: English
Subfield: Software
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W4410875297