Empirical Evaluation of Iterative Error-Feedback Loops in Automated TypeScript Unit Test Generation: A Cost-Efficiency and Turn-Budget Perspective
Year: 2026
Type: preprint
Abstract: Automated unit test generation using Large Language Models (LLMs) has emerged as a promising methodology to alleviate developer burden in continuous integration (CI) lifecycles. However, unguided single-shot generation often yields test suites plagued by subtle assertion errors and interface discrepancies. While recent literature demonstrates that iterative validation-and-repair loops improve test pass rates in dynamically typed environments like Python, empirical investigations into strictly typed environments—specifically TypeScript—remain scarce. In this empirical study, we investigate the multi-turn error-feedback loop for unit test generation across 30 diverse TypeScript functions categorized into three complexity tiers (algorithmic utilities, data validation, and asynchronous logic). We evaluate four cost-effective frontier models: Google Gemini 3.8 Flash, DeepSeek-V4.1 Flash, Alibaba Qwen 3.8 Flash, and Zhipu AI GLM-5.3 Flash, with an execution repair budget of up to K=3 iterations. We formulate four research questions addressing: (1) pass rate and repair convergence dynamics across turns and complexity tiers, (2) turn-budget saturation and error stagnation, (3) comparative model performance and Cost-per-Passing-Test (CPT), and (4) a failure taxonomy distinguishing compilation errors from runtime assertion failures. Our empirical findings reveal that three of the four models (Gemini 3.8 Flash, Qwen 3.8 Flash, and GLM-5.3 Flash) achieve a flawless 100.0% final pass rate, while DeepSeek-V4.1 Flash attains 96.7%, yielding an aggregate benchmark pass rate of 99.2% (119/120 test suites) across all evaluated cases. Crucially, a single feedback iteration (K=1) successfully recovers 94.7% (18/19) of initial failures, whereas subsequent iterations (K=2 and K=3) yield exactly 0.0% marginal gain, establishing a strict empirical stopping rule of K* = 1 for production CI/CD pipelines. Furthermore, GLM-5.3 Flash emerges as the most cost-efficient model at $0.00051 per passing test (5.36× cheaper than Gemini 3.8 Flash). Finally, we demonstrate that Server-Sent Events (SSE) streaming is an essential architectural requirement for reasoning-capable models to eliminate gateway read timeouts. This study provides actionable, cost-calibrated guidelines for engineering teams deploying LLMs for automated software testing.
Author: Fuadi, Syauqi
Language: English
Subfield: Software
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7236042145
