When the Total Is Right: Can small language models write tests for errors in payment reports?
Year: 2026
Type: preprint
Abstract: Testing a financial report requires two judgments. A reviewer must choose transactions on which a faulty report will differ from a correct one, then calculate the output that the correct report should produce. We evaluated these judgments using two fictional payment policies, two local language models, and 40 reviewer responses. In the settlement-day task, none of 20 responses satisfied the strict test contract. An exploratory replay recovered five Qwen inputs that exposed the fault, but all five predicted an incorrect breakdown. In the merchant-attribution task, seven of 20 responses supplied scorable inputs. All seven exposed the fault; five also predicted the correct result. These pilots illustrate why evaluations of generated tests should report input quality and expected-output accuracy separately. They provide limited evidence about the effect of showing reviewers the implementation.
Author: Yaredla, Sai Charith Reddy
Language: English
Subfield: Software
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7236237330
