Year: 2026
Type: conference-paper
Abstract: LLM-based software (LBS) integrates large language models as core components to deliver flexible and personalised responses. Unlike traditional software with deterministic outputs, LBSs exhibit context-dependent, stochastic behaviour that renders classical acceptance testing and test oracles insufficient: the same query may require fundamentally different responses depending on users' personas and software context. This gap creates an urgent need for automated acceptance testing frameworks that can autonomously interpret users' instructions, while also raising the challenge of how we can reliably interpret users' intentions in a changing environment. In this paper, we present an automated acceptance testing framework for LBS with calibrated verdict reliability through two technical contributions. First, we introduce REquirements-Augmented Generation (REAG), which interprets user intentions by retrieving relevant software requirements, domain knowledge, and personas via adaptive RAG and self-reasoning to generate context-aware test oracles. Second, recognising that oracle generation may retrieve irrelevant constraints, misinterpret intent, or hallucinate requirements, we introduce a confidence-calibrated cascade judgment that quantifies verdict reliability via simulated expert agreement, accepting high-confidence verdicts, escalating ambiguous cases, or abstaining when uncertain, with empirical reliability guarantees backed by conformal risk control. An industrial case study on a production nutrition advisory application demonstrates that REAG achieves a 3.91/5 oracle quality score, successfully achieving qualified or marginal oracle quality in 82% of cases. The confidence-calibrated cascade achieves 98.8% accuracy, improves oracle quality from 3.91 to 4.30 by filtering unqualified outputs, and delivers a 31.7% cost-efficiency improvement over single-judge baselines, validating industrial viability.
Subfield: Software
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W7203446506
FWCI:
Cites:
Cited by:
Related to: