When Tests Need an Answer: Oracle Underspecification Suppresses Diagnostic Exploration in LLM-Generated Verification
Year: 2026
Type: preprint
Abstract: Executable tests generated from natural-language requirements require definite expected outputs. When a requirement leaves an oracle unresolved, a generator can guess, abstain, or avoid inputs on which the unresolved choice matters. This controlled study tests whether requiring a definite executable oracle suppresses diagnostic-input selection under requirement underspecification, separates the effect from edge-case salience, and evaluates an explicit UNRESOLVED output channel. Across 12 finite-domain tasks in four semantic families, salience-matched ambiguity produced 0/192 diagnostic selections versus 52/96 after clarification. Allowing UNRESOLVED restored 69/192 diagnostic selections, although mitigation precision was 69/79 (87.3%), below the preregistered 90% gate. The study identifies a bounded oracle-coupled selection failure mode rather than estimating its prevalence.
Author: Kadri, Haitam
Language: English
Subfield: Software
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7208810336
Cites:
Cited by:
Related to: