Work

CI-DeceptiveTest: An Execution-Paired Benchmark for Test Assurance in AI-Assisted Development

Year: 2026
Type: preprint
Abstract: AI-assisted development weakens the meaning of a green continuous-integration (CI) check: generated tests may execute realistic code and pass while replacing semantic oracles with tautologies or coarse smoke checks. We introduce CI-DeceptiveTest, a Python execution-paired benchmark and Gatekeeper protocol for detecting such low-assurance tests. Its 600 pairs originate from real tests in 12 open-source repositories; each source-control test is paired with a candidate produced by controlled oracle weakening, yielding 1200 label-free static-binary views. Construction combines deterministic validation, execution evidence, a 20% manually annotated slice, random inspection of 30% of AI-assisted records. The evidence is GPT-5.5-first: GPT-5.5 supports construction, review, and full-dataset validation. Labels are transformation-defined, not model-assigned. On the full benchmark, transparent static rules achieve 0.928 low-assurance recall but a 0.458 false-block rate on accepted source-control tests. mimo-v2.5-pro is retained as an auxiliary external cross-check; it reaches 0.885 macro F1 and reduces false blocking to 0.048. Bounded mutation analysis over 42 pre-specified source/candidate pairs further supports the paired construction. The central stress result is that weak-oracle detection remains hard: plausible coarse assertions retain execution while losing semantic constraint. The benchmark enables CI gatekeeper evaluation in a controlled, reproducible setting.
Author: Qining Li
Language: English
Subfield: Software
Open Access status: green
OpenAlex ID: W7226035654
Cites: 8
Cited by:
Related to: 10