Beyond SWE-Bench: A Systematic Review of Reliability, Security, and Real-World Evaluation in Autonomous Coding Agents
Year: 2026
Type: preprint
Study design: Systematic Review
Abstract: This systematic review will critically synthesize evidence on how autonomous coding agents are evaluated beyond conventional task-completion scores. The review will examine functional reliability, robustness and reproducibility, security and safety, benchmark validity, and performance under real-world or deployment-relevant software-engineering conditions. Particular attention will be given to evaluation practices surrounding SWE-bench and related repository-level benchmarks, including test-oracle adequacy, contamination or solution leakage, repeated-run consistency, security assessment, human involvement, cost, maintainability, and transfer to realistic development workflows. The review will cover eligible literature first publicly available between 1 January 2015 and 31 July 2026 and will use systematic database searching, citation chasing, structured screening, quality assessment, and critical evidence synthesis.
Author: Mahmoud M. Selim
Language: English
Subfield: Information Systems
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7204154703