CVE-Pilot: Failure-Aware Tool Orchestration for Code LLM Agents on CVE-Bench
Year: 2026
Type: article
Abstract: CVE-Bench evaluates large language model (LLM) agents on 40 real-world, critical-severity web application CVEs in containerized sandboxes. The strongest reported baseline solves 10.0% of zero-day tasks and 12.5% of one-day tasks after five attempts. Common failures include missed endpoints, incorrect tool configuration, and loss of evidence needed for subsequent actions. We present CVE-Pilot, a code LLM agent that uses a typed state graph to record endpoints, credentials, tool outputs, and failed hypotheses. Coverage-guided planning directs exploration, tool contracts check execution and parse results, and target-aware verification tests the eight standard CVE-Bench attack outcomes. Successful and near-miss sandbox traces support self-training through CVE-TraceBank. On the evaluated tasks, CVE-Pilot achieves Success@5 of 25.0% in the zero-day setting and 32.5% in the one-day setting. CVE-level leave-one-out self-training increases these rates to 32.5% and 40.0%, respectively. The gains are accompanied by higher endpoint coverage, better evidence retention, and fewer exploration failures. The 40-CVE sample yields wide confidence intervals, and vulnerability reasoning remains a major source of failure.
Source: Applied Sciences
DOI: 10.3390/app162110747
Authors: You Zhai, Shouyou Song, Pu Wang, Lei Zhang
Language: English
Subfield: Information Systems
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
Estimated APC paid: $2,896
OpenAlex ID: W7224815732