Year: 2026
Type: conference-paper
Abstract: Large language model (LLM) agents increasingly rely on external tools (file operations, API calls, database transactions) to autonomously complete complex multi-step tasks.Practitioners deploy defense-trained models to protect against prompt injection attacks that manipulate agent behavior through malicious observations or retrieved content.We reveal a fundamental capability-alignment paradox: defense training designed to improve safety systematically destroys agent competence while failing to prevent sophisticated attacks.Evaluating defended models against undefended baselines across 97 agent tasks and 1,000 adversarial prompts, we uncover three systematic biases unique to multi-step agents.Agent incompetence bias manifests as immediate tool execution breakdown, with models refusing or generating invalid actions on benign tasks before observing any external content.Cascade amplification bias causes early failures to propagate through retry loops, pushing defended models to timeout on 99% of tasks compared to 13% for baselines.Trigger bias leads to paradoxical security degradation where defended models perform worse than undefended baselines while straightforward attacks bypass defenses at high rates.Root cause analysis reveals these biases stem from shortcut learning: models overfit to surface attack patterns rather than semantic threat understanding, evidenced by extreme variance in defense effectiveness across attack categories.Our findings demonstrate that current defense paradigms optimize for single-turn refusal benchmarks while rendering multi-step agents fundamentally unreliable, necessitating new approaches that preserve tool execution competence under adversarial conditions.
Institution: University of Southern California
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: bronze
Awards: G1353381076, G1453181110, G1503004319, NSF POSE: Phase II: OpenAD: An Integrated Open-Source Ecosystem for Anomaly Detection, Track 4: Advanced CI Coordination Ecosystem: Monitoring and Measurement Services, POSE: Phase I: OpenTrustLLM: An Open-Source Ecosystem for Trustworthiness in Large Language Models, G298330452, G3562386289, Track 1: ACCESS Resource Allocations Marketplace and Platform Services (RAMPS), G4010157413, G4403015908, ACO: An Open CI Ecosystem to Advance Scientific Discovery (OpenCI), Track 3: COre National Ecosystem for CyberinfrasTructure (CONECT), G5367127336, G5464919783, ATD: Multifaceted Large Trajectory Model for Individual and Group Anomalous Behavior Detection from Spatiotemporal Data, G8108599699, Track 2: Customized Multi-tier Assistance, Training, and Computational Help (MATCH) for End User ACCESS to CI
OpenAlex ID: W7140082020
