Year: 2025
Type: conference-paper
Abstract: Large Language Models (LLMs) are increasingly wrapped in agents that orchestrate external tools, memories, and planning components to perform mission-critical functions. Despite growing interest in security testing of LLMs, the dependability of such agents remains largely unexplored. Existing robustness evaluations focus on prompt-level attacks and overlook non-adversarial system-level faults that routinely arise in real deployments (e.g., tool crashes or timeouts). We present ChaosLLM, a lightweight fault injection framework that sits between an LLM agent and its environment, allowing researchers to emulate realistic failures and quantify their impact on task success and recovery behavior. This paper focuses on tool-calling capabilities as a starting point for a broader investigation. We detail our experimental design, define the dependability metrics, and run the experiments on a LangChain ReAct agent. Our preliminary results motivate the need for further research efforts. Finally, we discuss how ChaosLLM can be extended in future work.
Author: Antonio Ken Iannillo
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W4416874092