Get started
Work

ChainSWE: Benchmarking Coding Agents on Multi-Bug Software Maintenance

Year: 2026
Type: preprint
Abstract: Language model (LM) agents are increasingly deployed to maintain codebases over extended periods, fixing streams of related defects while carrying context from one fix to the next. Yet existing software engineering (SWE) benchmarks evaluate models one bug at a time: the repository is reset, the codebase is re-read, and a single self-contained issue is graded in isolation. This setting collapses a continuous maintenance workflow into a series of independent sessions, ignoring the cumulative dependencies that make real-world bug fixing challenging. To bridge this gap, we introduce ChainSWE, the first benchmark for evaluating agents on sequential, dependent bug fixes within a shared codebase. We collect chronological chains of 304 issues across 54 Python projects, mined from six SWE-bench-family datasets. Our evaluation across a range of agents and models reveals a consistent performance drop by up to 70% as the chain length increases.
DOI: 10.48550/arxiv.2607.02606
Authors: Jin, Qirui, Tung, Lingching, Li, Kenan, Shi, Qiyang, She, Yushi, Jia, Huanzhong, Zhao, Harrison, Xia, Kejing, Du, Zhenbang, Pei, Jiaxin, Zhang, Zhenyu, Qi, Zhen, Duan, Yuyan, Lee, Wenke, Jin, Zijian
Institutions: Georgia Institute of Technology, University College London, New York University, Cornell University, Stanford University, Northeastern University
Open Access status: green
OpenAlex ID: W7167571499
Cites:
Cited by:
Related to: