Year: 2026
Type: conference-paper
Abstract: Understanding and reasoning about entire software repositories is an essential capability for intelligent software engineering tools.While existing benchmarks such as CoSQA and Cod-eQA have advanced the field, they predominantly focus on small self-contained code snippets.These setups fail to capture the complexity of real-world repositories, where effective understanding and reasoning often require navigating multiple files, understanding software architecture, and grounding answers in long-range code dependencies.In this paper, we present SWE-QA, a repository-level code question answering (QA) benchmark designed to facilitate research on automated QA systems in realistic code environments.SWE-QA involves 720 high-quality question-answer pairs spanning diverse categories, including intention understanding, cross-file reasoning, and multi-hop dependency analysis.To construct SWE-QA, we first crawled 77,100 GitHub issues from 12 popular repositories.Based on an analysis of naturally occurring developer questions extracted from these issues, we developed a two-level taxonomy of repository-level questions and constructed a set of seed questions for each category.For each category, we manually curated and validated questions and collected their corresponding answers.We evaluated six advanced LLMs on SWE-QA under various context augmentation strategies.Experimental results highlight the promise of LLMs.
Authors: Weihan Peng, Yuling Shi, Yuhang Wang, Xinyun Zhang, Beijun Shen, Xiaodong Gu
Institution: Shanghai Jiao Tong University
Language: English
Subfield: Information Systems
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
Award: G4906496431
OpenAlex ID: W4416256154