Work

SecRepoBench: Benchmarking Code Agents for Secure Code Completion in Real-World Repositories

Year: 2026

Type: conference-paper

Abstract: This paper introduces SecRepoBench, a benchmark to evaluate code agents on secure code completion in real-world repositories. SecRepoBench has 318 code completion tasks in 27 C/C++ repositories, covering 15 CWEs. We evaluate 29 standalone LLMs and 15 code agents across 3 state-of-the-art agent frameworks using our benchmark. We find that state-of-the-art LLMs struggle with generating correct and secure code completions. However, code agents significantly outperform standalone LLMs. We show that SecRepoBench is more difficult than the prior state-of-the-art benchmark. Finally, our comprehensive analysis provides insights into potential directions for enhancing the ability of code agents to write correct and secure code in real-world repositories.

Source: International Workshop on Large Language Models for Code (LLM4Code)

DOI: 10.1145/3786181.3788703

Authors: Chihao Shen, Connor Dilgren, Purva Chiniya, Luke Griffith, Yu Ding, Yizheng Chen

Institutions: University of Maryland, College Park, Google (United States)

Language: English

Topic: Software Engineering Research

Subfield: Information Systems

Field: Computer Science

Domain: Physical Sciences

Open Access status: hybrid

Funders: National Science Foundation, Open Philanthropy Project, DeepMind

Award: CAREER: Secure Code Generation with Large Language Models