Year: 2025
Type: conference-paper
Abstract: While large language models (LLMs) have shown considerable promise in code generation, real-world software development demands advanced repository-level reasoning.This includes understanding dependencies, project structures, and managing multi-file changes.However, the ability of LLMs to effectively comprehend and handle complex code repositories has yet to be fully explored.To address challenges, we introduce a hierarchical benchmark designed to evaluate repository dependency understanding (DEPENDEVAL).Benchmark is based on 2683 repositories collected from real-world websites.It evaluates models on three core tasks: Dependency Recognition, Repository Construction, and Multi-file Editing, across 8 programming languages from actual code repositories.Our evaluation of over 25 LLMs reveals substantial performance gaps and provides valuable insights into repositorylevel code understanding 1
Authors: Junjia Du, Yadi Liu, Hongcheng Guo, Jiawei Wang, Haojian Huang, Yunyi Ni, Zhoujun Li
Language: English
Subfield: Information Systems
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
OpenAlex ID: W4412888528