RACE
Year: 2026
Type: dataset
Abstract: In recent years, researchers have proposed numerous benchmarks to evaluate the impressive coding capabilities of large language models (LLMs). However, current benchmarks primarily assess the accuracy of generated code while neglecting the LLMs' multi-dimensional instruction-following abilities, resulting in a gap between benchmark performance and real-world development requirements. Therefore, this paper proposes the RACE benchmark, which focuses on evaluating LLMs' instruction-following ability across four dimensions: Readability, mAintainability, Correctness, and Efficiency. We select specific, actionable attributes for each dimension and establish verifiable instructions grounded in common software development practices. We then develop deterministic evaluation methods to ensure objective and reliable assessment. We analyze 24 representative LLMs based on RACE and find that: 1) Using correctness as the only guiding indicator is insufficient to steer code LLMs towards comprehensive advancement. 2) RACE not only provides valuable insights that reveal the defects of current LLMs across multiple dimensions, but also serves as an effective tool for mitigating the risks of data contamination. These findings highlight the need for a multi-dimensional instruction-following evaluation for LLMs' code ability, emphasizing metrics beyond correctness for real-world applications.
Source: ScienceDB
Institution: Institute of Software
Language: English
Subfield: Information Systems
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7204751395