Year: 2026
Type: preprint
Abstract: We present OPENHARMONY BENCH, an app-level coding benchmark for evaluating LLM-based coding agents on OpenHarmony ArkTS applications. Unlike function-level benchmarks, it evaluates complete app-level changes: each task requires an agent to modify a buildable ArkTS project so that a requested behavior works end to end, involving UI state, data persistence, build configuration, and platform APIs. The benchmark installs and drives the delivered application on a device to check whether the behavior is observable. It covers three input sources: natural-language feature requests (new-feature), structured scenario specifications (spec-driven), and bug descriptions (bug-fix). The benchmark contains 153 top-level tasks and 242 Feature points (F-points), where an F-point is one executable behavior check. The snapshot includes 32 new-feature tasks, 50 spec-driven tasks with 139 F-points, and 71 bug-fix tasks. The main leaderboard is scored over top-level tasks rather than independently weighted F-points. We describe the benchmark construction, statistics, and build-and-test evaluation pipeline, and evaluate DevEco Code with eight LLMs across three independent full-suite runs per configuration. Three findings emerge. First, newer generations complete more tasks than their predecessors within evaluated model-family pairs. Second, buildability is close to saturated while behavioral correctness is not: mean Final Build Success Rate is 94.77% to 100.00%, whereas mean Task Completion is 48.36% to 58.39%. Third, spec-driven tasks have the lowest Task Completion under all-checks task scoring, with no configuration exceeding 35%. The code, data, tasks, reference solutions, tests, evaluation scripts, and leaderboard are released through the official OPENHARMONY BENCH website at https://bench.matrix.openharmony.cn/.
Source: arXiv (Cornell University)
Authors: Li, Li, Hu, Han, Zhang, Tianjian, Peng, Xin, Mao, Fangzhu, Zhang, Qingyu, Xie, Xiaoheng, Tang, Zhongmin, Lin, Zhihao, Ruan, Haolin, Dong, Miaomiao, Zhu, Liuchuan, Li, Yue, Chen, Chi, Zhong, Wenkang, Zhang, Mingfei, Yu, Yang, Sun, Bo, Zhang, Chaorui, Zhang, Weixi, Han, Wei, Bai, Bo, Liu, Kui, Fan, Gang, Liu, Siru, Zhou, Jiaqian, Sun, Jiali, Dong, Yunbiao, Zhong, Wenhao, Xu, Yunhong
Subfield: Information Systems
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7203669118
Cites:
Cited by:
Related to: