EVM-QuestBench: An Execution-Grounded Benchmark for Natural-Language Transaction Code Generation
Year: 2026
Type: conference-paper
Abstract: Large language models are increasingly applied to various development scenarios.However, in on-chain transaction scenarios, even a minor error can cause irreversible loss for users.Existing evaluations often overlook execution accuracy and safety.We introduce EVM-QuestBench, an execution-grounded benchmark for natural-language transaction-script generation on EVM-compatible chains.The benchmark employs dynamic evaluation: instructions are sampled from template pools, numeric parameters are drawn from predefined intervals, and validators verify outcomes against these instantiated values.EVM-QuestBench contains 107 tasks (62 atomic, 45 composite).Its modular architecture enables rapid task development.The runner executes scripts on a forked EVM chain with snapshot isolation; composite tasks apply step-efficiency decay.We evaluate 20 models with 5 independent rounds each and find large performance gaps, with split scores revealing persistent asymmetry between single-action precision and multi-step workflow completion.
Authors: Pei Yang, Wanyi Chen, Ke Wang, Lynn Ai, Eric Yang, Tianyu Shi
Institution: Soochow University
Language: English
Subfield: Information Systems
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W7166847040