MMSciCode: Real-world Evaluation of Multilingual Multi-Discipline Scientific Research Coding
Year: 2026
Type: conference-paper
Abstract: We introduce MMSciCode, a comprehensive expert-level, multilingual multi-discipline benchmark for evaluating foundation models in scientific code generation.It includes 624 expert-annotated research coding problems spanning six core scientific disciplines.Compared to prior benchmarks, MMSciCode features three key advancements.First, it challenges models to integrate domain-specific knowledge with algorithmic reasoning to implement core functions from research papers.Second, each problem is meticulously annotated by domain experts through a rigorous papergrounded process, with strict quality controls implemented to ensure dataset integrity and authenticity.Finally, each problem is equipped with comprehensive unit test suites and containerized environments, enabling reproducible and diagnostic evaluation of both functional correctness and domain validity.We conduct an extensive evaluation of 23 state-of-the-art foundation models and 2 coding agents on MMSciCode.We identify substantial performance gaps between models and human experts, providing actionable insights for advancing expert-level scientific code generation.
Authors: Xue Xia, Zheyuan Yang, Arman Cohan, Yilun Zhao
Institutions: Tongji University, Yale University
Language: English
Subfield: Information Systems
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W7166849583