Work

MolLangBench: A Comprehensive Benchmark for Language-Prompted Molecular Structure Recognition, Editing, and Generation

Year: 2025

Type: preprint

Abstract: Precise recognition, editing, and generation of molecules are essential prerequisites for both chemists and AI systems tackling various chemical tasks. We present MolLangBench, a comprehensive benchmark designed to evaluate fundamental molecule-language interface tasks: language-prompted molecular structure recognition, editing, and generation. To ensure high-quality, unambiguous, and deterministic outputs, we construct the recognition tasks using automated cheminformatics tools, and curate editing and generation tasks through rigorous expert annotation and validation. MolLangBench supports the evaluation of models that interface language with different molecular representations, including linear strings, molecular images, and molecular graphs. Evaluations of state-of-the-art models reveal significant limitations: the strongest model (GPT-5) achieves $86.2\%$ and $85.5\%$ accuracy on recognition and editing tasks, which are intuitively simple for humans, and performs even worse on the generation task, reaching only $43.0\%$ accuracy. These results highlight the shortcomings of current AI systems in handling even preliminary molecular recognition and manipulation tasks. We hope MolLangBench will catalyze further research toward more effective and reliable AI systems for chemical applications.The dataset and code can be accessed at https://huggingface.co/datasets/ChemFM/MolLangBench and https://github.com/TheLuoFengLab/MolLangBench, respectively.

Source: arXiv (Cornell University)

DOI: 10.48550/arxiv.2505.15054

Authors: Cai, Feiyang, Bai, Jiahui, Tang, Tao, He, Guijuan, Luo, Joshua, Zhu, Tianyu, Pilla, Srikanth, Li, Gang, Liu, Ling, Luo, Feng

Institutions: Clemson University, University of Delaware, Georgia Institute of Technology

Language: English

Topic: Computational Drug Discovery Methods

Subfield: Computational Theory and Mathematics

Field: Computer Science

Domain: Physical Sciences

Open Access status: green

Funders: National Science Foundation, U.S. Department of Energy, Cisco Systems, Clemson University, Directorate for Computer and Information Science and Engineering, Office of Science, Basic Energy Sciences

Awards: ARTIFICIALLY INTELLIGENT MANUFACTURING PARADIGM FOR COMPOSITES (AIM FOR COMPOSITES), G1486300252, G1636064349, NSF-CSIRO: RAI4IoE: Responsible AI for Enabling the Internet of Energy, ARTIFICIALLY INTELLIGENT MANUFACTURING PARADIGM FOR COMPOSITES (AIM FOR COMPOSITES), SHF: Medium: Cross-Stack Algorithm-Hardware-Systems Optimization Towards Ubiquitous On-Device 3D Intelligence

OpenAlex ID: W4415328436

Cites:

Cited by: 1

Related to: