Work

M3SciQA: A Multi-Modal Multi-Document Scientific QA Benchmark for Evaluating Foundation Models

Year: 2024

Type: conference-paper

Abstract: Existing evaluation benchmarks for foundation models in understanding scientific literature predominantly focus on single-document, text-only tasks.Such benchmarks often do not adequately represent the complexity of research workflows, which typically also involve interpreting non-textual data, such as figures and tables, and gathering information across multiple documents and related literature.To address this gap, we introduce M3SCIQA, a multi-modal, multi-document scientific question answering benchmark designed for a more comprehensive evaluation of foundation models.M3SCIQA consists of 1,452 expertannotated questions spanning 70 natural language processing paper clusters, where each cluster represents a primary paper along with all its cited documents, mirroring the workflow of comprehending a single paper by requiring multi-modal and multi-document data.With M3SCIQA, we conduct a comprehensive evaluation of 18 frontier foundation models.Our results indicate that current foundation models still significantly underperform compared to human experts in multi-modal information retrieval and in reasoning across multiple scientific documents.Additionally, we explore the implications of these findings for the future advancement of applying foundation models in multi-modal scientific literature analysis."The results presented in the Information Value paper are intriguing.The figure clearly indicates that the DialoGPT family of models exhibits the most negative Spearman's Correlation Coefficient compared to all other models." 1 Reading an anchor paper Locality-Specific Question: "Which model exhibits the most negative Spearman's Correlation Coefficient?"Locality-Specific Reasoning: DialoGPT Large exhibits the most negative Spearman's Correlation Coefficient.

Source: Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP)

DOI: 10.18653/v1/2024.findings-emnlp.904

Authors: Chuhan Li, Ziyao Shangguan, Yilun Zhao, Deyuan Li, Yixin Liu, Arman Cohan

Institutions: Yale University, Allen Institute for Artificial Intelligence

Language: English

Topic: Multimodal Machine Learning Applications

Subfield: Computer Vision and Pattern Recognition

Field: Computer Science

Domain: Physical Sciences

Open Access status: hybrid

Funders: Microsoft Research, Tata Consultancy Services, Tata Sons

OpenAlex ID: W4404781475