Year: 2025
Type: conference-paper
Abstract: We introduce SCIVER, the first benchmark specifically designed to evaluate the ability of foundation models to verify claims within a multimodal scientific context.SCIVER consists of 3,000 expert-annotated examples over 1,113 scientific papers, covering four subsets, each representing a common reasoning type in multimodal scientific claim verification.To enable fine-grained evaluation, each example includes expert-annotated supporting evidence.We assess the performance of 21 state-of-the-art multimodal foundation models, including o4mini, Gemini-2.5-Flash,Llama-3.2-Vision, and Qwen2.5-VL.Our experiment reveals a substantial performance gap between these models and human experts on SCIVER.Through an in-depth analysis of retrieval-augmented generation (RAG), and human-conducted error evaluations, we identify critical limitations in current open-source models, offering key insights to advance models' comprehension and reasoning in multimodal scientific literature tasks.
Authors: Chengye Wang, Yifei Shen, Zexi Kuang, Arman Cohan, Yilun Zhao
Institution: Yale University
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W4412945493