Year: 2025
Type: preprint
Abstract: AI agents with advanced reasoning and tool-use capabilities have demonstrated impressive performance in web browsing for deep search. However, existing benchmarks such as BrowseComp primarily focus on textual content, overlooking the prevalence of multimodal content. To bridge this gap, we introduce MM-BrowseComp, a novel benchmark comprising 400 challenging, hand-crafted questions designed to evaluate multimodal retrieval and reasoning capabilities. Unlike prior work, MM-BrowseComp incorporates visual prompts and necessitates the extraction of key evidence from web images and videos to complete questions, rendering text-only approaches insufficient. Additionally, we provide a verified checklist for each question, enabling fine-grained analysis of multimodal dependencies and reasoning paths. Our comprehensive evaluation of 27 state-of-the-art models reveals that even leading models like GPT-5-High with tools achieve only 24.25\% accuracy, highlighting the suboptimal multimodal browsing capabilities, establishing MM-BrowseComp as a rigorous new standard for the field.
Source: arXiv (Cornell University)
Authors: Li, Shilong, Bu, Xingyuan, Wang, Wenjie, Liu, Jiaheng, Dong, Jun, He, Haoyang, Lu, Hao, Zhang, Haozhe, Jing, Chenchen, Li, Zhen, Li, Chuanhao, Tian, Jiayi, Zhang, Chenchen, Peng, Tianhao, He, Yancheng, Gu, Jihao, Huang, Hui, Zhou, Donghao, Zhang, Yuanxing, Yang, Jian, Zhang, Ge, Huang, Wenhao, Zhang, Zhaoxiang, Yang, Qiangpeng, Wen, Shilei
Language: English
Subfield: Information Systems
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W4414530236
Cites:
Cited by:
Related to: