Work

VisualWebInstruct: Scaling up Multimodal Instruction Data through Web Search

Year: 2025

Type: conference-paper

Abstract: Vision-Language Models have made significant progress on many perception-focused tasks.However, their progress on reasoningfocused tasks remains limited due to the lack of high-quality and diverse training data.In this work, we aim to address the scarcity of reasoning-focused multimodal datasets.We propose VisualWebInstruct, a novel approach that leverages search engines to create a diverse and high-quality dataset spanning multiple disciplines, including mathematics, physics, finance, and chemistry, etc. Starting with a meticulously selected set of 30,000 seed images, we employ Google Image Search to identify websites containing similar images.We collect and process HTML data from over 700K unique URLs.Through a pipeline of content extraction, filtering, and synthesis, we construct a dataset of approximately 900K question-answer (QA) pairs, with 40% consisting of visual QA pairs and the remaining comprising text-based QA pairs.Models finetuned on VisualWebInstruct demonstrate significant performance improvements: (1) finetuning on Llava-OV results in 10-20 absolute points improvement across benchmarks, and (2) fine-tuning from MAmmoTH-VL yields a 5 absolute points gain across benchmarks.Our best model, MAmmoTH-VL2, achieves the best known performance with SFT without RL within the 10B parameter class on MMMU-Pro (40.7),MathVerse (42.6), and DynaMath (55.7).These results highlight the effectiveness of our dataset in enhancing the reasoning capabilities of vision-language models for complex multimodal tasks.

Source: Conference on Empirical Methods in Natural Language Processing (EMNLP)

DOI: 10.18653/v1/2025.emnlp-main.72