Burn After Reading: Do Multimodal Large Language Models Truly Capture Order of Events in Image Sequences?
Year: 2025
Type: conference-paper
Abstract: This paper introduces the TempVS benchmark, which focuses on temporal grounding and reasoning capabilities of Multimodal Large Language Models (MLLMs) in image sequences.TempVS consists of three main tests (i.e., event relation inference, sentence ordering and image ordering), each accompanied with a basic grounding test.TempVS requires MLLMs to rely on both visual and linguistic modalities to understand the temporal order of events.We evaluate 38 state-of-the-art MLLMs, demonstrating that models struggle to solve Tem-pVS, with a substantial performance gap compared to human capabilities.We also provide fine-grained insights that suggest promising directions for future research.Our Tem-pVS benchmark data and code are available at https://github.com/yjsong22/TempVS. Two-event Relation InferenceTrue: Eddy opened the door of the house after Rody waved to Eddy from his green four-wheel bike.False: Rody waved to Eddy from his green four-wheel bike after Eddy opened the door of
Authors: Yingjin Song, Yupei Du, Denis Paperno, Albert Gatt
Institution: Utrecht University
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
OpenAlex ID: W4412944654