Year: 2026
Type: conference-paper
Abstract: Despite the impressive progress of recent MLLMs on spatio-temporal video grounding (STVG), existing evaluations and training data focus primarily on simple queries.They largely overlook the compositional queries prevalent in real-world scenarios, where a target must be disambiguated by jointly reasoning about its attributes and relations to other entities.To bridge this gap, we propose Compositional Spatio-Temporal Video Grounding (COMPSTVG), a task that requires models to process complex textual queries where every intertwined attribute and relational cue is essential for disambiguation.To facilitate this task at scale, we build a synthetic data engine that leverages a spatio-temporal scene graph as a difficulty measure and casts difficulty-controlled query synthesis as a constraint programming problem, producing difficulty-graded data for both evaluation and training.Built on this engine, we introduce STVG-COMPBENCH, a benchmark stratified by explicit difficulty levels that jointly capture temporal complexity and spatial interference.Evaluating 11 representative STVG models on STVG-COMPBENCH reveals that current models perform poorly on compositional queries, exhibiting a sharp performance drop that is typically obscured by overall dataset-level averages.We further construct synthetic training data and propose CURRSTVG, a curriculum reinforcement learning framework that delivers consistent gains, with the largest improvements observed on the most challenging compositional queries.
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: bronze
OpenAlex ID: W7204875952