HowToNarrate: A General-Domain Benchmark for Synchronized Video Narration with External Knowledge
Year: 2026
Type: conference-paper
Abstract: We present HowToNarrate, the first generaldomain benchmark for Synchronized Video Narration.The benchmark contains 3.2K videos across seven domains, segmented into 37.5K clips with aligned narrations and associated external knowledge.Effective narration requires models to understand visual scenes, incorporate relevant knowledge, and produce coherent, length-appropriate descriptions.We systematically benchmark current Multimodal LLMs (MLLMs) on these abilities.Our analysis shows that existing MLLMs overemphasize knowledge retrieval while largely neglecting prior context (receiving less than 10% attention).Moreover, they often conflate narration context with external knowledge, leading to redundancy and incoherence.To mitigate these issues, we propose VideoNarrationAgent, a multi-agent framework that combines context compression, knowledge retrieval, and narration generation.Experiments demonstrate that our method significantly improves MLLM performance.Furthermore, instruction tuning on HowToNarrate enhances both context-awareness and length control, boosting Qwen2.5-VL'sscore from 25 to 84.Our dataset and codes are released at https://github. com/wangxueyan666/HowToNarrate.
Authors: Xueyan Wang, Dingyi Yang, Qin Jin
Institutions: Renmin University of China, Nanyang Technological University
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W7166830911