Year: 2025
Type: conference-paper
Abstract: Multi-grained alignment for text-video retrieval takes advantage of both coarse- and fine-grained semantic alignments, which can enhance the representation capabilities of cross-modal information retrieval. However, the current research fails to fully explore the informative semantics embedded in fine-grained levels of both text and video spaces. In this work, we propose a Selective Multi-Grained Alignment framework, termed SMA, which achieves video-sentence and object-phrase alignments through coarse- and fine-grained alignment modules. Specifically, we introduce a token aggregation module, which implicitly selects important patches per frame and words in text and then aggregates them to generate object-level features for frames and phrase-level features for text. Moreover, we apply a similarity-aware selection module to select keyframes in video efficiently. Extensive experiments on four benchmarks demonstrate that our SMA achieves superior performance on MSR-VTT (52.8%), MSVD (48.2%), ActivityNet (48.9%), and DiDeMo (50.1%). The code of this paper is available at https://github.com/bzy-source/SMA.
Authors: Ziyi Bian, Cong Jiang, Fangzhi Zhu, Zheng Zhang
Institution: Harbin Institute of Technology
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
Awards: G1943805037, G3206267056
OpenAlex ID: W4410638172