Work

Learning Joint Embedding with Multimodal Cues for Cross-Modal Video-Text Retrieval

Year: 2018

Type: conference-paper

Abstract: Constructing a joint representation invariant across different modalities (e.g., video, language) is of significant importance in many multimedia applications. While there are a number of recent successes in developing effective image-text retrieval methods by learning joint representations, the video-text retrieval task, however, has not been explored to its fullest extent. In this paper, we study how to effectively utilize available multimodal cues from videos for the cross-modal video-text retrieval task. Based on our analysis, we propose a novel framework that simultaneously utilizes multi-modal features (different visual characteristics, audio inputs, and text) by a fusion strategy for efficient retrieval. Furthermore, we explore several loss functions in training the embedding and propose a modified pairwise ranking loss for the task. Experiments on MSVD and MSR-VTT datasets demonstrate that our method achieves significant performance gain compared to the state-of-the-art approaches.

Source: International Conference on Multimedia Retrieval

DOI: 10.1145/3206025.3206064

Authors: Niluthpol Chowdhury Mithun, Juncheng Li, Florian Metze, Amit K. Roy-Chowdhury

Institutions: University of California, Riverside, Robert Bosch (United States), Carnegie Mellon University

Language: English

Topic: Multimodal Machine Learning Applications

Subfield: Computer Vision and Pattern Recognition

Field: Computer Science

Domain: Physical Sciences

Open Access status: bronze

Funder: National Science Foundation

Awards: CPS: Synergy: Collaborative Research: Extracting Time-Critical Situational Awareness from Resource Constrained Networks, EAGER: Joint Modeling and Querying of Social Media and Video Data