A Crossmodal Approach to Multimodal Fusion in Video Hyperlinking
Year: 2018
Type: article
Abstract: With the recent resurgence of neural networks and the proliferation of massive amounts of unlabeled multimodal data, recommendation systems and multimodal retrieval systems based on continuous representation spaces and deep learning methods are becoming of great interest. Multimodal representations are typically obtained with autoencoders that reconstruct multimodal data. In this article, we describe an alternative method to perform high-level multimodal fusion that leverages crossmodal translation by means of symmetrical encoders cast into a bidirectional deep neural network (BiDNN). Using the lessons learned from multimodal retrieval, we present a BiDNN-based system that performs video hyperlinking and recommends interesting video segments to a viewer. Results established using TRECVIDs 2016 video hyperlinking benchmarking initiative show that our method obtained the best score, thus defining the state of the art.
Source: IEEE Multimedia
Institutions: Institut national de recherche en sciences et technologies du numérique, Institut de Recherche en Informatique et Systèmes Aléatoires, Institut National des Sciences Appliquées de Rennes, LINKMEDIA: Creating and exploiting explicit links between multimedia fragments, Centre National de la Recherche Scientifique
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W2885641986