Year: 2019
Type: conference-paper
Abstract: In this paper, we propose a new approach to learn multimodal multilingual embeddings for matching images and their relevant captions in two languages.We combine two existing objective functions to make images and captions close in a joint embedding space while adapting the alignment of word embeddings between existing languages in our model.We show that our approach enables better generalization, achieving state-of-the-art performance in text-to-image and image-to-text retrieval task, and caption-caption similarity task.Two multimodal multilingual datasets are used for evaluation: Multi30k with German and English captions and Microsoft-COCO with English and Japanese captions.
DOI: 10.18653/v1/d19-6605
Institution: École Polytechnique Fédérale de Lausanne
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
Funder: Nvidia
OpenAlex ID: W2979866563