Work

Learning Deep Structure-Preserving Image-Text Embeddings

Year: 2016

Type: conference-paper

Abstract: This paper proposes a method for learning joint embeddings of images and text using a two-branch neural network with multiple layers of linear projections followed by nonlinearities. The network is trained using a largemargin objective that combines cross-view ranking constraints with within-view neighborhood structure preservation constraints inspired by metric learning literature. Extensive experiments show that our approach gains significant improvements in accuracy for image-to-text and textto-image retrieval. Our method achieves new state-of-theart results on the Flickr30K and MSCOCO image-sentence datasets and shows promise on the new task of phrase localization on the Flickr30K Entities dataset.

Source: IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings

DOI: 10.1109/cvpr.2016.541

Authors: Liwei Wang, Yin Li, Svetlana Lazebnik

Institutions: University of Illinois Urbana-Champaign, Georgia Institute of Technology

Language: English

Topic: Multimodal Machine Learning Applications

Subfield: Computer Vision and Pattern Recognition

Field: Computer Science

Domain: Physical Sciences

Open Access status: green

Funders: National Science Foundation, Alfred P. Sloan Foundation

Award: CIF: Medium:Collaborative Research: Nonasymptotic Analysis of Feature-Rich Decision Problems with Applications to Computer Vision