Work

MUTAN: Multimodal Tucker Fusion for Visual Question Answering

Year: 2017

Type: conference-paper

Abstract: Bilinear models provide an appealing framework for mixing and merging information in Visual Question Answering (VQA) tasks. They help to learn high level associations between question meaning and visual concepts in the image, but they suffer from huge dimensionality issues. We introduce MUTAN, a multimodal tensor-based Tucker decomposition to efficiently parametrize bilinear interactions between visual and textual representations. Additionally to the Tucker framework, we design a low-rank matrix-based decomposition to explicitly constrain the interaction rank. With MUTAN, we control the complexity of the merging scheme while keeping nice interpretable fusion relations. We show how the Tucker decomposition framework generalizes some of the latest VQA architectures, providing state-of-the-art results.

Source: IEEE/CVF International Conference on Computer Vision (ICCV)

DOI: 10.1109/iccv.2017.285

Authors: Hedi Ben-younes, Remi Cadene, Matthieu Cord, Nicolas Thome

Institutions: Sorbonne Université, LIP6, Conservatoire National des Arts et Métiers, Centre d'études et de recherche en informatique et communications

Language: English

Topic: Multimodal Machine Learning Applications

Subfield: Computer Vision and Pattern Recognition

Field: Computer Science

Domain: Physical Sciences

Open Access status: green

Funder: Agence Nationale de la Recherche

Award: Interactions humain/Machine/Humain intelligentes dans la société numérique