Year: 2016
Type: preprint
Abstract: This paper presents an approach for answering fill-in-the-blank multiple choice questions from the Visual Madlibs dataset. Instead of generic and commonly used representations trained on the ImageNet classification task, our approach employs a combination of networks trained for specialized tasks such as scene recognition, person activity classification, and attribute prediction. We also present a method for localizing phrases from candidate answers in order to provide spatial support for feature extraction. We map each of these features, together with candidate answers, to a joint embedding space through normalized canonical correlation analysis (nCCA). Finally, we solve an optimization problem to learn to combine scores from nCCA models trained on multiple cues to select the best answer. Extensive experimental results show a significant improvement over the previous state of the art and confirm that answering questions from a wide range of types benefits from examining a variety of image cues and carefully choosing the spatial support for feature extraction.
Source: arXiv (Cornell University)
Institution: Sapienza University of Rome
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
Awards: CI-New: Collaborative Research: Federated Data Set Infrastructure for Recognition Problems in Computer Vision, RI: Medium: Collaborative Research: Text-to-Image Reference Resolution for Image Understanding and Manipulation, CAREER: Toward a General Framework for Words and Pictures, CIF: Medium:Collaborative Research: Nonasymptotic Analysis of Feature-Rich Decision Problems with Applications to Computer Vision, BIGDATA: F: Collaborative Research: From Visual Data to Visual Understanding, CAREER: Situated Recognition: Learning to understand our local visual environment, RI: Medium: Collaborative Research: Text-to-Image Reference Resolution for Image Understanding and Manipulation
OpenAlex ID: W4299639487