Work

Generating Semantically Precise Scene Graphs from Textual Descriptions for Improved Image Retrieval

Year: 2015

Type: conference-paper

Abstract: Semantically complex queries which include attributes of objects and relations between objects still pose a major challenge to image retrieval systems.Recent work in computer vision has shown that a graph-based semantic representation called a scene graph is an effective representation for very detailed image descriptions and for complex queries for retrieval.In this paper, we show that scene graphs can be effectively created automatically from a natural language scene description.We present a rule-based and a classifierbased scene graph parser whose output can be used for image retrieval.We show that including relations and attributes in the query graph outperforms a model that only considers objects and that using the output of our parsers is almost as effective as using human-constructed scene graphs (Recall@10 of 27.1% vs. 33.4%).Additionally, we demonstrate the general usefulness of parsing to scene graphs by showing that the output can also be used to generate 3D scenes.

Source: Workshop on Vision and Language (VL)

DOI: 10.18653/v1/w15-2812

Authors: Sebastian Schuster, Ranjay Krishna, Angel Chang, Li Fei-Fei, Christopher D. Manning

Institution: Stanford University

Language: English

Topic: Multimodal Machine Learning Applications

Subfield: Computer Vision and Pattern Recognition

Field: Computer Science

Domain: Physical Sciences

Open Access status: hybrid

Funders: Brown Institute for Media Innovation, Defense Advanced Research Projects Agency, Advanced Research Projects Agency

OpenAlex ID: W2250378130