Year: 2024
Type: dissertation
Abstract: In recent years, large language models have developed a strong ability to generate texts. However, it is still challenging for a large language model to understand 3D scenes. In 2023, ConceptGraphs pipeline was proposed. It can build a scene graph as the 3D scene representation and bridge the gap between the 3D scene and the large language model. In this thesis project, we investigate how this pipeline works and evaluate the workflow of this pipeline. The evaluation contains two parts as the ConceptGraphs pipeline can be divided into two parts. First, we evaluate the 3D scene reconstruction ability of ConceptGraphs pipeline with the AI2-THOR dataset. Second, we evaluate the spatial reasoning ability of LLaVA and GPT with the 3D representation form that could be obtained from the first part. The experiment results of the first part show that the 3D reconstruction ability needs more development. The results of the second part show that language models can have a better performance when there are not many objects in the scene. The results also show that for LLaVA, the best performance happens when the input contains the scene graph and image at the same time, while for GPT, the best performance happens when the input contains only the scene graph.
Author: Qian, Siyi
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W6991210175