Semantic Querying of Reconstructed 3D Environments Using Pre-trained 2D Foundation Models
Year: 2024
Type: dissertation
Abstract: Understanding and interacting with complex 3D environments is increasingly important in robotics, virtual reality, and autonomous systems. To address this need, this thesis project aims to develop a system that reconstructs a 3D model of the environment enhanced with semantic features. We project in the 3D model the semantic features extracted using pre-trained 2D foundational models, such as Segment Anything and OpenCLIP. This allows open-vocabulary queries about the environment in natural language that are not restricted by predefined categories or information and without requiring additional training. The system is tested on real RGB-D data captured from a HoloLens 2 device.
Author: FASANO, PAOLO
Language: English
Subfield: Computational Mechanics
Field: Engineering
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7024353537
Cites:
Cited by:
Related to: