Year: 2024
Type: dissertation
Abstract: This doctoral thesis explores advanced deep learning methods for three pivotal tasks in 3D computer vision: point matching, visual localization, and 3D reconstruction. These tasks are crucial for enabling machines to perceive and understand 3D environments, which is essential for applications ranging from virtual reality to robotics and autonomous driving. In point matching, this thesis investigates learning-based, visual descriptor-free matching pipelines that leverage geometric and color cues in conjunction with Graph Neural Networks. This approach significantly enhances the accuracy of 2D-3D keypoint matching while reducing memory usage and improving data privacy. For visual localization, the thesis introduces two hierarchical scene coordinate network architectures to establish dense 2D-3D matches for accurate 6-DoF camera pose estimation. These architectures incorporate conditioning mechanism and transformer to encode global context to local patches, overcoming scene ambiguities. Additionally, a novel few-shot learning setting is proposed, which reduces the training load for scene coordinate regression and improves training efficiency from days to minutes. A significant contribution of the thesis is the development of an end-to-end dense unconstrained 3D reconstruction pipeline based on Vision Transformers. This pipeline directly regresses point coordinates from image pairs without relying on camera parameters, simplifying traditional 3D reconstruction methods and introducing a unified framework for monocular and binocular reconstruction tasks. The thesis also explores methods to improve local feature matching by calculating the curvature of local 3D surface patches for detected points, enhancing matching accuracy with off-the-shelf learned matchers. Furthermore, it addresses the challenge of continual learning for visual localization by proposing an experience-replay-based baseline to prevent catastrophic forgetting and reduce computational and storage costs. Throughout the thesis, the importance of end-to-end learning is emphasized, where models are trained to directly produce desired outputs from raw input data. This paradigm shift can simplify the development pipelines and enhance the adaptability and scalability of 3D computer vision systems. By integrating deep learning with traditional geometric principles, this research provides a comprehensive framework for addressing key challenges in point matching, visual localization, and 3D reconstruction.
Source: Aaltodoc (Aalto University)
Author: Wang, Shuzhe
Institution: Aalto University
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7008899760
