Uncertainty-aware perception, localization and mapping for autonomous driving
Year: 2026
Type: dissertation
Abstract: Advances in deep learning have revolutionized scene understanding in recent years, achieving unprecedented performance levels. As a result, nearly all autonomous vehicle perception systems now rely on learning-based Deep Neural Networks (DNNs). These methods are trained on curated datasets and evaluated on test sets using task-specific metrics. This approach faces two significant limitations. First, the training and test datasets typically share similar data distributions and conditions, which may differ substantially from those encountered during real-world deployment. Real-world scenarios often present objects and situations absent from the training data. Second, evaluation metrics frequently focus solely on task-specific performance and fail to account for the network's confidence in the evaluation. Consequently, these methods tend to produce deterministic or overly confident outputs without a reliable uncertainty estimation, posing safety risks in critical applications such as autonomous driving. Moreover, the absence of uncertainty estimations in modern perception systems complicates their integration with classical probabilistic methods, that forms the foundation of successful autonomous systems across various applications, particularly in downstream tasks like localization and mapping. In this thesis, we focus on developing perception methods that enable real-time uncertainty estimation and leverage this uncertainty to integrate perception information with probabilistic approaches for localization and mapping. First, we introduce a new perception task that aims to achieve uncertainty-aware holistic scene understanding. To facilitate this, we provide a formal definition of novel metrics designed for holistic, uncertainty-aware perception evaluation, along with a DNN-based architecture tailored to perform this task using camera images. Next, we focus on the segmentation of 3D LiDAR point clouds, given the indispensable role the LiDAR sensor plays for 3D scene understanding in an autonomous vehicle. As the LiDAR point cloud brings challenges like sparse and unstructured data, we develop specialized methods to leverage 3D information from LiDAR point cloud into 2D convolutional neural networks and set the new benchmark for LiDAR panoptic segmentation. Furthermore, we present the first method that combines holistic perception capabilities with uncertainty estimation for LiDAR point clouds. By leveraging both the 3D spatial information of the point cloud and prediction uncertainties, our approach resolves ambiguities in segmenting challenging objects, significantly improving perception performance. Finally, we focus on the downstream tasks by utilizing the perception uncertainty as a bridge between modern perception and probabilistic approaches for localization and mapping. Especially in the final chapter, we showcase that utilizing perception uncertainties can improve mapping and localization performance. Moreover, our approach also propagates the perception uncertainties into the final mapping and localization results, thus providing their respective uncertainties. Our methods demonstrate that high-accuracy, uncertainty-aware perception based on deep learning is achievable, and perception uncertainty is essential for a robust downstream performance.
DOI: 10.6094/unifr/279476
Author: Sirohi, Kshitij
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7147585983