Optimal and Robust Category-Level Perception: Object Pose and Shape Estimation From 2-D and 3-D Semantic Keypoints
Year: 2023
Type: article
Abstract: In this article, we consider acategory-level perceptionproblem, where one is given 2-D or 3-D sensor data picturing an object of a given category (e.g., a car) and has to reconstruct the 3-D pose and shape of the object despite intraclass variability (i.e., different car models have different shapes). We consider anactive shape model, where—for an object category—we are given a library of potential computer-aided design models describing objects in that category, and we adopt a standard formulation where pose and shape are estimated from 2-D or 3-D keypoints via nonconvex optimization. Our first contribution is to developPACE3D$^{\star }$andPACE2D$^{\star }$, the firstcertifiably optimalsolvers for pose and shape estimation using 3-D and 2-D keypoints, respectively. Both the solvers rely on the design of tight (i.e., exact) semidefinite relaxations. Our second contribution is to develop outlier-robust versions of both the solvers, namedPACE3D# andPACE2D#. Toward this goal, we propose ROBIN(Reject Outliers Based on INvariants), a general graph-theoretic framework to prune outliers, which usescompatibility hypergraphsto model measurements' compatibility. We show that in category-level perception problems, these hypergraphs can be built from the winding orders of the keypoints (in 2-D) or their convex hulls (in 3-D), and many outliers can be filtered out via maximum hyperclique computation. The last contribution is an extensive experimental evaluation. Besides providing an ablation study on simulated datasets and on thePASCAL3D+ dataset, we combine our solver with a deep keypoint detector and show thatPACE3D# improves over the state of the art in vehicle pose estimation in theApolloScapedatasets, and its runtime is compatible with practical applications. We release our code athttps://github.com/MIT-SPARK/PACE.
Source: IEEE Transactions on Robotics
Authors: Jingnan Shi, Heng Yang, Luca Carlone
Institutions: Massachusetts Institute of Technology, Harvard University
Language: English
Subfield: Computational Mechanics
Field: Engineering
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W4386320489