Large vision-language models enabled novel objects 6D pose estimation for human-robot collaboration
Year: 2025
Type: article
Abstract: Six-dimensional (6D) pose estimation is essential for robotic manipulation tasks, especially in human-robot collaborative manufacturing environments. Recently, 6D pose estimation has been extended from seen objects to novel objects due to the frequent encounters with unfamiliar items in real-life scenarios. This paper presents a three-stage pipeline for 6D pose estimation of previously unseen objects, leveraging the capabilities of large vision-language models. Our approach consists of vision-language model-based object detection and segmentation, mask selection with pose hypothesis generated from CAD models, and refinement and scoring of pose candidates. We evaluate our method on the YCB-Video dataset, achieving a state-of-the-art Average Recall (AR) score of 75.8, demonstrating its effectiveness in accurately estimating 6D poses for a diverse range of objects. The effectiveness of each operation stage is investigated in the ablation study. To validate the practical applicability of our approach, we conduct case studies on a real-world robotic platform, focusing on object pick-up tasks by integrating our 6D pose estimation pipeline with human intention prediction and task analyzation algorithms. Results show that the proposed method significantly enhances the robot's ability to interact with novel objects in unstructured environments. Our work contributes to the field of human-robot collaborative manufacturing by introducing a flexible, generalizable approach to 6D pose estimation, enabling robots to adapt to new objects without requiring extensive retraining—a vital capability for advancing human-robot collaboration in dynamic environments. More information can be found in the project GitHub page: https://github.com/WanqingXia/HRC_DetAnyPose.
Authors: Wanqing Xia, Hao Zheng, Weiliang Xu, Xun Xu
Institution: University of Auckland
Language: English
Subfield: Control and Systems Engineering
Field: Engineering
Domain: Physical Sciences
Open Access status: hybrid
Estimated APC paid: $4,540
OpenAlex ID: W4409834879