Work

HandySense: A Multimodal Collection System for Human Two-Handed Dexterous Manipulation

Year: 2024

Type: conference-paper

Abstract: Humanoid robots with dexterous hands have gained significant attention due to their manipulation capabilities. Recent advancements are driven by large-scale real robot data and teleoperation technology, enabling precise operation demonstrations and smooth trajectories. Common methods like virtual reality devices, cameras, wearable gloves, and custom hardware face the inability to capture real information about human-object contact, such as tactile information. In this study, we present HandySense, a multimodal system integrating visual, tactile, motion, and spatial perception for robust and comprehensive two-handed manipulation tracking. HandySense includes RGB-D cameras, visual-inertial tracking cameras, and a motion capture glove with fingertip tactile sensors. Our framework achieved 99.45% accuracy in classifying 12 task stages, exhibiting the potential for large-scale human demonstration data collection and representing a pivotal step towards empowering humanoid robots to execute complex manipulations.

Source: Proceedings of the 28th Annual International Conference on Mobile Computing And Networking

DOI: 10.1145/3636534.3694728

Authors: Shilong Mu, Jingyang Wang, Xinyue Chai, Xingting Li, Tong Wu, Wenbo Ding

Institutions: Tsinghua–Berkeley Shenzhen Institute, Tsinghua Shenzhen International Graduate School, Tsinghua University, University of Science and Technology Beijing

Language: English

Topic: Robot Manipulation and Learning

Subfield: Control and Systems Engineering

Field: Engineering

Domain: Physical Sciences

Open Access status: hybrid

OpenAlex ID: W4405014441

FWCI:

Cites: 32

Cited by:

Related to: 10