Work

General In-Hand Object Rotation with Vision and Touch

Year: 2023

Type: preprint

Abstract: We introduce RotateIt, a system that enables fingertip-based object rotation along multiple axes by leveraging multimodal sensory inputs. Our system is trained in simulation, where it has access to ground-truth object shapes and physical properties. Then we distill it to operate on realistic yet noisy simulated visuotactile and proprioceptive sensory inputs. These multimodal inputs are fused via a visuotactile transformer, enabling online inference of object shapes and physical properties during deployment. We show significant performance improvements over prior methods and the importance of visual and tactile sensing.

Source: arXiv (Cornell University)

DOI: 10.48550/arxiv.2309.09979

Authors: Qi, Haozhi, Yi, Brent, Suresh, Sudharshan, Lambeta, Mike, Ma, Yi, Calandra, Roberto, Malik, Jitendra

Institutions: Meta (United States), University of California, Berkeley, Carnegie Mellon University, Centre for Tactile Internet with Human-in-the-Loop, Technische Universität Dresden

Language: English

Topic: Robot Manipulation and Learning

Subfield: Control and Systems Engineering

Field: Engineering

Domain: Physical Sciences

Open Access status: green

Funders: National Science Foundation, Deutsche Forschungsgemeinschaft, Technische Universität Dresden, Defense Advanced Research Projects Agency, Office of Naval Research

Awards: EXC 2050: Centre for Tactile Internet with HumanintheLoop (CeTI), G1933938177, G2668607730, G4021765181, G4720003262, G5154814024, Graduate Research Fellowship Program (GRFP), G8751405378, G8876996369

OpenAlex ID: W4386874604

Cites: 12

Cited by: 44

Related to: 10