Work

Connecting Touch and Vision via Cross-Modal Prediction

Year: 2019
Type: conference-paper
Abstract: Humans perceive the world using multi-modal sensory inputs such as vision, audition, and touch. In this work, we investigate the cross-modal connection between vision and touch. The main challenge in this cross-domain modeling task lies in the significant scale discrepancy between the two: while our eyes perceive an entire visual scene at once, humans can only feel a small region of an object at any given moment. To connect vision and touch, we introduce new tasks of synthesizing plausible tactile signals from visual inputs as well as imagining how we interact with objects given tactile data as input. To accomplish our goals, we first equip robots with both visual and tactile sensors and collect a large-scale dataset of corresponding vision and tactile image sequences. To close the scale gap, we present a new conditional adversarial model that incorporates the scale and location information of the touch. Human perceptual studies demonstrate that our model can produce realistic visual images from tactile data and vice versa. Finally, we present both qualitative and quantitative experimental results regarding different system designs, as well as visualizing the learned representations of our model.
Language: English
Field: Psychology
Open Access status: green
Award: IN JUNE 2012 MIT DECIDED TO FORM A TEAM TO COMPETE IN THE DARPA ROBOTICS CHALLENGE DRC DESPITE HAVING NEVER WORKED ON HUMANOID ROBOTS BEFORE. THANKS TO A PRINCIPLED OPTIMIZATION-BASED APPROACH TO PERCEPTION PLANNING AND CONTROL THAT EMPHASIZED TASK LEVEL AUTONOMY A MANAGEMENT STRATEGY THAT EMPHASIZED RAPID PROTOTYPING OF RESEARCH ALGORITHMS COMBINED WITH RIGOROUS SOFTWARE ENGINEERING PRACTICES AND AN INCREDIBLY STRONG TEAM WE WERE ABLE TO RAPIDLY DEVELOP FROM SCRATCH ALL OF THE INFRASTRUCTURE AND ALGORITHMS REQUIRED TO COMPETE WITH THE TOP HUMANOID ROBOTICS LABS IN THE WORLD. THIS TEAM IS JUST GETTING STARTED. HOSTING R5 AT MIT WOULD PROVIDE THE OPPORTUNITY TO TRANSITION OUR MOMENTUM ONTO AN INCREDIBLE NEW HARDWARE PLATFORM AND TOWARDS A SERIES OF NASA-RELEVANT CHALLENGES WHICH MATCH OUR CORE STRENGTHS IN AUTONOMY. OUR DRC CODEBASE IS ALREADY BEING USED ON R5 BY THE TEAM IN EDINBURGH CO-LED BY FORMER TEAM MIT LEAD; THIS TIGHT EXISTING INTEGRATION WITH THE EDINBURGH TEAM IS A MAJOR OPPORTUNITY THAT WOULD BENEFIT BOTH MIT AND EDINBURGH'S EFFORTS. THE EXPECTED TASKS FOR VALIDATION OUTLINED IN THE SOLITICATION ARE WELL-ALIGNED WITH OUR CORE CAPABILITIES IN PERCEPTION PLANNING AND CONTROL BUT ALSO ADD EXCITING NEW DEMANDS SUCH AS BEING ABLE TO PERFORM ROBUST LOCOMOTION IN CONFINED SPACES AND MORE DEXTEROUS MANIPULATION EVEN WITH NON-RIGID STRUCTURES. THE DRC TASKS HAD VERY LITTLE UNCERTAINTY AND THEREFORE REQUIRED VERY LITTLE IN THE WAY OF HIGH-LEVEL (OR MISSION-LEVEL) AUTONOMY. IN ORDER TO EVALUATE HOW THE STATE-OF-THE-ART IN MISSION-LEVEL AUTONOMY CAN ADDRESS THE EXPECTED TASKS FOR THE SPACE ROBOTICS CHALLENGE SRC THIS PROPOSAL PAIRS THE MIT DRC TEAM WITH CO-PIS TO INTEGRATE THEIR STATE-OF-THE-ART MISSION-LEVEL AUTONOMY CAPABILITIES WITH OUR DRC PERCEPTION PLANNING AND CONTROL ALGORITHMS. HOSTING R5 IS A FANTASTIC OPPORTUNITY FOR MIT WE AIM TO PROVIDE OUR STUDENTS WITH ACCESS TO THE MOST ADVANCED HARDWARE IN THE WORLD AND TO FOCUS THEIR BASIC RESEARCH TOWARDS IMPORTANT PROBLEMS OF NATIONAL NEED. HOSTING R5 AT MIT CAN ALSO HELP THE SRC PROVIDING AN EXCITING WORLD-CLASS VENUE FOR THE COMPETING TEAMS TO COME AND DO THEIR RESEARCH.
OpenAlex ID: W2952866402
FWCI: 46.42
Cites: 59
Cited by: 206
Related to: 10