Enabling Robot Learning Through Capable Hardware: Preliminary Studies In Multimodal Tactile Sensing, Bimanual Mobile Manipulation, And Semantic Demonstration Synthesis
Year: 2026
Type: article
Abstract: Physical AI is continuing to advance rapidly, but its current capabilities are limited by the underlying hardware and perception stack. Robot learning, in particular, is often bottlenecked by current sensing capabilities, which limit the amount of information available to the learning algorithm and the range of tasks that can be attempted. Most robotics operations are contact-rich, meaning it requires robotic systems to have robust tactile sensing and whole-body perception to understand the state of the world during grasping and object manipulation. In this thesis, I present how there is a need to develop more capable hardware and perception capabilities to enable robot learning in robotic dexterous manipulation and mobile manipulation. To show this, I discuss how DenseTact sensors can be improved to provide more reliable tactile sensing in robotic end effectors, how we can utilize the TidyBot++ mobile manipulator platform to enable bimanual mobile manipulation research and education, and how we can use a single RGB video of a human performing a manipulation task to generate a large set of robot demonstrations for downstream policy learning. DenseTact is an optical tactile sensor that provides high-resolution tactile sensing for robotic manipulation. It’s current capabilities include shape and force sensing. Currently, it has reliability limitations in terms of its manufacturing process, calibration pipeline, and overall hardware design. Additionally, it lacks the ability to sense vibration and temperature, which are important for many manipulation tasks. As a result, I am presenting the current progress to move DenseTact to be more antrhropomorphic, reliable in manufacturing and calibration, and to have multimodal sensing capabilities. The strategy for this is to introduce a new version of the DenseTact Mini sensor, an automated calibration pipeline that builds on TensorTouch, and prototyping hardware and software for localized vibration and distributed temperature sensing modalities co-localized with the existing shape and force channels. MR. CRABS is a holonomic bimanual mobile manipulator that is built on the open-source TidyBot++ base. It includes two Realsense D435 depth cameras, one of which has an active pan-and-tilt mechanism for near-ground perception. It also includes a top LIDAR for long-range perception. It also utilizes two Trossen Robotics Widow-X arms. MR. CRABS is designed both as a research platform for bimanual mobile manipulation and as the next-generation educational platform for Stanford’s ME 326, replacing the LoCoBots used previously. The more capable hardware proved to be essential to allow students to complete the course’s final project, allowing them to better navigate the scene and perform more complex manipulation tasks that cannot be done with the LoCoBots. I present that more teams were able to complete more of the final project, and the projects were more complex than in previous years. Finally, I present Semantics as a Present (SAP), a demonstration-augmentation pipeline that turns a single RGB video of a human performing a manipulation task into an N ×M set of robot demonstrations spanning a family of pre-scanned cousin objects, with no depth sensor, no teleoperation, and no per-object annotation. The method recovers per-frame SE(3) object pose from RGB alone via distilled-DINO and depth-ranking losses, grafts the recovered motion onto each cousin via regionweighted semantic 3D Gaussian splat alignment, and replays the grafted trajectories in IsaacLab to mint demonstrations for any downstream policy (diffusion policy, ACT, π0). SAP shares its vision front end with the first two contributions, illustrating the thesis’s central claim: that perception-grounded, capable hardware is what enables modern robot learning to translate from benchmarks to real, contact-rich, bimanual, semantically rich tasks. It is compared with human demonstrations collected with an Realsense camera and teleoperation, then trained on a VLA benchmark and evaluated on a real robot to show that it can achieve comparable performance to the human demonstrations. We evaluate SAP on seven different tasks.
Source: Stanford Digital Repository
DOI: 10.25740/pb604sg3074
Institution: Stanford University
Language: English
Subfield: Control and Systems Engineering
Field: Engineering
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7225746273
FWCI:
Cites:
Cited by:
Related to: