Learning Where to Grasp and Place: Role-Specific Attention for Language-Conditioned Manipulation
Year: 2026
Type: article
Abstract: Language-conditioned tabletop manipulation requires identifying the object to be grasped and the spatial region for its placement relative to a reference object. This study presents an end-to-end model that predicts grasp position, grasp angle, and placement position jointly from an RGB image and a relational instruction. Role-specific attention maps weight a shared visual feature grid separately for grasping and placement, preserving spatial structure for action prediction. The maps are conditioned on the instruction through low-rank bilinear fusion, and the resulting model predicts operation parameters directly at inference. Existing scene-instruction pairs from the previous study are re-annotated with two position maps and grasp-angle classes to supervise the three output branches jointly. The model reaches full-pipeline success rates of 94.70% across 2000 simulated trials and 85.50% across 400 real-robot trials. In simulation and on the real robot, respectively, these rates exceed CLIPort-adapted by 4.15 and 4.25 percentage points and the modular baseline by 7.35 and 6.75 percentage points. Separate attention improves simulated full-pipeline success by 10.55 percentage points over shared attention; domain randomization improves real-robot success by 11.00 percentage points. Paraphrase and held-out-combination tests reveal remaining sensitivity to linguistic and compositional variation. These results indicate that role-specific attention supports joint learning of instruction-conditioned grasp and placement parameters in planar manipulation.
Source: Sensors
DOI: 10.3390/s26227292
Institution: China Aerodynamics Research and Development Center
Language: English
Subfield: Control and Systems Engineering
Field: Engineering
Domain: Physical Sciences
Open Access status: gold
Estimated APC paid: $3,137
OpenAlex ID: W7236342738
