Safety-Gated LLM Manipulation on a Collaborative Robot: A Reproducible Simulation-to-Hardware Framework
Year: 2026
Type: conference-paper
Abstract: Large language models (LLMs) can make robot operation accessible without specialist programming, but their non-deterministic outputs should not have direct motion authority in safety-relevant cells. This paper presents a MATLAB-in-the-loop framework for a UR5e with OnRobot RG2 gripper that separates language interpretation from deterministic motion authority. A local LLM receives an ArUco-grounded scene and may propose only a predefined, schema-constrained skill. An input guard screens the utterance, a symbolic gate checks the task, and a geometric gate checks every retained trajectory sample before any controller sends; a rule-based fallback passes through the same gates. Four fixed-decoding simulation repetitions reproduced one another exactly, assessing determinism rather than independent replication. Hardware evaluation comprised one 80-row repeated-command configuration with 15 distinct texts and three 80-row sessions reusing the same 80 paraphrases across layouts. Because wording, scene, and software revision changed together, the grounding comparison is descriptive: 72/80 in A versus 163/240 in B-D. All 311 sends passed the implemented sampled geometric checks, and all 246 recorded controller-status samples were normal. Nevertheless, 14/120 must-not-move commands caused motion, including 8/45 prompt-injection trials. These geometrically certified but semantically incorrect motions show that geometric admissibility does not establish alignment with operator intent.
Institution: University of Sheffield
Language: English
Subfield: Control and Systems Engineering
Field: Engineering
Domain: Physical Sciences
Open Access status: gold
OpenAlex ID: W7225409754