Work

A Model-Class Comparison of Large Language Models for Translating Chinese Route Instructions into Robot Command Sequences

Year: 2026
Type: article
Abstract: Advances in large language models (LLMs) have accelerated the real-world deployment of embodied intelligent robots, supporting intuitive human–robot interaction without specialized robotics expertise. However, few systematic assessments examine how well LLMs translate natural-language route instructions into executable robot command sequences, particularly for non-English instructions and across deployment-relevant model classes. In this paper, a Chinese route-instruction corpus was constructed, which comprised 212 instructions collected from six native speakers across 50 procedurally generated town maps. Each was annotated with a ground-truth command sequence verified by simulation to reach its goal, and a 100% performance ceiling was established. Using this corpus, 12 LLMs spanning three capability tiers—cloud proprietary models, mid-sized open-weight models, and small open-weight models—were evaluated under three system prompts differing only in the number of worked examples (zero, one, and three). Open-weight models in the 3–9B range reached at most 9.2% of goals, while proprietary cloud models reached 80.0%, a substantial gap that increasing the number of worked examples did not measurably narrow. A paired analysis across the three prompt conditions detected no reliable effect of demonstration count: pooled path_reach rose modestly from 25.0% under zero-shot prompting to 27.2% with one example and 27.7% with three examples (three versus zero examples: odds ratio: 1.15; 95% CI: 0.98–1.34). None of these differences was significant after multiple-comparison correction (Holm-adjusted p ≥ 0.247). Low-Rank Adaptation (LoRA) fine-tuning on 126 examples raised mid-sized open-weight models to 27.7–35.4%, and the gain spanned both output-format compliance and route competence.
Language: English
Open Access status: gold
Estimated APC paid: $2,413
OpenAlex ID: W7236081145
FWCI:
Cites: 20
Cited by:
Related to: 10