Work

A Spoken Dialogue Framework for Bidirectional Interaction with Robot Knowledge Graphs via Grammar-Constrained LLM Generation

Year: 2026
Type: conference-paper
Abstract: Enabling robots to maintain and communicate structured world knowledge through natural spoken language is essential for transparent and accessible human-robot interaction. Moreover, there are several solutions that employ knowledge graph representations as part of their behavior generation pipeline, but the knowledge creation can be a limitation for autonomous robots. This paper presents a bidirectional speech-based knowledge-grounding tool for creating and querying new knowledge for autonomous robots. The presented framework integrates speech-to-text and text-to-speech components with a locally deployed Large Language Model whose output is constrained by a context-free grammar, ensuring that every generated token sequence is a syntactically valid structured object that encodes knowledge graph operations. This grammar-constrained decoding eliminates the need for posthoc output validation and reduces malformed responses. The system is evaluated on a purpose-built benchmark of diverse utterances spanning eight interaction categories, comparing five locally-deployed language models. Results demonstrate that the proposed solution achieves full structural validity while maintaining competitive semantic accuracy across different models.
Source: IEEE RO-MAN
Language: English
Open Access status: hybrid
OpenAlex ID: W7220955356
FWCI:
Cites: 20
Cited by:
Related to: 10