Year: 2026
Type: conference-paper
Abstract: In conversation, humans use multimodal cues, such as speech, gestures, and gaze, to manage turn-taking.While linguistic and acoustic features are informative, gestures provide complementary cues for modeling these transitions.To study this, we introduce DnD Gesture++, an extension of the multi-party DnD Gesture corpus enriched with 2,663 semantic gesture annotations spanning iconic, metaphoric, deictic, and discourse types.Using this dataset, we model turn-taking prediction through a MoE framework integrating text, audio, and gestures.Experiments show that incorporating semantically guided gestures yields consistent performance gains over baselines, demonstrating their complementary role in multimodal turn-taking.
Institutions: Saarland University, Max Planck Institute for Informatics
Language: English
Topic: Speech and dialogue systems
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
Funder: Deutsche Forschungsgemeinschaft
OpenAlex ID: W7140082981