Work

Gelina: Unified Speech and Gesture Synthesis Via Interleaved Token Prediction

Year: 2026

Type: conference-paper

Abstract: Human communication is multimodal, with speech and gestures tightly coupled, yet most computational methods for generating speech and gestures synthesize them sequentially, weakening synchrony and prosody alignment. We introduce Gelina, a unified framework that jointly synthesizes speech and co-speech gestures from text using interleaved token sequences in a discrete autoregressive backbone, with modality-specific decoders. Gelina supports multi-speaker and multi-style cloning and enables gesture-only synthesis from speech inputs. Subjective and objective evaluations demonstrate competitive speech quality and improved gesture generation over unimodal baselines.

Source: IEEE International Conference on Acoustics Speech and Signal Processing

DOI: 10.1109/icassp55912.2026.11464562

Authors: Téo Guichoux, Théodor Lemerle, Shivam Mehta, Jonas Beskow, Gustav Eje Henter, Laure Soulier, Catherine Pelachaud, Nicolas Obin

Institutions: Sorbonne Université, Institut Systèmes Intelligents et de Robotique, Institut de Recherche et Coordination Acoustique Musique, Sciences et Technologies de la Musique et du Son, KTH Royal Institute of Technology

Language: English

Topic: Speech Recognition and Synthesis

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: green

Funder: Technology Development

OpenAlex ID: W4415273624

FWCI:

Cites:

Cited by:

Related to: