Work

Transformer Network for Semantically-Aware and Speech-Driven Upper-Face Generation

Year: 2022

Type: conference-paper

Abstract: We propose a semantically-aware speech driven model to generate expressive and natural upper-facial and head motion for Embodied Conversational Agents (ECA). In this work, we aim to produce natural and continuous head motion and upper-facial gestures synchronized with speech. We propose a model that generates these gestures based on multimodal input features: the first modality is text, and the second one is speech prosody. Our model makes use of Transformers and Convolutions to map the multimodal features that correspond to an utterance to continuous eyebrows and head gestures. We conduct subjective and objective evaluations to validate our approach and compare it with state of the art.

Source: European Signal Processing Conference (EUSIPCO)

DOI: 10.23919/eusipco55093.2022.9909519

Authors: Mireille Fares, Catherine Pelachaud, Nicolas Obin

Institutions: Sorbonne Université, Institut Systèmes Intelligents et de Robotique, Sciences et Technologies de la Musique et du Son, Centre National de la Recherche Scientifique, Institut de Recherche et Coordination Acoustique Musique

Language: English

Topic: Human Motion and Animation

Subfield: Computer Graphics and Computer-Aided Design

Field: Computer Science

Domain: Physical Sciences

Open Access status: green

Funder: Agence Nationale de la Recherche

Awards: G2647246237, Sorbonne Universités à Paris pour l'Enseignement et la Recherche