Seeing is Believing: Emotion-Aware Audio-Visual Language Modeling for Expressive Speech Generation
Year: 2025
Type: preprint
Abstract: We present an Audio-Visual Language Model (AVLM) for expressive speech generation by integrating full-face visual cues into a pre-trained expressive speech model. We explore multiple visual encoders and multimodal fusion strategies during pre-training to identify the most effective integration approach. Subsequent fine-tuning on emotion recognition and expressive dialogue tasks yields substantial gains over speech-only baselines (e.g., +5 F1 in emotion recognition). AVLM highlights the value of expressive visual information in guiding speech generation and offers a foundation for end-to-end multimodal conversational systems.
Source: arXiv (Cornell University)
Authors: Tan, Weiting, Lian, Jiachen, Inaguma, Hirofumi, Tomasello, Paden, Koehn, Philipp, Ma, Xutai
Language: English
Topic: Speech and dialogue systems
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7077674658
Cites:
Cited by:
Related to: