Work

ImmersiveTTS: Environment-Aware Text-to-Speech with Multimodal Diffusion Transformer and Domain-Specific Representation Alignment

Year: 2026

Type: conference-paper

Abstract: Recent advancements in text-guided audio generation have yielded promising results in diverse domains, including sound effects, speech, and music.However, jointly generating speech with environmental audio remains challenging due to the inherent disparities in their acoustic patterns and temporal dynamics.We propose ImmersiveTTS, an environment-aware text-to-speech (TTS) model that generates natural speech seamlessly integrated within environmental contexts by explicitly modeling cross-modal interactions.Our model builds on a multimodal diffusion transformer and fuses transcript-aligned speech latent with textconditioned environmental context via joint attention.To enhance semantic consistency, we introduce a domain-specific representation alignment objective tailored to environmentaware TTS, leveraging complementary selfsupervised representations from speech and audio encoders.Experimental results show that ImmersiveTTS achieves higher naturalness, intelligibility, and audio fidelity than existing approaches across objective metrics and human listening tests.

Source: Proceedings of the conference - Association for Computational Linguistics. Meeting

DOI: 10.18653/v1/2026.acl-long.1774

Authors: Jun-Hak Yun, Seung-Bin Kim, Seong-Whan Lee

Institution: Korea University

Language: English

Topic: Speech Recognition and Synthesis

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: hybrid

Funders: Korea University, Ministry of Science and ICT, South Korea, Institute for Information and Communications Technology Promotion

Awards: G6969530878, G8458676037, G960095468