Year: 2024
Type: conference-paper
Abstract: This work introduces Sample-Efficient Speech Diffusion (SESD), an algorithm for effective speech synthesis in modest data regimes through latent diffusion.It is based on a novel diffusion architecture, that we call U-Audio Transformer (U-AT), that efficiently scales to long sequences and operates in the latent space of a pre-trained audio autoencoder.Conditioned on character-aware language model representations, SESD achieves impressive results despite training on less than 1k hours of speech -far less than current state-of-the-art systems.In fact, it synthesizes more intelligible speech than the state-of-the-art auto-regressive model, VALL-E, while using less than 2% the training data.Our implementation is available at https://github.com/justinlovelace/SESD.
Source: Interspeech (USB)
Institution: Cornell University
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W4402115993