Year: 2025
Type: article
Abstract: Many speech translation systems strip away a speaker's vocal identity, leading to monotonous outputs.We present a speaker-aware multilingual method that conditions translation and synthesis on compact speaker embeddings.These embeddings encode pitch, timbre, rhythm, and other paralinguistic features from the source utterance.By injecting embeddings into both translation and TTS modules, our pipeline preserves vocal nuances across languages.The model is trained on multilingual, multi-speaker datasets with objectives that disentangle content from speaker traits.Experimental results show competitive BLEU scores alongside significant gains in cosine similarity of speaker embeddings.Human listening tests further validate that translated audio is perceived as coming from the original speaker.With only a few seconds of enrollment audio, our system supports zero-shot voice cloning in new languages.Real-time inference makes it suitable for live interpretation, personalized dubbing, and other interactive multilingual applications.
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W4410528095