Year: 2025
Type: dissertation
Abstract: Recent advancements in supervised automatic speech recognition (ASR) and text-to-speech synthesis (TTS) have achieved remarkable performance, primarily driven by the increasing availability of large transcribed speech corpora.In this work, we propose an unsupervised TTS system and a wholeword unsupervised ASR system, aiming to advance fully unsupervised speech technology that can be developed with minimal supervision.An unsupervised TTS system learns to generate the speech waveform corresponding to any written sentence in a language by observing: 1) a collection of untranscribed speech waveforms in that language; 2) a collection of texts written in that language without access to any transcribed speech.Developing such a system can significantly improve the availability of speech technology to languages without a large amount of parallel speech and text data.This work proposes an unsupervised TTS system that trains on the pseudo-transcripts from an unsupervised ASR system.Our unsupervised system can achieve comparable performance to the supervised system in seven languages with about 10-20 hours of speech each.A careful study on the effect of text units and vocoders has also been conducted to better understand what factors may affect unsupervised TTS performance.We further tackle the existing challenge of developing ASR systems without paired speech and text corpora by proposing the removal of reliance on a phoneme lexicon.We explore a new research direction: word-level unsupervised ASR and experimentally demonstrate that an unsupervised speech recognizer can emerge from joint speech-to-speech and text-to-text masked token-infilling.Using a curated speech corpus containing a fixed number of English words, our system iteratively refines the word segmentation structure and achieves a word error rate of between 20-23%, depending on the vocabulary size, without parallel transcripts, oracle word boundaries, or a pronunciation lexicon.This innovative model surpasses the performance of All the work presented in this dissertation would not have been possible without the unwavering support of my graduate advisor, Prof. Mark Hasegawa-Johnson.I am deeply grateful to Prof. Hasegawa-Johnson for securing the necessary funding and for his constant guidance, encouragement, and support throughout every stage of this research.This is especially meaningful given that the work on whole-word unsupervised speech recognition spanned over two years, from conceptualization to achieving initial publishable results, followed by manuscript submission, and the extensive process of editing and resubmission.His dedication and mentorship, along with the valuable insights and guidance from Prof. Chang D. Yoo, have been pivotal in bringing this research to fruition.I would also like to express my gratitude to Dr. Liming
Author: Ni, Junrui
Institution: University of Illinois Urbana-Champaign
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7220749886
