Unsupervised learning of cross-modal mappings between speech and text
Year: 2019
Type: dissertation
Abstract: Deep learning is one of the most prominent machine learning techniques nowadays, being the state-of-the-art on a broad range of applications in computer vision, natural language processing, and speech and audio processing.Current deep learning models, however, rely on significant amounts of supervision for training to achieve exceptional performance.For example, commercial speech recognition systems are usually trained on tens of thousands of hours of annotated data, which take the form of audio paired with transcriptions for training acoustic models, collections of text for training language models, and (possibly) linguist-crafted lexicons mapping words to their pronunciations.The immense cost of collecting these resources makes applying state-of-the-art speech recognition algorithm to under-resourced languages infeasible.In this thesis, we propose a general framework for mapping sequences between speech and text.Each component in this framework can be trained without any labeled data so the entire framework is unsupervised.We first propose a novel neural architecture that learns to represent a spoken word in an unlabeled speech corpus as an embedding vector in a latent space, in which word semantics and relationships between words are captured.In parallel, we train another latent space that captures similar information about written words using a corpus of unannotated text.By exploiting the geometrical properties exhibited in the speech and text embedding spaces, we develop an unsupervised learning algorithm that learns a cross-modal alignment between speech and text.As an example application of the learned alignment, we develop a unsupervised speech-to-text translation system using only unlabeled speech and text corpora.
Author: Yu-An Chung
Institution: Massachusetts Institute of Technology
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W2990626918
