Work

CoBERT: Self-Supervised Speech Representation Learning Through Code Representation Learning

Year: 2022

Type: preprint

Abstract: Speech is the surface form of a finite set of phonetic units, which can be represented by discrete codes. We propose the Code BERT (CoBERT) approach for self-supervised speech representation learning. The idea is to convert an utterance to a sequence of discrete codes, and perform code representation learning, where we predict the code representations based on a masked view of the original speech input. Unlike the prior self-distillation approaches of which the teacher and the student are of the same modality, our target model predicts representations from a different modality. CoBERT outperforms the most recent state-of-the-art performance on the ASR task and brings significant improvements on the SUPERB speech translation (ST) task. Our code and models are released at https://github.com/mct10/CoBERT.

Source: arXiv (Cornell University)

DOI: 10.48550/arxiv.2210.04062

Authors: Meng, Chutong, Ao, Junyi, Ko, Tom, Wang, Mingxuan, Li, Haizhou

Institutions: Chinese University of Hong Kong, Shenzhen Research Institute of Big Data, Chinese University of Hong Kong, Shenzhen

Language: English

Topic: Speech Recognition and Synthesis

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: green

Funders: National Natural Science Foundation of China, Chinese University of Hong Kong, Shenzhen Research Institute of Big Data, Shenzhen Research Institute, City University of Hong Kong

Awards: G2799040894, G5088964179, G861645084

OpenAlex ID: W4304701018

Cites:

Cited by:

Related to: 10