Work

Knowledge Distillation Methods for Sequence-to-Sequence Learning in Speech and Language Processing

Year: 2024
Type: dissertation
Abstract: Recently, sequence-to-sequence learning has shown remarkable performance in speech and natural language processing. However, high-performing sequence models commonly require expensive computational costs, such as lengthy training times, large model sizes, and multiple graphics processing units. To reduce this burden, knowledge distillation (KD), which is a popular model compression method, has been used to transfer knowledge from a deep and complex model (teacher) to a shallower and simpler model (student). This thesis explores the existing shortcomings of conventional KD, and we propose novel frameworks to overcome them. \nFirstly, we introduce a new KD framework to overcome the structural limitations of KD. Previous KD approaches have commonly designed the architecture of the student model by reducing the width per layer or the number of layers of the teacher model. This structural reduction scheme might limit the flexibility of model selection since the student model structure should be similar to that of the given teacher. To cope with this limitation, we propose a novel KD method for end-to-end speech recognition, namely TutorNet, that can transfer knowledge across different types of neural networks at the hidden representation-level as well as the output-level. For concrete realizations, we firstly apply representation-level knowledge distillation (RKD) during the initialization step and then apply the softmax-level knowledge distillation (SKD) combined with the original task learning. When the student is trained with RKD, we make use of frame weighting that points out the frames to which the teacher model pays more attention. Through a number of experiments on LibriSpeech dataset, it is verified that the proposed method not only distills the knowledge between networks with different topologies but also significantly contributes to improving the word error rate (WER) performance of the distilled student. Interestingly, TutorNet allows the student model to surpass its teacher's performance in some particular cases. \nSecondly, we focus on the shortage of the original teacher model in the KD framework. Conventional KD methods usually employ a teacher model with high capacity that has been trained in a supervised manner, where the ground-truth labels are required as a target. However, training the teacher from scratch can be costly since many of the current state-of-the-art (SOTA) models suffer from excessive training time and difficult hyper-parameters tuning. Also, there is no guarantee that the teacher can find the correct solution for every complicated problem in an optimal way. In this thesis, we introduce a new type of teacher model for connectionist temporal classification (CTC)-based sequence models, namely Oracle Teacher, that leverages both the source inputs and the output labels as the teacher model's input. Extensive experiments are conducted on two sequence learning tasks: speech recognition and scene text recognition. From the experimental results, we empirically show that the proposed model improves the students across these tasks while achieving a considerable speed-up in the teacher model's training time. \nLastly, we propose a novel self-distillation approach that effectively leverages target information for supervised sequence-to-sequence learning. In the existing self-distillation setting, where the student network acts as its own teacher, there is no guarantee that the distilled knowledge is sufficient to provide an accurate solution for the task. Also, some self-distillation methods rely on an additional update strategy, such as the exponential moving average (EMA) of parameters, for stable convergence. In this thesis, we propose a new framework termed EM-Network that can effectively utilize the target information for sequence-to-sequence learning, improving the quality of the latent alignment. Our model can be stably optimized without any additional complex update strategy. In addition, we theoretically demonstrate that the proposed objective function is justified from the Expectation-Maximization (EM)-like algorithm perspective. We conduct comprehensive experiments on two types of seq2seq models: CTC for speech recognition and attention-based encoder-decoder (AED) for machine translation. Experimental results demonstrate that the EM-Network significantly advances the current state-of-the-art approaches, improving over the best prior work on speech recognition and establishing SOTA performance on WMT'14 and IWSLT'14.
Author: 윤지원
Language: English
Open Access status: green
OpenAlex ID: W7001488138
Cites: 98
Cited by:
Related to: 10