Year: 2023
Type: dissertation
Abstract: Creating machines that can ``understand’’ our language and ``interact’’ with us as we interact with each other has been a dream that motivated many and captured the imaginations of even more. Attention-Based Transformer models have demonstrated remarkable success in getting us closer to this dream by tackling many natural language understanding tasks.However, for these impressive algorithmic developments to be of practical use, they need to be matched by capable hardware devices. Unfortunately, these models' storage, data transfer, and computation demands are already taxing even the most powerful computing hardware available today. This thesis' goal is to reduce the cost of inference for Transformer models in terms of latency and energy consumption to help enabling widespread application in our daily lives. Since in these models, the primary performance and energy efficiency bottleneck is memory transactions, this dissertation develops model compression techniques to reduce the model's memory footprint. The model compression techniques are post-training and require no fine-tuning. To reach this goal, this thesis investigates the underlying distribution of values in Attention-Based models and shows most of the weights and activations throughout the model follow a bell-shaped distribution. This motivated us to develop GOBO and Mokey, two quantization techniques for Transformer models that focus on the bulk of values to map FP32 or FP16 numbers into 3- or 4-bit indexes to dictionaries of representative values. This means forcing most of the values in a layer to be one of 8 or 16 unique values (instead of 2^32 in single-precision floating-point, FP32, format) while maintaining the models' accuracy. To exploit the quantized model even during inference computations, we designed custom hardware accelerators for GOBO and Mokey to enable computation directly on dictionary indexes instead of representative values, eliminating the need to build look-up tables for index-value conversion. GOBO and Mokey hardware accelerators improve the performance and energy efficiency of such models by one to two orders of magnitude.
Source: TSpace (University of Toronto)
Author: Hadi Zadeh, Ali
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7133061632