Work

A Survey on Model Compression and Acceleration for Pretrained Language Models

Year: 2023

Type: conference-paper

Abstract: Despite achieving state-of-the-art performance on many NLP tasks, the high energy cost and long inference delay prevent Transformer-based pretrained language models (PLMs) from seeing broader adoption including for edge and mobile computing. Efficient NLP research aims to comprehensively consider computation, time and carbon emission for the entire life-cycle of NLP, including data preparation, model training and inference. In this survey, we focus on the inference stage and review the current state of model compression and acceleration for pretrained language models, including benchmarks, metrics and methodology.

Source: Proceedings of the AAAI Conference on Artificial Intelligence

DOI: 10.1609/aaai.v37i9.26255

Authors: Canwen Xu, Julian McAuley

Institution: University of California San Diego

Language: English

Topic: Advanced Neural Network Applications

Subfield: Computer Vision and Pattern Recognition

Field: Computer Science

Domain: Physical Sciences

Open Access status: diamond

Funder: National Science Foundation

Award: CAREER: Structured Output Models of Recommendations, Activities, and Behavior