Work

How to Train BERT with an Academic Budget

Year: 2021
Type: conference-paper
Abstract: While large language models à la BERT are used ubiquitously in NLP, pretraining them is considered a luxury that only a few wellfunded industry labs can afford.How can one train such models with a more modest budget?We present a recipe for pretraining a masked language model in 24 hours using a single lowend deep learning server.We demonstrate that through a combination of software optimizations, design choices, and hyperparameter tuning, it is possible to produce models that are competitive with BERT BASE on GLUE tasks at a fraction of the original pretraining cost. 1
Language: English
Open Access status: hybrid
OpenAlex ID: W3155183406
FWCI: 7.357
Cites: 36
Cited by: 167
Related to: 10