Year: 2022
Type: conference-paper
Abstract: Compute, data, and algorithmic advances are the three fundamental factors that drive progress in modern Machine Learning (ML). In this paper we study trends in the most readily quantified factor - compute. We make three novel contributions: (1) we curate a dataset with the training compute of 123 milestone ML systems, 3× larger than previous such datasets. (2) We frame the trends in compute in in three eras - the Pre Deep Learning Era, the Deep Learning Era, and the Large-Scale Era, based on our identification of a novel trend emerging around 2015. (3) We find a Deep Learning Era compute doubling time of around 6 months, significantly longer than previous findings. Overall, our work highlights the fast-growing compute requirements for training advanced ML systems.
Authors: Jaime Sevilla, Lennart Heim, Anson Ho, Tamay Besiroglu, Marius Hobbhahn, Pablo Villalobos
Language: English
Subfield: Hardware and Architecture
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
Funder: Open Philanthropy Project
OpenAlex ID: W4312968147