Year: 2025
Type: dissertation
Abstract: In recent years, transformers have revolutionized machine learning, catalyzing huge performance gains across multiple disciplines. Due to the vast benefits to transformer performance of scaling up training data and model size, the computational power needed to train and deploy modern state-of-the-art models has increased drastically. Today, training and deploying frontier models costs millions of dollars and consumes terawatt-hours of energy, raising both financial and environmental concerns. This report explores a novel method of training and deploying transformers by using Matryoshka Representation Learning (MRL) in all major components of the model to improve the compute-performance tradeoff. We show how the introduction of appropriate coarse-to-fine granularities across transformer blocks efficiently reduces model parameters, optimizing model size. The proposed method has the potential to significantly reduce inference time and resource use while causing minimal overhead during training. Furthermore, we outline a dynamic nesting strategy for elastic inference, for which empirical validation is left to future research.
Authors: Riley, Axel, Oxelström, Edvard
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7047635990