Work

An Experimental Design Framework for Label-Efficient Supervised Finetuning of Large Language Models

Year: 2024

Type: preprint

Abstract: Supervised finetuning (SFT) on instruction datasets has played a crucial role in achieving the remarkable zero-shot generalization capabilities observed in modern large language models (LLMs). However, the annotation efforts required to produce high quality responses for instructions are becoming prohibitively expensive, especially as the number of tasks spanned by instruction datasets continues to increase. Active learning is effective in identifying useful subsets of samples to annotate from an unlabeled pool, but its high computational cost remains a barrier to its widespread applicability in the context of LLMs. To mitigate the annotation cost of SFT and circumvent the computational bottlenecks of active learning, we propose using experimental design. Experimental design techniques select the most informative samples to label, and typically maximize some notion of uncertainty and/or diversity. In our work, we implement a framework that evaluates several existing and novel experimental design techniques and find that these methods consistently yield significant gains in label efficiency with little computational overhead. On generative tasks, our methods achieve the same generalization performance with only $50\%$ of annotation cost required by random sampling.

Source: arXiv (Cornell University)

DOI: 10.48550/arxiv.2401.06692

Authors: Bhatt, Gantavya, Chen, Yifang, Das, Arnav M., Zhang, Jifan, Truong, Sang T., Mussmann, Stephen, Zhu, Yinglun, Bilmes, Jeffrey, Du, Simon S., Jamieson, Kevin, Ash, Jordan T., Nowak, Robert D.

Institutions: University of Washington, University of Wisconsin–Madison, Stanford University, Georgia Institute of Technology, University of California, Riverside, Microsoft Research New York City (United States)

Language: English

Topic: Machine Learning and Algorithms

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: green

Funders: National Science Foundation, National Institutes of Health, National Human Genome Research Institute, Air Force Office of Scientific Research, Air Force Research Laboratory

Awards: G1222291137, G1523888516, G1766360814, G2020385801, AI Institute for Future Edge Networks and Distributed Intelligence (AI-EDGE), RINGS: Bumblebee: A Neural Network Transformer Architecture for Summarization and Prediction in Interactive XR Applications, TRIPODS: Institute for Foundations of Data Science, Collaborative Research: RI: Medium: Submodular Information Functions with Applications to Machine Learning, G8289759875

OpenAlex ID: W4390897950

Cites:

Cited by:

Related to: 10