Polynomial Regression as a Task for Understanding In-context Learning Through Finetuning and Alignment
Year: 2024
Type: preprint
Abstract: Simple function classes have emerged as toy problems to better understand in-context-learning in transformer-based architectures used for large language models. But previously proposed simple function classes like linear regression or multi-layer-perceptrons lack the structure required to explore things like prompting and alignment within models capable of in-context-learning. We propose univariate polynomial regression as a function class that is just rich enough to study prompting and alignment, while allowing us to visualize and understand what is going on clearly.
Source: arXiv (Cornell University)
Authors: Wilcoxson, Max, Svendgård, Morten, Doshi, Ria, Davis, Dylan, Vir, Reya, Sahai, Anant
Institution: University of California, Berkeley
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W4401201608