Year: 2023
Type: conference-paper
Abstract: In-context learning (ICL) in Large Language Models (LLMs) has emerged as a powerful new learning paradigm.However, its underlying mechanism is still not well understood.In particular, it is challenging to map it to the "standard" machine learning framework, where one uses a training set S to find a best-fitting function f (x) in some hypothesis class.Here we make progress on this problem by showing that the functions learned by ICL often have a very simple structure: they correspond to the transformer LLM whose only inputs are the query x and a single "task vector" calculated from the training set.Thus, ICL can be seen as compressing S into a single task vector θ(S) and then using this task vector to modulate the transformer to produce the output.We support the above claim via comprehensive experiments across a range of models and tasks. 1
Authors: Roee Hendel, Mor Geva, Amir Globerson
Institutions: Tel Aviv University, Google DeepMind (United Kingdom)
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
Funder: H2020 European Research Council
OpenAlex ID: W4389518997