Year: 2024
Type: conference-paper
Abstract: Understanding in-context learning (ICL) capability that enables large language models (LLMs) to excel in proficiency through demonstration examples is of utmost importance.This importance stems not only from the better utilization of this capability across various tasks, but also from the proactive identification and mitigation of potential risks, including concerns regarding truthfulness, bias, and toxicity, that may arise alongside the capability.In this paper, we present a thorough survey on the interpretation and analysis of in-context learning.First, we provide a concise introduction to the background and definition of in-context learning.Then, we give an overview of advancements from two perspectives: 1) the theoretical perspective, emphasizing studies on mechanistic interpretability and delving into the mathematical foundations behind ICL; and 2) the empirical perspective, concerning studies that empirically analyze factors associated with ICL.We conclude by discussing open questions and the challenges encountered and, by suggesting potential avenues for future research.We believe that our work establishes the basis for further exploration into the interpretation of incontext learning.To aid this effort, we have created a repository 1 containing resources that will be continually updated.1 https://github.com/zyxnlp
Institutions: King's College London, The Alan Turing Institute
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
Awards: Turing AI Fellowship: Event-Centric Framework for Natural Language Understanding, Elandi: Trustworthy generative AI for affordable personalised L&D - Phase 2, Turing AI Fellowship: Event-Centric Framework for Natural Language Understanding, A Lebesgue Integral based Approximation for Language Modelling
OpenAlex ID: W4404782475