Work

Differential learning kinetics govern the transition from memorization to generalization during in-context learning

Year: 2024

Type: preprint

Abstract: at which these sub-circuits learn explains the transition from memorization to generalization, rather than capacity constraints. We uncover a memorization scaling law, which determines the task diversity threshold at which the network generalizes. The theory quantitatively explains a variety of other ICL-related phenomena, including the long-tailed distribution of when ICL is acquired, the bimodal behavior of solutions close to the task diversity threshold, the influence of contextual and data distributional statistics on ICL, and the transient nature of ICL.

Source: PubMed

DOI: 10.48550/arxiv.2412.00104

Authors: Nguyen, Alex, Reddy, Gautam

Institution: Princeton University

Language: English

Topic: Domain Adaptation and Few-Shot Learning

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: green

Funders: Princeton University, National Institutes of Health

Award: Cloud-based Software Framework to Simplify and Standardize Real-Time fMRI

OpenAlex ID: W4405032428

Cites:

Cited by:

Related to: 10