Year: 2025
Type: conference-paper
Abstract: Children acquire language despite being exposed to several orders of magnitude less data than large language models require.Metalearning has been proposed as a way to integrate human-like learning biases into neuralnetwork architectures, combining both the structured generalizations of symbolic models with the scalability of neural-network models.But what does meta-learning exactly imbue the model with?We investigate the meta-learning of formal languages and find that, contrary to previous claims, meta-trained models are not learning simplicity-based priors when metatrained on datasets organised around simplicity.Rather, we find evidence that meta-training imprints neural mechanisms (such as counters) into the model, which function like cognitive primitives for the network on downstream tasks.Most surprisingly, we find that meta-training on a single formal language can provide as much improvement to a model as meta-training on 5000 different formal languages, provided that the formal language incentivizes the learning of useful neural mechanisms.Taken together, our findings provide practical implications for efficient meta-learning paradigms and new theoretical insights into linking symbolic theories and neural mechanisms.
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W4412945100