Year: 2025
Type: conference-paper
Abstract: Sample efficiency is a crucial property of language models with practical implications for training efficiency.In real-world text, information follows a long-tailed distribution.Yet, we expect models to learn and recall frequent and infrequent facts.Sample-efficient models are better equipped to handle this challenge of learning and retaining rare information without requiring excessive exposure.This study analyzes multiple models of varying architectures and sizes, all trained on the same pre-training data.By annotating relational facts with their frequencies in the training corpus, we examine how model performance varies with fact frequency.Our findings show that most models perform similarly on high-frequency facts but differ notably on low-frequency facts.This analysis provides new insights into the relationship between model architecture, size, and factual learning efficiency.
Institution: Humboldt-Universität zu Berlin
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
Funder: Deutsche Forschungsgemeinschaft
OpenAlex ID: W4412944322