Get started
Work

Enhancing Linguistic Competence of Language Models through Pre-training with Language Learning Tasks

Year: 2026
Type: conference-paper
Abstract: Language models (LMs) are pre-trained on raw text datasets to generate text sequences token-by-token.While this approach facilitates the learning of world knowledge and reasoning, it does not explicitly optimize for linguistic competence.To bridge this gap, we propose L2T, a pre-training framework integrating Language Learning Tasks alongside standard next-token prediction.Inspired by human language acquisition, L2T transforms raw text into structured input-output pairs to provide explicit linguistic stimulation.Pre-training LMs on a mixture of raw text and L2T data not only improves overall performance on linguistic competence benchmarks but accelerates its acquisition, while maintaining competitive performance on general reasoning tasks. 1
DOI: 10.18653/v1/2026.acl-short.27
Language: English
Open Access status: hybrid
Funders: Government of the United Kingdom, UK Research and Innovation, University of Oxford, University of Bristol, Japan Student Services Organization, Engineering and Physical Sciences Research Council, Science and Technology Facilities Council
Awards: G2485087852, G2831959038, AI UK: Creating an International Ecosystem for Responsible AI Research and Innovation, G3881378079, G5306878388, DTP 2224 University of Sheffield, UKRI Centre for Doctoral Training in Speech and Language Technologies and their Applications, G963556061
OpenAlex ID: W7166890814
FWCI:
Cites:
Cited by:
Related to: 10