Work

Probability Distributions Computed by Autoregressive Transformers

Year: 2025

Type: preprint

Abstract: Most expressivity results for transformers treat them as language recognizers -- devices that accept or reject strings -- rather than as they are used in practice: as language models that generate strings autoregressively and probabilistically. We characterize the probability distributions that transformer language models can express. We show that making transformer language recognizers autoregressive can sometimes increase their expressivity, and that making them probabilistic can break equivalences that hold in the non-probabilistic case. Our overall contribution is to tease apart what functions transformers are capable of expressing in their most common use case as language models.

Source: arXiv (Cornell University)

DOI: 10.48550/arxiv.2510.27118

Authors: Yang, Andy, Svete, Anej, Li, Jiaoda, Lin, Anthony Widjaja, Rawski, Jonathan, Cotterell, Ryan, Chiang, David

Institutions: University of Notre Dame, ETH Zurich, Max Planck Institute for Software Systems, Rheinland-Pfälzische Technische Universität Kaiserslautern-Landau, San Jose State University

Topic: Topic Modeling

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: green

Funders: National Science Foundation, Leibniz-Gemeinschaft, HORIZON EUROPE European Research Council

Awards: Graduate Research Fellowship Program (GRFP), MFAI: Logic and the Theory of Neural Language Models, Logic and Automata over Sequences with Data

OpenAlex ID: W4415914949

Cites:

Cited by:

Related to: