Work

A Primer on the Inner Workings of Transformer-based Language Models

Year: 2024

Type: preprint

Abstract: The rapid progress of research aimed at interpreting the inner workings of advanced language models has highlighted a need for contextualizing the insights gained from years of work in this area. This primer provides a concise technical introduction to the current techniques used to interpret the inner workings of Transformer-based language models, focusing on the generative decoder-only architecture. We conclude by presenting a comprehensive overview of the known internal mechanisms implemented by these models, uncovering connections across popular approaches and active research directions in this area.

Source: arXiv (Cornell University)

DOI: 10.48550/arxiv.2405.00208

Authors: Ferrando, Javier, Sarti, Gabriele, Bisazza, Arianna, Costa-jussà, Marta R.

Language: English

Topic: Explainable Artificial Intelligence (XAI)

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: green

Funders: Nederlandse Organisatie voor Wetenschappelijk Onderzoek, Ministerio de Ciencia e Innovación, Agencia Estatal de Investigación

Awards: InDeep: Interpreting Deep Learning Models for Text and Sound, G1855990336, G2262748287, G2954395134, G3480869486, G3690828678, G425668148, G5421974903, G8684520763

OpenAlex ID: W4396648991

Cites:

Cited by: 10

Related to: 10