Language processing in brains and deep neural networks: computational convergence and its limits
Year: 2020
Type: preprint
Abstract: Deep Learning has recently led to major advances in natural language processing. Do these models process sentences similarly to humans, and is this similarity driven by specific principles? Using a variety of artificial neural networks, trained on image classification, word embedding, or language modeling, we evaluate whether their architectural and functional properties lead them to generate activations linearly comparable to those of 102 human brains measured with functional magnetic resonance imaging (fMRI) and magnetoencephalography (MEG). We show that image, word and contextualized word embeddings separate the hierarchical levels of language processing in the brain. Critically, we compare 3,600 embeddings in their ability to linearly map onto these brain responses. The results show that (1) the position of the layer in the network and (2) the ability of the network to accurately predict words from context are the main factors responsible for the emergence of brain-like representations in artificial neural networks. Together, these results show how perceptual, lexical and compositional representations precisely unfold within each cortical region and contribute to uncovering the governing principles of language processing in brains and algorithms.
Authors: Charlotte Caucheteux, Jean-Rémi King
Language: English
Subfield: Cognitive Neuroscience
Field: Neuroscience
Domain: Life Sciences
Keywords: language processing, deep neural network, brain-computer analogy, functional MRI, magnetoencephalography, word embeddings, contextual word embeddings, hierarchical representation, compositional representations, perceptual representation, lexical representation, network depth, language modeling
Open Access status: green
OpenAlex ID: W3039556919
