Lexical-Semantic Content, Not Syntactic Structure, Is the Main Contributor to ANN-Brain Similarity of fMRI Responses in the Language Network
Year: 2023
Type: article
Abstract: = 627 naturalistic English sentences (Pereira et al., 2018) and systematically manipulated the stimuli for which ANN representations were extracted. In particular, we (i) perturbed sentences' word order, (ii) removed different subsets of words, or (iii) replaced sentences with other sentences of varying semantic similarity. We found that the lexical-semantic content of the sentence (largely carried by content words) rather than the sentence's syntactic form (conveyed via word order or function words) is primarily responsible for the ANN-to-brain similarity. In follow-up analyses, we found that perturbation manipulations that adversely affect brain predictivity also lead to more divergent representations in the ANN's embedding space and decrease the ANN's ability to predict upcoming tokens in those stimuli. Further, results are robust as to whether the mapping model is trained on intact or perturbed stimuli and whether the ANN sentence representations are conditioned on the same linguistic context that humans saw. The critical result-that lexical-semantic content is the main contributor to the similarity between ANN representations and neural ones-aligns with the idea that the goal of the human language system is to extract meaning from linguistic strings. Finally, this work highlights the strength of systematic experimental manipulations for evaluating how close we are to accurate and generalizable models of the human language network.
Source: Neurobiology of Language
DOI: 10.1162/nol_a_00116
Language: English
Subfield: Cognitive Neuroscience
Field: Neuroscience
Domain: Life Sciences
Open Access status: gold
Estimated APC paid: $1,100
Funders: National Science Foundation, American Association of University Women, Massachusetts Institute of Technology, National Institutes of Health, Simons Center for the Social Brain, Massachusetts Institute of Technology, McGovern Institute for Brain Research, Massachusetts Institute of Technology, Quest for Intelligence, Massachusetts Institute of Technology, National Institute on Deafness and Other Communication Disorders, National Institute of Neurological Disorders and Stroke, MIT-IBM Watson AI Lab
Awards: G155546341, CompCog: Noisy-channel processing in human language understanding, G4473135385, G6843184352, Functional reorganization of the language and domain-general multiple demand systems in aphasia, The neural architecture of pragmatic processing, G8959081239, G9000244425, Computational Neuroscience of Language Processing in the Human Brain
OpenAlex ID: W4384697713