Work

Let’s Play Mono - Poly : BERT Can Reveal Words’ Polysemy Level and Partitionability into Senses

Year: 2021

Type: article

Abstract: Pre-trained language models (LMs) encode rich information about linguistic structure but their knowledge about lexical polysemy remains unclear. We propose a novel experimental setup for analyzing this knowledge in LMs specifically trained for different languages (English, French, Spanish, and Greek) and in multilingual BERT. We perform our analysis on datasets carefully designed to reflect different sense distributions, and control for parameters that are highly correlated with polysemy such as frequency and grammatical category. We demonstrate that BERT-derived representations reflect words’ polysemy level and their partitionability into senses. Polysemy-related information is more clearly present in English BERT embeddings, but models in other languages also manage to establish relevant distinctions between words at different polysemy levels. Our results contribute to a better understanding of the knowledge encoded in contextualized representations and open up new avenues for multilingual lexical semantics research.

Source: Transactions of the Association for Computational Linguistics

DOI: 10.1162/tacl_a_00400

Authors: Aina Garí Soler, Marianna Apidianaki

Institutions: Centre National de la Recherche Scientifique, Université Paris-Saclay, Laboratoire d'Informatique pour la Mécanique et les Sciences de l'Ingénieur, Laboratoire Interdisciplinaire des Sciences du Numérique, University of Helsinki

Language: English

Topic: Topic Modeling

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: diamond

Funders: European Commission, Agence Nationale de la Recherche

Awards: G434337789, G4459410915, G5420063140, Advanced Models for Multilingual Semantic Processing