Are Pre-trained Language Models Aware of Phrases? Simple but Strong Baselines for Grammar Induction
Year: 2020
Type: preprint
Abstract: With the recent success and popularity of pre-trained language models (LMs) in natural language processing, there has been a rise in efforts to understand their inner workings. In line with such interest, we propose a novel method that assists us in investigating the extent to which pre-trained LMs capture the syntactic notion of constituency. Our method provides an effective way of extracting constituency trees from the pre-trained LMs without training. In addition, we report intriguing findings in the induced trees, including the fact that pre-trained LMs outperform other approaches in correctly demarcating adverb phrases in sentences.
Source: arXiv (Cornell University)
Authors: Kim, Taeuk, Choi, Jihun, Edmiston, Daniel, Lee, Sang-goo
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
Award: G5619053062
OpenAlex ID: W2995856824
