Year: 2018
Type: conference-paper
Abstract: We present a dataset for evaluating the grammaticality of the predictions of a language model.We automatically construct a large number of minimally different pairs of English sentences, each consisting of a grammatical and an ungrammatical sentence.The sentence pairs represent different variations of structure-sensitive phenomena: subject-verb agreement, reflexive anaphora and negative polarity items.We expect a language model to assign a higher probability to the grammatical sentence than the ungrammatical one.In an experiment using this data set, an LSTM language model performed poorly on many of the constructions.Multi-task training with a syntactic objective (CCG supertagging) improved the LSTM's accuracy, but a large gap remained between its performance and the accuracy of human participants recruited online.This suggests that there is considerable room for improvement over LSTMs in capturing syntax in a language model.
DOI: 10.18653/v1/d18-1151
Authors: Rebecca Marvin, Tal Linzen
Institution: Johns Hopkins University
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W2888922637