Year: 2020
Type: conference-paper
Abstract: Typically, machine learning systems solve new tasks by training on thousands of examples.In contrast, humans can solve new tasks by reading some instructions, with perhaps an example or two.To take a step toward closing this gap, we introduce a framework for developing NLP systems that solve new tasks after reading their descriptions, synthesizing prior work in this area.We instantiate this framework with a new English language dataset, ZEST, structured for task-oriented evaluation on unseen tasks.Formulating task descriptions as questions, we ensure each is general enough to apply to many possible inputs, thus comprehensively evaluating a model's ability to solve each task.Moreover, the dataset's structure tests specific types of systematic generalization.We find that the state-of-the-art T5 model achieves a score of 12% on ZEST, leaving a significant challenge for NLP researchers. 1
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
Funder: National Science Foundation
OpenAlex ID: W3099338292