Work

EQUATE: A Benchmark Evaluation Framework for Quantitative Reasoning in Natural Language Inference

Year: 2019

Type: conference-paper

Abstract: Quantitative reasoning is a higher-order reasoning skill that any intelligent natural language understanding system can reasonably be expected to handle.We present EQUATE 1 (Evaluating Quantitative Understanding Aptitude in Textual Entailment), a new framework for quantitative reasoning in textual entailment.We benchmark the performance of 9 published NLI models on EQUATE, and find that on average, state-of-the-art methods do not achieve an absolute improvement over a majority-class baseline, suggesting that they do not implicitly learn to reason with quantities.We establish a new baseline Q-REAS that manipulates quantities symbolically.In comparison to the best performing NLI model, it achieves success on numerical reasoning tests (+24.2%),but has limited verbal reasoning capabilities (-8.1%).We hope our evaluation framework will support the development of models of quantitative reasoning in language understanding.

Source: Conference on Computational Natural Language Learning

DOI: 10.18653/v1/k19-1033

Authors: Abhilasha Ravichander, Aakanksha Naik, Carolyn Rose, Eduard Hovy

Institution: Carnegie Mellon University

Language: English

Topic: Natural Language Processing Techniques

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: hybrid

Funders: National Science Foundation, Brandeis University, Carnegie Mellon University, Defense Advanced Research Projects Agency

Awards: SaTC: CORE: Medium: Collaborative: Contextual Integrity: From Theory to Practice, G4713059963, G6399177790, SaTC: CORE: Medium: Collaborative: Automatically Answering People's Privacy Questions, TWC SBE: Option: Frontier: Collaborative: Towards Effective Web Privacy Notice and Choice: A Multi-Disciplinary Prospective