Year: 2023
Type: conference-paper
Abstract: Large language models underestimate the impact of negations on how much they change the meaning of a sentence.Therefore, learned evaluation metrics based on these models are insensitive to negations.In this paper, we propose NegBLEURT, a negation-aware version of the BLEURT evaluation metric.For that, we designed a rule-based sentence negation tool and used it to create the CANNOT negation evaluation dataset.Based on this dataset, we fine-tuned a sentence transformer and an evaluation metric to improve their negation sensitivity.Evaluating these models on existing benchmarks shows that our fine-tuned models outperform existing metrics on the negated sentences by far while preserving their base models' performances on other perturbations.
Institution: Technical University of Munich
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W4389009552