Word Overlap Artifacts in Textual Entailment Models: A Study with Multilingual Back Translation on the SNLI Dataset
Year: 2024
Type: conference-paper
Abstract: This paper investigates the domain of Natural Language Inference (NLI), with an emphasis on Recognizing Textual Entailment (RTE). We utilize the Stanford Natural Language Inference (SNLI) dataset, a benchmark for RTE tasks, to examine the efficacy of machine back translation and model performance in textual entailment. Our methodology employs a cost-effective approach using an open-source machine translation library like MarianMT with Helsinki-NLP/opus-mt models for back translation, applied to the comprehensive SNLI dataset. The concluding analysis demonstrates that no single model, whether back translated or augmented, consistently outperforms the reference English model in all aspects. The performance variations are particular to certain word overlap ranges and categories, suggesting that these models are essentially equivalent to the reference. This study contributes to the comprehension of machine translation′s impact on textual entailment models, emphasizing the complexities in multilingual NLI tasks.
Authors: Franklin Quilumba, Fausto Valencia, Hugo Arcos
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
OpenAlex ID: W4404489829