Get started
Work

LCHAIM - Investigating Long Context Reasoning in Hebrew

Year: 2025
Type: conference-paper
Abstract: Natural Language Inference (NLI) has gained significant attention recently due to its importance in understanding how machines comprehend and reason about language.While English has received tremendous interest, low-resource languages like Hebrew require more research.In this paper, we address the evaluation of Hebrew NLI models by introducing LCHAIM, a dataset designed to evaluate these models on tasks involving long premises and complex reasoning.The dataset, created by translating and validating the English ConTRoL dataset, consists of 8,325 context-hypothesis pairs that require coreferential, temporal, logical and analytical reasoning.Our experiments show the difficulty of contextual reasoning in Hebrew, as evidenced by the performance of different models.Fine-tuning the LongHero model on both the shorter premise Hebrew NLI and the LCHAIM datasets yielded a mean accuracy of 52%, that is 35% less than human performance.Similarly, Large language Models (LLMs) like Gemma-9B, Dicta-LM-2.0-7B, and GPT-4o achieved a top mean accuracy of 60.12% in fewshot setting.The code and dataset are available in Github 1 .* Authors contributed equally 1 https://github.com/oriel9p/lchaim.git ‫האחרון‬ ‫ובעשור‬ ‫חוב‬ ‫באגרות‬ ‫מושקעים‬ ‫שקלים‬ ‫מיליארד‬ ‫וארבעה‬ ‫עשרים‬ :P ‫לאחרונה‬ ‫השתנו‬ ‫הזכייה‬ ‫סיכויי‬ ‫שבעה.‬‫פי‬ ‫גדל‬ ‫בהגרלה‬ ‫החוב‬ ‫אגרות‬ ‫מספר‬ ‫מכשיר‬ ‫שנדרש‬ ‫לכך‬ ‫הביאו‬ ‫שיא‬ ‫מכירות‬ ‫לאחד.‬‫ל-000,42‬ ‫לאחד‬ ‫מ-005,72‬ ‫להשלים‬ ‫שעות‬ ‫וחצי‬ ‫חמש‬ ‫לקח‬ ‫קודמו‬ ‫אקראי.‬‫באופן‬ ‫מספרים‬ ‫לבחירת‬ ‫חדש‬ ‫בכל‬ ‫מהזמן.‬‫בחצי‬ ‫המשימה‬ ‫את‬ ‫להשלים‬ ‫יכול‬
DOI: 10.18653/v1/2025.findings-acl.413
Language: English
Open Access status: gold
OpenAlex ID: W4412888484
FWCI:
Cites:
Cited by:
Related to: 10