Evaluating Relation Hallucination in Text Summarization: An Introduction to the Relation Hallucination Index
Year: 2024
Type: conference-paper
Abstract: Text summarization involves generating a concise, precise, and coherent summary that captures the essence of a longer document.Relation Hallucination refers to the generation of summary sentences that imply or state relationships between entities or events that were not present or implied in the original document.Relation Hallucination is a pivotal concern in abstractive text summarization, where models fabricate or exaggerate connections, leading to misleading summaries.This paper delves deep into the evaluation of Relation Hallucination in abstractive summarization.We introduce a novel metric, the Relation Hallucination Index , designed to evaluate and quantify Relation hallucinations across various state-of-the-art models.Emphasizing the paramount importance of maintaining accurate relational context in summaries, this article showcases the efficacy of the Relation hallucination index in discerning fabricated relationships.The Relation hallucination index provides a quantitative measure of the various levels of Relation Hallucination present in generated summaries, enabling practitioners from industry as well as Academia to select models aligned with desired hallucination parameters for their tailored applications.
Authors: Praveen Kumar Katwe, Rakesh Chandra Balabantaray, Kali Prasad Vittala
Institution: International Institute of Information Technology
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W4412638229