Work

Beyond Factual Accuracy: Evaluating Coverage of Diverse Factual Information in Long-form Text Generation

Year: 2025

Type: conference-paper

Abstract: This paper presents ICAT, 1 an evaluation framework for measuring coverage of diverse factual information in long-form text generation.ICAT breaks down a long output text into a list of atomic claims and not only verifies each claim through retrieval from a (reliable) knowledge source, but also computes the alignment between the atomic factual claims and various aspects expected to be presented in the output.We study three implementations of the ICAT framework, each with a different assumption on the availability of aspects and alignment method.By adopting data from the diversification task in the TREC Web Track and the ClueWeb corpus, we evaluate the ICAT framework.We demonstrate strong correlation with human judgments and provide comprehensive evaluation across multiple state-ofthe-art LLMs.Our framework further offers interpretable and fine-grained analysis of diversity and coverage.Its modular design allows for easy adaptation to different domains and datasets, making it a valuable tool for evaluating the qualitative aspects of long-form responses produced by LLMs.

DOI: 10.18653/v1/2025.findings-acl.693

Authors: Chris Samarinas, Alexander Krubner, Alireza Salemi, Youngwoo Kim, Hamed Zamani

Institutions: University of Massachusetts Amherst, Fachhochschule Salzburg

Language: English

Topic: Topic Modeling

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: gold

Funders: National Science Foundation, Office of Naval Research

Awards: CAREER: Enriching Conversational Information Retrieval via Mixed-Initiative Interactions, G6128554767, III: Medium: Collaborative Research: Athena: Learning-oriented Search with Personalized Learning Flows

OpenAlex ID: W4412888310