Work

ImaginE: An Imagination-Based Automatic Evaluation Metric for Natural Language Generation

Year: 2023

Type: conference-paper

Abstract: Automatic evaluations for natural language generation (NLG) conventionally rely on tokenlevel or embedding-level comparisons with the text references.This is different from human language processing, for which visual imagination often improves comprehension.In this work, we propose IMAGINE, an imaginationbased automatic evaluation metric for natural language generation.With the help of Sta-bleDiffusion (Rombach et al., 2022), a stateof-the-art text-to-image generator, we automatically generate an image as the embodied imagination for the text snippet and compute the imagination similarity using contextual embeddings.Experiments spanning several text generation tasks demonstrate that adding machinegenerated images with our IMAGINE displays great potential in introducing multi-modal information into NLG evaluation, and improves existing automatic metrics' correlations with human similarity judgments in both referencebased and reference-free evaluation scenarios.

Source: Findings of the Association for Computational Linguistics: EACL

DOI: 10.18653/v1/2023.findings-eacl.6

Authors: Wanrong Zhu, Xin Wang, An Yan, Miguel Eckstein, William Yang Wang

Institution: University of California, Santa Cruz

Language: English

Topic: Natural Language Processing Techniques

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: hybrid

Funders: National Science Foundation, Army Research Office, Institute for Collaborative Biotechnologies

Award: CAREER: Faithful Natural Language Generation