Year: 2023
Type: conference-paper
Abstract: Evaluation of natural language generation (NLG) is complex and multi-dimensional.Generated text can be evaluated for fluency, coherence, factuality, or any other dimensions of interest.Most frameworks that perform such multi-dimensional evaluation require training on large manually or synthetically generated datasets.In this paper, we study the efficacy of large language models as multi-dimensional evaluators using in-context learning, obviating the need for large training datasets.Our experiments show that in-context learning-based evaluators are competitive with learned evaluation frameworks for the task of text summarization, establishing state-of-the-art on dimensions such as relevance and factual consistency.We then analyze the effects of factors such as the selection and number of incontext examples on performance.Finally, we study the efficacy of in-context learningbased evaluators in evaluating zero-shot summaries written by large language models such as GPT-3.Our code is available at https: //github.com/JainSameer06/ICE
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W4385571776