Year: 2024
Type: conference-paper
Abstract: Metaphors in natural language are a reflection of fundamental cognitive processes such as analogical reasoning and categorisation, and are deeply rooted in everyday communication.Metaphor understanding is therefore an essential task for large language models (LLMs).We release the Metaphor Understanding Challenge Dataset (MUNCH), designed to evaluate the metaphor understanding capabilities of LLMs.The dataset provides over 10k paraphrases for sentences containing metaphor use, as well as 1.5k instances containing inapt paraphrases.The inapt paraphrases were carefully selected to serve as control to determine whether the model indeed performs full metaphor interpretation or rather resorts to lexical similarity.All apt and inapt paraphrases were manually annotated.The metaphorical sentences cover natural metaphor uses across 4 genres (academic, news, fiction, and conversation), and they exhibit different levels of novelty.Experiments with LLaMA and GPT-3.5 demonstrate that MUNCH presents a challenging task for LLMs.The dataset is freely accessible at https://github.com/xiaoyuisrain/ metaphor-understanding-challenge.
Authors: Xiaoyu Tong, Rochelle Choenni, Martha Lewis, Ekaterina Shutova
Institutions: University of Amsterdam, Santa Fe Institute, University of Bristol
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
Funder: University of Bristol
OpenAlex ID: W4402672083