Year: 2025
Type: conference-paper
Abstract: Significant efforts have been made in the NLP community to facilitate the automatic analysis of climate-related corpora by tasks such as climate-related topic detection, climate risk classification, question answering over climate topics, and many more.In this work, we perform a reproducibility study on 8 tasks and 29 datasets, testing 6 models.We find that many tasks rely heavily on surface-level keyword patterns rather than deeper semantic or contextual understanding.Moreover, we find that 96% of the datasets contain annotation issues, with 16.6% of the sampled wrong predictions of a zero-shot classifier being actually clear annotation mistakes, and 38.8% being ambiguous examples.These results call into question the reliability of current benchmarks to meaningfully compare models and highlight the need for improved annotation practices.We conclude by outlining actionable recommendations to enhance dataset quality and evaluation robustness.
Authors: Tom Calamai, Oana Balalau, Fabian M. Suchanek
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
SDG: Climate action
Open Access status: gold
OpenAlex ID: W4412888085