Temporal Robustness of Large Language Models for Thematic Classification of UN General Assembly Debates
Year: 2026
Type: article
Abstract: Thematic analysis of large-scale political discourse remains a challenge due to semantic complexity and overlapping policy areas and changing diplomatic vocabulary. Although large language models (LLMs) offer promise for scalable thematic classification, their reliability in politically sensitive contexts requires systematic validation against expert human annotations. We evaluate LLM-based thematic classification of United Nations General Assembly (UNGA) speeches across a decade (2014–2023), using 7680 human-annotated themes mapped into 12 policy domains. Our results show that DeepSeek R1 achieves the highest accuracy 77% (F1 = 0.73), followed by ChatGPT, Gemini and LLaMA, with strong performance in lexically stable domains but substantial degradation in semantically overlapping categories such as governance and international cooperation. A unique dimension of our work is timeline analysis, which shows that the performance of LLMs over the years varies strongly and the precision decreases during times of rhetorical transformation, including pandemic-related discussions and the discourses of cooperation determined by the Russia–Ukraine conflict. By linking domain-level ambiguity and geopolitical shifts to temporal instability, this study introduces a dynamic robustness perspective for evaluating LLMs in computational political discourse analysis.
Source: Information
DOI: 10.3390/info17060589
Language: English
Subfield: General Social Sciences
Field: Social Sciences
Domain: Social Sciences
Open Access status: gold
Estimated APC paid: $1,930
OpenAlex ID: W7164824022