Few-Shot Prompting, Full-Scale Confusion: Evaluating Large Language Models for Humor Detection in Croatian Tweets
Year: 2025
Type: conference-paper
Abstract: Humor detection in low-resource languages is hampered by cultural nuance and subjective annotation.We test two large language models, GPT-4 and Gemini 2.5 Flash, on labeling humor in 6,000 Croatian tweets with expert gold labels generated through a rigorous annotation pipeline.LLM-human agreement (κ = 0.28) matches human-human agreement (κ = 0.27), while LLM-LLM agreement is substantially higher (κ = 0.63).Although concordance with expert adjudication is lower, additional metrics imply that the models equal a second human annotator while working far faster and at negligible cost.These findings suggest, even with simple prompting, LLMs can efficiently bootstrap subjective datasets and serve as practical annotation assistants in linguistically underrepresented settings.
Authors: Petra Bago, Nikola Bakarić
Language: English
Subfield: Social Psychology
Field: Psychology
Domain: Social Sciences
Open Access status: gold
Funder: Hrvatska Zaklada za Znanost
OpenAlex ID: W4412889119