When Annotators Disagree: Re-evaluating Hausa Sentiment Models
Year: 2026
Type: preprint
Abstract: This study investigates how disagreement among human annotators affects the evaluation of sentiment analysis models for Hausa, a low-resource African language. Using the Hausa subset of the AfriSenti benchmark, we examine whether evaluating models against individual annotator judgments produces different conclusions from conventional evaluation using a single reference label. The analysis covers 4,562 eligible social media posts with three distinct annotator votes and compares classical machine learning baselines with multilingual transformer models, including AfriBERTa, XLM-R, and mBERT. Model performance is assessed using macro-F1 scores, empirical vote agreement, and paired uncertainty estimates, with particular attention to unanimous and contested annotations. Results show that overall model rankings remain consistent across both evaluation metrics, although small, statistically uncertain differences emerge on posts where annotators disagree. The study also identifies 64 instances where released sentiment labels differ from the majority of individual annotator votes. These findings highlight the importance of preserving annotator perspectives, documenting annotation coverage, and accounting for uncertainty when evaluating natural language processing systems. The research contributes a reproducible evaluation audit for Hausa sentiment analysis and provides insights into more transparent and disagreement-aware evaluation practices for low-resource languages.
Institutions: Digital Umuganda (Rwanda), African Leadership University
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7230282629
