Work

Reading the Meaning, Missing the Language: AI and Human Agreement on Nigerian Public Discourse.

Year: 2026
Type: preprint
Abstract: What we did. We report a reliability study and a multi-model evaluation of meaning in Nigerian public discourse. It uses the Meaning Intelligence Framework (MIF™), a nine-dimension schema that separates surface sentiment from communicative intent. Five Certified Meaning Intelligence Analysts independently triple-annotated 500 naturalistic social-media, news and speech items in Standard English, Nigerian English, Nigerian Pidgin and code-mixed speech, for 1,500 annotations. How far trained readers agree. No dimension reached Krippendorff's α = 0.667. Agreement was highest on register (0.60) and lowest on risk tier (0.24). Analysts certified at 85–100% on constructed items agreed far less on real ones. What we tested. We built a two-tier gold standard and evaluated six models from OpenAI, Google and Anthropic, scoring each against the same analyst pairs as a leave-one-out human baseline. A second expert, blind to the first adjudicator's labels, re-adjudicated 50 contested items. The two experts agreed on 78–82% of register, sentiment, irony and risk labels, but on only 56% of recommended actions. Where models stand. Models matched analysts on intent family (0.73–0.77 against 0.76) and exceeded them on subtext presence. They fell short on recommended action, though part of that gap reflects contested expert judgement: against the second adjudicator, three models' action agreement rose by 0.18–0.24. All models fell short on register. Analysts recognised Nigerian Pidgin at 0.92; the best models reached 0.59–0.60, and Gemini models 0.11–0.15. What changed the result. Expert knowledge from the MIF changed the register result. A sharpened MIF register definition, which describes Pidgin by its grammatical markers, raised model Pidgin recognition from 0.20–0.47 to 0.90–0.96, level with the analysts. It came at a cost to code-mixed accuracy and awaits confirmation on held-out items. In a controlled probe, OpenAI models refused three benign Pidgin and code-mixed posts in 6 of 6 calls, and answered their Standard English renderings in 6 of 6. Conclusion. Current models increasingly read Nigerian meaning but misrecognise Nigerian language varieties. A small amount of expert knowledge closes much of that gap, and responsible deployment requires variety-level auditing.
Language: English
Open Access status: green
OpenAlex ID: W7232156231
Cites:
Cited by:
Related to: 10