Year: 2026
Type: preprint
Abstract: Large Language Models are now being developed specifically for African languages, with models like N-ATLaS and Pawa-Gemma-Swahili achieving meaningful capability gains on standard NLP benchmarks. Yet while capability evaluation has advanced through benchmarks like IrokoBench and AfroBench, safety evaluation for these models remains less or nonexistent. Existing multilingual safety research focuses on using low-resource languages to attack frontier models such as GPT-4, leaving African-language LLMs themselves unexamined. We present the first systematic red-team evaluation of African-language LLMs, testing two publicly available models across four languages (Hausa, Swahili, Yoruba, Igbo), eight harm categories organized into four HEVP threat domains, and ten adversarial techniques. Across 220 trials using 24 culturally grounded attack configurations referencing real African systems (M-Pesa mobile money, Nigeria's National Identification Number and Bank Verification Number (NIN/BVN), Nigeria's Independent National Electoral Commission (INEC), and Safaricom telecommunications), we find that 44.1% of attacks elicit harmful content, with all four languages vulnerable at comparable rates (40.0--48.0%). The HEVP taxonomy reveals structured vulnerability: Propaganda (70.0%) and Harm Generation (58.0%) are most exploitable, while hate/harassment shows the strongest defense (12.0%). Role Exploitation techniques achieve 82.5% success, outperforming all other technique classes. Model size does not predict safety; the 8B-parameter and 2.6B-parameter models show equivalent vulnerability despite the smaller model incorporating Direct Preference Optimization (DPO) alignment. Defense mechanisms vary qualitatively by language, from disclaimers in Hausa to truncation in Swahili. Safety evaluation should become routine in African-language model development.
Source: Research Square
Institutions: Carnegie Mellon University Africa, Carnegie Mellon University
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7225399035