Work

That Ain’t Right: Assessing LLM Performance on QA in African American and West African English Dialects

Year: 2025

Type: conference-paper

Abstract: As Large Language Models (LLMs) gain mainstream public usage, understanding how users interact with them becomes increasingly important.Limited variety in training data raises concerns about LLM reliability across different language inputs.To explore this, we test several LLMs using functionally equivalent prompts expressed in different English dialects.We frame this analysis using Question-Answer (QA) pairs, which allow us to detect and evaluate appropriate and anomalous model behavior.We contribute a cross-LLM testing method and a new QA dataset translated into AAVE and WAPE variants.Results show a notable drop in accuracy for one dialect relative to the baseline.

Source: Widening NLP Workshop (WiNLP)

DOI: 10.18653/v1/2025.winlp-main.21