Work

Disability-First AI Dataset Annotation: Co-designing Stuttered Speech Annotation Guidelines with People Who Stutter

Year: 2026

Type: conference-paper

Abstract: Despite efforts to increase the representation of disabled people in AI datasets, accessibility datasets are often annotated by crowdworkers without disability-specific expertise, leading to inconsistent or inaccurate labels. This paper examines these annotation challenges through a case study of annotating speech data from people who stutter (PWS). Given the variability of stuttering and differing views on how it manifests, annotating and transcribing stuttered speech remains difficult, even for trained professionals. Through interviews and co-design workshops with PWS and domain experts, we identify challenges in stuttered speech annotation and develop practices that integrate the lived experiences of PWS into the annotation process. Our findings highlight the value of embodied knowledge in improving dataset quality, while revealing tensions between the complexity of disability experiences and the rigidity of static labels. We conclude with implications for disability-first and multiplicity-aware approaches to data interpretation across the AI pipeline.

Source: CHI Conference on Human Factors in Computing Systems

DOI: 10.1145/3772318.3790405

Authors: Xinru Tang, Jingjin Li, Shaomei Wu

Institution: University of California, Irvine

Language: English

Topic: Stuttering Research and Treatment

Subfield: Clinical Psychology

Field: Psychology

Domain: Social Sciences

SDG: Reduced inequalities

Open Access status: hybrid

Funder: National Science Foundation