Work

Weaving Sound Information to Support Real-Time Sensemaking of Auditory Environments: Co-Designing with a DHH User

Year: 2025

Type: conference-paper

Abstract: Current AI sound awareness systems can provide deaf and hard of hearing people with information about sounds, including discrete sound sources and transcriptions. However, synthesizing AI outputs based on DHH people's ever-changing intents in complex auditory environments remains a challenge. In this paper, we describe the co-design process of SoundWeaver, a sound awareness system prototype that dynamically weaves AI outputs from different AI models based on users’ intents and presents synthesized information through a heads-up display. Adopting a Research through Design perspective, we created SoundWeaver with one DHH co-designer, adapting it to his personal contexts and goals (e.g., cooking at home and chatting in a game store). Through this process, we present design implications for the future of “intent-driven” AI systems for sound accessibility.

Source: CHI Conference on Human Factors in Computing Systems

DOI: 10.1145/3706598.3714268

Authors: Jeremy Zhengqi Huang, Jaylin Herskovitz, Liang-Yuan Wu, Cecily Morrison, Dhruv Jain

Institutions: University of Michigan, Microsoft Research (United Kingdom)

Language: English

Topic: Assistive Technology in Communication and Mobility

Subfield: Occupational Therapy

Field: Health Professions

Domain: Health Sciences

SDG: Reduced inequalities

Open Access status: hybrid

OpenAlex ID: W4409735443

FWCI: 28.27

Cites: 53

Cited by: 9

Related to: 10