MAIDR Meets AI: Exploring Multimodal LLM-Based Data Visualization Interpretation by and with Blind and Low-Vision Users
Year: 2024
Type: conference-paper
Abstract: This paper investigates how blind and low-vision (BLV) users interact with multimodal large language models (LLMs) to interpret data visualizations. Building upon our previous work on the multimodal access and interactive data representation (MAIDR) framework, our mixed-visual-ability team co-designed maidrAI, an LLM extension providing multiple AI responses to users’ visual queries. To explore generative AI-based data representation, we conducted user studies with 8 BLV participants, tasking them with interpreting box plots using our system. We examined how participants personalize LLMs through prompt engineering, their preferences for data visualization descriptions, and strategies for verifying LLM responses. Our findings highlight three dimensions affecting BLV users’ decision-making process: modal preference, LLM customization, and multimodal data representation. This research contributes to designing more accessible data visualization tools for BLV users and advances the understanding of inclusive generative AI applications.
Authors: JooYoung Seo, Sanchita S. Kamath, Aziz Zeidieh, Saairam Venkatesh, Sean McCurry
Institution: University of Illinois Urbana-Champaign
Language: English
Subfield: Human Factors and Ergonomics
Field: Social Sciences
Domain: Social Sciences
SDG: Reduced inequalities
Open Access status: hybrid
OpenAlex ID: W4403582792