Naviblind: A multimodal AI assistant for visually impaired users to identify product information from images and speech
Year: 2025
Type: article
Abstract: People with visual impairments often face significant challenges in identifying and accessing product information in their daily lives, particularly when visual cues such as packaging details, labels, or expiration dates are inaccessible. In this paper, we present NaviBlind, a multimodal AI-powered assistive system designed to help visually impaired individuals understand key product details through natural interactions. Our system combines image understanding using Gemini Flash vision models with Vietnamese speech recognition powered by PhoWhisper for extracting information needs directly from user voice commands. By uploading an image of the product and speaking what kind of information is needed, such as name, color, type, or expiry date, the system analyzes the image and returns a concise, structured textual description, which is then converted into Vietnamese speech. To ensure reliability, we incorporate mechanisms to detect uncertain or hallucinated outputs from the vision model, especially in cases of low-quality images. The system is deployed as a user-friendly web application, enabling real-time accessibility for users with limited visual capabilities. Experimental evaluation demonstrates the potential of NaviBlind in promoting autonomy and independence for the visually impaired in everyday shopping and product recognition tasks.
Authors: Minh-Quan Tran, Duy Truong, Duy-Tan Pham, Minh-Anh Nguyen, Duc-Tung Le, Di-Hao Le, Quang-Huy Duong
Language: English
Subfield: Occupational Therapy
Field: Health Professions
Domain: Health Sciences
Open Access status: diamond
OpenAlex ID: W4415238961
FWCI:
Cites:
Cited by:
Related to: