Year: 2025
Type: article
Abstract: AIM: Sensory substitution devices (SSDs) can convert environmental information into an accessible format for people who are blind or have low vision (pBLV). Yet, current SSDs often passively deliver all of the information available with limited user control, potentially leading to confusion and/or cognitive overload. To address this issue, this work proposes a selective, gesture-controlled system intended to improve information relevance and reduce cognitive overload. METHODS: We present Point-To-Tell 2, a system that enables pBLV to privately and efficiently select which information to convey through simple pointing-based gestural control. By integrating a monocular camera with AI-driven pipelines for depth estimation, hand pose tracking, and object detection/segmentation, the system identifies the users' 3D pointing direction and announces the names and distances of objects as they are pointed at, thereby connecting an object's spatial position and identity through hand proprioception. RESULTS: Validation tests in controlled indoor environments show high hand pose tracking accuracy, ensuring reliable ray-casting and object selection despite declining object recognition at longer distances. Distance estimates are stable at close range, though a systematic bias is present. CONCLUSION: This work introduces and technically validates an assistive system designed to improve the usability of assistive technologies by focusing system feedback-potentially reducing users' cognitive load and enhancing their spatial comprehension by leveraging concurrent hand proprioception. Future work will involve user testing and expanding system features to further enhance its practicality across more diverse scenarios.
Institution: New York University
Language: English
Subfield: Cognitive Neuroscience
Field: Neuroscience
Domain: Life Sciences
Open Access status: hybrid
Awards: G286800492, NSF Convergence Accelerator Track H: Smart Wearables for Expanding Workplace Access for People with Blindness and Low Vision, Co-Development of Telehealth, Remote Patient Monitoring, and AI-based Tools for Inclusive Technology-Facilitated Healthcare Work of the Future, G5102883099, SCC-IRG Track 2: Transportation Gaps and Disability-Related Unemployment: Smarter Cities and Wearables combating Commuting Challenges for the Visually Impaired, NSF Convergence Accelerator Track H: Phase II Smart Wearables for Expanding Workplace Access for People with Blindness and Low Vision, G9000048389, Accommodations for Blindness and Low Vision to optimize Access in Healthcare
OpenAlex ID: W4416409702