A Modular AI-Driven Multimodal Framework for Real-Time Classroom Interpretation for Deaf Students
Year: 2026
Type: preprint
Abstract: Despite significant advances in inclusive education, real-time classroom communication remains a major challenge for students with profound hearing impairment, particularly in science, technology, engineering, and mathematics (STEM) learning environments where continuous verbal interaction and concept-oriented instruction are essential. Existing assistive technologies, including speech-to-text captioning, frequency modulation (FM) systems, and human sign language interpreters, either provide limited contextual understanding, lack scalability, or are unavailable in many educational institutions. This paper presents a modular artificial intelligence (AI)-driven multimodal framework for real-time classroom interpretation designed to enhance accessibility for deaf learners. The proposed framework integrates Automatic Speech Recognition (ASR), educational semantic processing using Large Language Models (LLMs), Indian Sign Language (ISL) gloss generation, intelligent sign retrieval, and avatar-assisted visual communication within a unified architecture. In addition, the framework incorporates a systematic dataset engineering pipeline comprising vocabulary organization, automated metadata generation, annotation management, and gloss-based indexing to facilitate efficient retrieval and future scalability. Unlike conventional systems that primarily focus on isolated speech transcription or sign language translation, the proposed architecture emphasizes modularity, educational context awareness, extensibility, and deployment readiness for real classroom environments. The paper details the overall system architecture, workflow, and implementation methodology while highlighting the integration of multiple AI components into a cohesive assistive learning platform. The proposed framework establishes a scalable technological foundation for developing next-generation intelligent classroom interpretation systems and contributes toward advancing accessible, inclusive, and AI-enabled education for deaf students.
Source: Research Square
Authors: Shahir VK, Fathima Hisa
Institution: AWH Engineering College
Language: English
Subfield: Developmental and Educational Psychology
Field: Psychology
Domain: Social Sciences
Open Access status: green
OpenAlex ID: W7225747246