Phoneme-Level Visual Speech Recognition via Point-Visual Fusion and Language Model Reconstruction
Year: 2026
Type: conference-paper
Abstract: Visual Automatic Speech Recognition (V-ASR) is a challenging task that involves interpreting spoken language solely from visual information, such as lip movements and speaker-dependent variance. A major difficulty arises from viseme ambiguity, where multiple phonemes share similar visual appearances, making word- or character-level prediction errorprone. To address this, our work focuses on phoneme-level modeling, which provides a finer linguistic granularity than visemes while reducing the reliance on large-scale pretraining. We propose a novel phoneme-based two-stage framework that fuses visual and landmark motion features, followed by a Large Language Model (LLM) for sentence reconstruction to address these challenges. Stage 1 consists of Point Visual Automatic Speech Recognition (PV-ASR), which outputs the predicted phonemes addressing speaker-specific facial characteristics. Stage 2 comprises No Language Left Behind (NLLB) LLM, which reconstructs the output phonemes back to words. Our method demonstrates promising performance in terms of Word Error Rates (WER), especially on LRS2 (16.0% WER) while maintaining competitive results on LRS3 (20.3% WER).
Institution: Kyushu Institute of Technology
Language: English
Topic: Speech and Audio Processing
Subfield: Signal Processing
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W4417262956
FWCI:
Cites:
Cited by:
Related to: