SignFlow: Real-Time Multi-Modal Sign Language Recognition via 12-Stream Hybrid Architecture on the Full-Vocabulary ASL Citizen Benchmark
Year: 2026
Type: preprint
Abstract: Isolated sign language recognition (ISLR) across comprehensive natural vocabularies is constrained by signer heterogeneity, subtle phonemic contrasts, and high computational latency on edge hardware. We present SignFlow, an end- to-end, on-device sign language recognition system evaluated on the complete official ASL Citizen benchmark (2,731 glosses, 32,940 test clips, evaluated across strictly held-out signers). SignFlow introduces an asymmetrical 12- stream hybrid architecture integrating: (i) four Landmark Transformers initialized via self-supervised masked-frame pretraining; (ii) three 1D-CNN temporal convolutional experts with dilated multi-round receptive fields; and (iii) five VideoMAE appearance models capturing full-frame dynamics and hand- crop details. Fusion weights and class-conditional temperature parameters are fit strictly on the official validation split via L-BFGS log-linear cross- entropy optimization. Evaluated on the 32,940-clip signer-independent test split, SignFlow achieves 87.04% Top-1 (28,670/32,940 correct), 98.27% Top-5, and 98.89% Top-10 accuracy. A single 1D-CNN expert (conv3) attains 85.16% Top- 1 solo. Signer-clustered bootstrap resampling (B=2,000) establishes a 95% confidence interval of [85.61%, 88.35%]. The entire pipeline operates locally on consumer hardware with an inference latency of 72 ms per clip on an NVIDIA RTX 3070 Ti laptop GPU.
Author: G, Nithin
Language: English
Subfield: Human-Computer Interaction
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7225658074
Cites:
Cited by:
Related to: