An efficient spatial-temporal attention network for low-latency isolated Indian Sign Language Recognition
Year: 2026
Type: article
Abstract: About 63 million people in India live with significant hearing impairment, yet Indian Sign Language (ISL) is still poorly served by assistive digital tools even though it is central to communication within the Deaf and Hard-of-Hearing (DHH) community. We present STANet, a compact hybrid model for isolated ISL recognition that pairs a Time-Distributed Conv2D frame encoder with causal temporal convolution and multi-head self-attention. The contribution is not an individual module but the composition itself: the encoder feeds temporal convolution and attention with no recurrent layers, so the model captures motion across a clip while keeping per-clip inference cost low. We evaluate STANet on INCLUDE-50 and ISL-50 and compare it, under one shared pipeline, against 3D CNN baselines built earlier in the same study. The main evaluation is signer-dependent: the splits are stratified at the clip level, so clips from the same signers appear in both training and test. In that setting STANet reaches 88.36% accuracy and a macro F1-score of 0.8841 on INCLUDE-50, and 94.38% accuracy with a macro F1-score of 0.9399 on ISL-50. Residual errors do not spread across classes but concentrate in a few visually similar sign pairs. In an encoder-controlled ablation on a shared backbone, each temporal stage adds little on its own, while the two together account for most of the accuracy gain over the backbone. Paired McNemar tests on those per-clip predictions do not reach significance once corrected for multiple comparisons at this test-set size, so the complementarity of the two stages is reported as an observed effect size and not as statistically significant evidence. A separate leave-one-signer-out test tells a harder story: RGB accuracy falls to roughly 19%, but feeding pose landmarks to the same architecture recovers it to 58.4%, which points to pose geometry rather than raw appearance as what carries recognition across signers. Offline clip-level inference through an OpenCV pipeline takes 3.488 ms per sample on INCLUDE-50 and 3.481 ms on ISL-50 on an NVIDIA GeForce GTX 1650; these timings come from that single desktop GPU, and the model has not been profiled on mobile or embedded hardware.
Source: Discover Computing
Language: English
Subfield: Human-Computer Interaction
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
Estimated APC paid: $1,520
OpenAlex ID: W7225730151