SHuBERT: Self-Supervised Sign Language Representation Learning via Multi-Stream Cluster Prediction
Year: 2025
Type: conference-paper
Abstract: Sign language processing has traditionally relied on task-specific models, limiting the potential for transfer learning across tasks.Pretraining methods for sign language have typically focused on either supervised pre-training, which cannot take advantage of unlabeled data, or context-independent (frame or video segment) representations, which ignore the effects of relationships across time in sign language.We introduce SHuBERT (Sign Hidden-Unit BERT), a self-supervised contextual representation model learned from approximately 1,000 hours of American Sign Language video.SHu-BERT adapts masked token prediction objectives to multi-stream visual sign language input, learning to predict multiple targets corresponding to clustered hand, face, and body pose streams.SHuBERT achieves state-of-theart performance across multiple tasks including sign language translation, isolated sign language recognition, and fingerspelling detection.
Language: English
Subfield: Human-Computer Interaction
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W4412889773