Work

H andscribe : A gloss-free framework for sign language translation and gloss sequence generation

Year: 2026

Type: article

Abstract: Sign language translation systems traditionally rely on intermediate gloss representations to bridge the gap between visual input and written language output. However, manual gloss annotation is costly, language-dependent, and often lossy, prompting growing interest in gloss-free alternatives. This paper introduces H andscribe , a novel two-stage framework for gloss-free sign language translation and gloss sequence generation. H andscribe first translates continuous sign language videos into written language sentences using a lightweight decoder built atop SlowFast-based spatiotemporal features and a frozen mBART model. Then, in the second stage, it generates gloss sequences from these sentences using a Large Language Model (LLaMa3.1-8B-Instruct) that has been fine-tuned with weak supervision. Our experiments on PHOENIX-2014-T and Wav2Gloss Fieldwork demonstrate strong translation performance and state-of-the-art multilingual gloss generation, even in zero-shot settings. The proposed framework reduces annotation bottlenecks while maintaining flexibility and interpretability, paving the way for scalable and inclusive sign language technologies. The code and fine-tuning scripts are available at https://github.com/colonnaemanuele/Handscribe .

Source: Computer Vision and Image Understanding

DOI: 10.1016/j.cviu.2026.104674

Authors: Emanuele Colonna, Ivan Rinaldi, David Landi, Gennaro Vessio, Giovanna Castellano

Institution: University of Bari Aldo Moro

Language: English

Topic: Hand Gesture Recognition Systems

Subfield: Human-Computer Interaction

Field: Computer Science

Domain: Physical Sciences

Open Access status: hybrid

Estimated APC paid: $2,090

Funder: Universita degli Studi di Bari Aldo Moro