A HamNoSys-Guided Dataset and Baselines for Fine-Grained Isolated Handshape Recognition in Sign Language

Research Question and Approach

Computational sign language recognition often struggles with the lack of standardized, fine-grained handshape inventories. While many datasets exist for fingerspelling or specific lexical signs, there is a need for a resource that captures a broad, phonetically defined range of handshapes. This paper addresses this by creating a dataset grounded in the Hamburg Notation System (HamNoSys), a language-independent phonetic transcription system. The researchers collected 144,000 RGB images from 15 participants, covering 160 distinct handshape classes derived from the official HamNoSys 4 Handshapes Chart. They evaluated four baseline model families—ResNet-18, ViT-B/16, a graph convolutional network, and XGBoost—using both subject-dependent and leave-one-subject-out (LOSO) protocols to measure generalization capabilities.

The study establishes a robust benchmark for fine-grained handshape recognition. In subject-dependent evaluations, where test data includes participants seen during training, the models achieved high performance. However, the LOSO protocol revealed a substantial performance gap, highlighting the difficulty of generalizing handshape recognition to unseen signers. The authors also performed cross-dataset evaluations on LSWH100 and ASL Fingerspelling Dataset A, confirming that their models provide a reliable reference for isolated handshape classification across different contexts.

By providing a large, balanced, and systematically curated dataset, this work offers a critical resource for developing more accessible sign language technologies. The use of HamNoSys ensures that the dataset is linguistically grounded and cross-linguistically applicable, moving beyond the limitations of language-specific fingerspelling alphabets. The inclusion of both subject-dependent and subject-independent evaluation protocols provides a clear roadmap for researchers to assess how well their models will perform in real-world scenarios where the system encounters new users.