Ushnish Sarkar, Suvajit Patra, Bhaswar Chattopadhyay, Pranab Singha Roy, Tapas Samanta
6 min
Computational sign language recognition often struggles with the lack of standardized, fine-grained handshape inventories. While many datasets exist for fingerspelling or specific lexical signs, there is a need for a resource that captures a broad, phonetically defined range of handshapes. This paper addresses this by creating a dataset grounded in the Hamburg Notation System (HamNoSys), a language-independent phonetic transcription system. The researchers collected 144,000 RGB images from 15 participants, covering 160 distinct handshape classes derived from the official HamNoSys 4 Handshapes Chart. They evaluated four baseline model families—ResNet-18, ViT-B/16, a graph convolutional network, and XGBoost—using both subject-dependent and leave-one-subject-out (LOSO) protocols to measure generalization capabilities.
The study establishes a robust benchmark for fine-grained handshape recognition. In subject-dependent evaluations, where test data includes participants seen during training, the models achieved high performance. However, the LOSO protocol revealed a substantial performance gap, highlighting the difficulty of generalizing handshape recognition to unseen signers. The authors also performed cross-dataset evaluations on LSWH100 and ASL Fingerspelling Dataset A, confirming that their models provide a reliable reference for isolated handshape classification across different contexts.
By providing a large, balanced, and systematically curated dataset, this work offers a critical resource for developing more accessible sign language technologies. The use of HamNoSys ensures that the dataset is linguistically grounded and cross-linguistically applicable, moving beyond the limitations of language-specific fingerspelling alphabets. The inclusion of both subject-dependent and subject-independent evaluation protocols provides a clear roadmap for researchers to assess how well their models will perform in real-world scenarios where the system encounters new users.
Purpose: Fine-grained handshape recognition supports computational sign-language transcription, recognition, and translation, but broad, phonetically defined visual inventories with signer-aware evaluation remain limited. This work introduces a benchmark grounded in the language-independent Hamburg Notation System (HamNoSys). Methods: A balanced dataset of 144,000 RGB images was collected from 15 participants for 160 handshape classes defined by the official HamNoSys 4 Handshapes Chart. ResNet-18 and ViT-B/16 were evaluated as appearance-based models, while a graph convolutional network and XGBoost were evaluated from hand landmarks. Both a class-stratified subject-dependent split and a 15-fold leave-one-subject-out (LOSO) protocol were used. The same model families were additionally assessed on LSWH100 and ASL Fingerspelling Dataset A for external context. Results: The subject-dependent benchmarks established reproducible reference performance across all four model families, whereas LOSO evaluation exposed a substantial reduction when recognition was required to generalise to unseen participants. On ASL Fingerspelling Dataset A, mean LOSO top-1 accuracy ranged from 82.20% to 87.40%. Conclusion: The documented acquisition, curation, and complementary evaluation protocols pro-vide a reproducible resource for fine-grained isolated-handshape research and for developing more accessible sign-language technologies.
Alex: Think of it this way. If the same person appears in both the training data and the test data, the model might quietly learn that person's specific hand size, skin tone, or the angle they tend to hold their wrist. It looks like it's recognizing handshapes, but it's partly just recognizing that person. That's a shortcut, and it inflates the results.
Sam: So how do you stop that from happening?
Alex: You design the test so that one participant is completely hidden from the model during training. The model learns from the other fourteen people, then gets tested on the one it never saw. This is called leave-one-subject-out testing. You repeat that process until every participant has had a turn as the hidden test case. That way, the test is asking a genuinely harder question: does the model understand the handshape itself, or just the people it's already met?
Sam: And the results drop when you use that stricter test?
Alex: Noticeably. The seen-signer test — where the same people appear in training and testing — was considerably easier. The held-out-signer test was far harder. The gap between those two results is actually one of the paper's most informative findings, because it shows exactly where the shortcut was happening. The same handshape, shown by a new person with a slightly different hand size or finger angle, becomes much harder to classify.
Sam: So the model is less like a reader and more like a memorizer, unless you force it to generalize.
Alex: That's a fair way to put it. And the 160-class inventory makes this especially demanding, because some of those classes differ only in small finger details — one finger slightly bent, or the thumb in a slightly different position. The paper suggests that better ways to capture those fine local details may be needed.
Sam: What kinds of models did they actually test?
Alex: Two broad approaches. One looks directly at the hand image — the raw photograph. The other uses a simplified map of the hand's key points, like the wrist, knuckles, and fingertip positions. Think of it like the difference between looking at a full photograph of a hand versus looking at a stick-figure outline of the same hand.
Sam: Which one worked better?
Alex: On the easier seen-signer test, the image-based models did better — the strongest one got roughly 86% correct on its top guess. That suggests fine visual detail in the photograph still carries useful information for distinguishing subtle hand differences. But the paper also found that connecting the key points in a skeleton-like structure — rather than treating them as isolated dots — helped the landmark-based models perform better. Linking the points the way bones connect in a real hand lets the model use the hand's structure, not just the positions of individual points.
Sam: So the structure matters, not just the location of the points.
Alex: Exactly. And that's an intuitive result — a hand is a connected system, so treating it as one makes sense.
Sam: What are the limits of this work?
Alex: The paper is clear about them. The participants were university students, so the dataset doesn't represent the full range of people who use sign language. The controlled indoor setting means it won't automatically transfer to messier real-world conditions. And the dataset only covers static, single-hand forms — it doesn't include two-handed signs, movement, or the other components that make up a full sign. So this is one carefully constructed piece of a much larger puzzle.
Sam: But the benchmark itself is useful because it separates easy recognition from real generalization.
Alex: That's the core value. For anyone building sign language transcription tools — systems that convert visible signing into a written or symbolic form for analysis — knowing where a model actually fails is more useful than a number that looks good but hides shortcuts. This paper gives the field a more honest way to measure progress on one important piece of the problem.
Sam: That seems like a meaningful contribution, even if it's just one piece.
Alex: It is. Clean labels, a controlled setting, and a test that actually checks generalization — that combination makes the benchmark useful for comparing future methods fairly. The hard part is still unsolved, but now there's a clearer way to measure it. Thanks for listening to ResearchPod.