Cross-Sign Language Transfer Learning Using Domain Adaptation with Multi-scale Temporal Alignment

Sign Language Recognition (SLR) is critical for bridging communication barriers, yet high-resource datasets exist for only a handful of the more than 100 distinct sign languages used worldwide. Low-resource sign languages are prone to severe overfitting when trained independently. While previous transfer learning efforts often rely on pre-training networks on general computer vision datasets like ImageNet, these large-scale datasets only provide superficial, low-level visual features. This paper investigates whether transferring knowledge directly between different sign languages using domain adaptation can more effectively resolve the low-resource dataset problem.

The authors implement a two-step framework consisting of video pre-processing and domain adaptation. First, input videos from source and target sign languages are converted into frame-level representations using two modalities: RGB and Optical Flow via the Gunner Farneback algorithm. For domain adaptation, the study adopts the Temporal Attentive Adversarial Adaptation Network (TA3N), which integrates a Temporal Relation Network (TRN) module. This architecture aligns multi-scale temporal relations between source and target domains by assigning higher attention weights to N-frame temporal relations exhibiting greater domain discrepancy. The experiments systematically evaluate five multi-scale temporal relation configurations (3, 5, 7, 10, and 15 frames) across both full-scale and few-shot learning settings, using Argentine Sign Language (LSA) and Chinese Sign Language (CSL) as source domains and American Sign Language (ASL) as the target domain.

Experimental evaluations yield three primary insights. First, domain adaptation consistently outperforms conventional neural network-based pre-training and transfer learning methods for cross-sign language tasks. Second, aligning shorter-term temporal relations (such as 3-frame configurations) rather than longer-term timescales proves universally more effective at improving target model accuracy across all language pairs and input modalities. Third, comparisons between input modalities demonstrate that RGB frames outperform Optical Flow in the vast majority of experimental conditions, despite the added motion-tracking benefits of Optical Flow.