SIGN LANGUAGE RECOGNITION USING ORIGINAL AND SYNTHETIC DEPTH IMAGE BASED POINT CLOUD DATA MODELS

Sign language recognition (SLR) is a challenging task in computer vision, often hindered by the scarcity of high-quality depth datasets. While RGB images are widely available, they are susceptible to lighting and color variations. Depth images, which provide 3D spatial information, are more robust but require specialized hardware. This paper investigates whether synthetic depth images, generated from monocular RGB images using the Depth Anything V2 network, can serve as a viable alternative to original depth data for point cloud-based SLR.

The authors utilized three public sign language datasets (Real-time ASL Fingerspelling, KArSL, and AUTSL) that contain both RGB and depth information. They generated synthetic depth maps from the RGB data and converted both original and synthetic depth images into point clouds. These point clouds were then processed using PointNet architectures under three data models: frame-based classification, Point Gesture Maps (PGM) for spatial aggregation, and Long Short-Term Memory (LSTM) networks for capturing temporal dynamics in continuous sign language.

The results demonstrate that synthetic depth-based point clouds are capable of achieving competitive classification accuracy across most models. While original depth data generally yielded superior performance, the synthetic models performed better in specific configurations, suggesting that generative depth estimation is a promising approach for expanding the utility of existing RGB-only datasets. This work is significant because it provides a pathway to leverage the vast amount of available RGB video data for 3D-based gesture recognition without the need for specialized depth-sensing hardware.