Text2Sign: A Single-GPU Diffusion Baseline for Text-to-Sign Language Video Generation

Generating sign language video from text is computationally expensive, often requiring multi-node infrastructure to handle the complex spatiotemporal modeling of human motion. This paper asks whether a diffusion-based architecture can be optimized to run on a single consumer-grade GPU while maintaining motion coherence and text-to-video alignment.

The author introduces Text2Sign, a 3D UNet-based diffusion model designed for efficiency. To overcome the memory bottleneck of full 3D attention, the model employs factorized spatiotemporal attention, which separates spatial and temporal operations. It also uses a frozen CLIP text encoder to provide semantic conditioning without the overhead of training a custom text-processing module. The model was evaluated on the How2Sign dataset using a signer-disjoint protocol to ensure the model generalizes to new signers rather than memorizing training data.

Text2Sign demonstrates that high-quality, temporally consistent sign language video can be generated on a single NVIDIA L4 GPU. The factorized attention mechanism successfully reduces memory usage while preserving motion smoothness. Quantitative results on a compact evaluation slice show a temporal consistency score of 1.0000, indicating stable motion. However, a denoising audit revealed that the model's output is relatively insensitive to specific text prompts, as shuffled prompts produced results similar to correct ones. This suggests that while the model captures the general visual style of signing, it does not yet achieve deep linguistic grounding.

This work provides a practical, open-source baseline for researchers working on sign language production who lack access to large-scale computing clusters. By demonstrating that single-GPU training is feasible, the paper lowers the barrier to entry for developing assistive communication technologies, though it emphasizes that further work is needed to improve linguistic accuracy and prompt adherence.