FineMoLA: Towards Fine-Grained Motion-Language Alignment from Clip-Level Supervision

Recent advancements in text-to-motion generation have been driven by large-scale datasets with long-form descriptions. However, these datasets typically provide supervision only at the clip level, lacking explicit temporal correspondence between specific motion frames and textual phrases. This paper addresses the challenge of learning fine-grained, frame-level alignment from these coarse, clip-level annotations without requiring expensive manual frame-level labels.

The authors propose FineMoLA, a framework that treats motion-language alignment as an optimal transport (OT) problem. The process involves three main steps:

1. Text Segmentation: An LLM decomposes long-form descriptions into discrete, action-bearing phrases.
2. Optimal Transport Formulation: The model computes a cost matrix based on the cosine similarity between motion frame features and text phrase features. It then uses the Sinkhorn algorithm with entropic regularization to solve for a transport plan that maps motion frames to text phrases under global marginal constraints.
3. Contrastive Training: The model is trained using a symmetric contrastive loss, where the pairwise alignment logits are derived from the computed OT cost rather than simple global feature similarity.

To handle motion frames that do not correspond to any specific action phrase, the model includes an [UNK] (unknown) token that acts as a sink for non-descriptive motion.

Experiments conducted on the SnapMoGen dataset demonstrate that FineMoLA successfully learns meaningful, fine-grained correspondences between motion frames and text. By leveraging the optimal transport formulation, the model outperforms baseline methods in motion-text grounding tasks. The results suggest that modeling many-to-many relationships through OT is a robust way to bridge the gap between coarse supervision and fine-grained temporal control.