BooST: Bridging Semantics and Motions for Efficient Skill Transfer

How can robots learn reusable, temporally extended behaviors that are simultaneously robust to visual noise, generalizable across different tasks and embodiments, and computationally efficient enough for real-world deployment? The authors identify that existing methods often fail to bridge the gap between high-level semantic intent (what the robot should do) and low-level motion dynamics (how the robot should move), leading to weak priors that require excessive data for downstream adaptation.

BooST (Bridging Semantics and Motions for Efficient Skill Transfer) addresses this through a decoupled, two-stage training paradigm:

1. Unified Skill Pretraining: The model uses a cross-modal VQ-VAE with two parallel pathways. A visuo-linguistic pathway uses a pretrained CLIP encoder and cross-attention to extract semantic intent from images and language. Simultaneously, an action pathway encodes motion dynamics from trajectories. These are fused into a shared, discrete skill codebook, optimized via action reconstruction.
2. Downstream Adaptation: The rich, pretrained representations are distilled into a lightweight policy and a causal skill prior. This stage uses only past observations, making it suitable for real-time execution on physical robots. By training on a smaller, in-domain dataset, the model adapts to new tasks with minimal demonstrations.

By explicitly grounding semantic intent in executable motion dynamics, BooST achieves superior few-shot adaptation compared to methods that rely solely on visual features or raw action sequences. Its ability to ignore dynamic visual distractors and background variations—while maintaining a lightweight architecture—makes it a practical solution for deploying general-purpose robotic agents in unpredictable, real-world environments.