In-Context Model Predictive Generation: Open-Vocabulary Motion Synthesis from Language Models to Physics

How can we synthesize human motion from open-vocabulary text that is both semantically accurate and physically plausible, without the need for task-specific policy retraining?

The authors propose In-Context Model Predictive Generation (ICMPG), a framework that treats motion synthesis as a closed-loop, receding-horizon optimization problem. The system consists of two primary modules:

1. Context-Aware Motion Generation (CAMG): An LLM-based planner decomposes natural language instructions into atomic motion primitives. It uses in-context learning to map these instructions to discrete motion tokens, which are then decoded into 3D motion sequences.
2. Model Predictive Generation (MPG): This module acts as a controller. It evaluates candidate motion sequences generated by the CAMG module by simulating them in a physical environment and calculating a composite reward based on both physical feasibility (e.g., gravity, contact) and semantic alignment with the original text. The best-performing sequence is selected and used to guide the next step of generation.

This approach avoids the "open-loop" problem where LLMs generate semantically correct but physically impossible movements. By performing optimization at inference time, ICMPG adapts to diverse, unseen prompts without requiring expensive offline reinforcement learning.

Existing text-to-motion models often struggle with a trade-off: LLM-based methods are flexible but physically unstable, while physics-aware models are rigid and struggle with complex, open-vocabulary instructions. ICMPG bridges this gap by leveraging the reasoning power of frozen LLMs alongside the grounding capabilities of physics engines. This makes it a highly versatile tool for applications like virtual reality and character animation, where both semantic nuance and physical realism are critical.