Co-policy: Responsive Human-Robot Co-Creation for Musical Performances

How can robots move beyond simple playback devices to become true co-creative partners in musical performance? The authors address the challenge of creating an embodied AI agent that can interpret human musical intent, generate complementary responses that respect physical constraints, and execute these responses with the low latency required for real-time interaction.

The authors introduce Co-policy, a modular framework that separates the process into three distinct stages: semantic intent grounding, constrained musical variation, and visuomotor execution.

1. Semantic Grounding: The system uses a fine-tuned Qwen-vl (F-Qwen) model, augmented with a semantic anchor bank, to translate human speech, live musical seeds, and visual observations into structured co-creation plans.
2. Constrained Variation: A planning layer ensures that the robot's musical contribution is not merely a copy of the human input. It enforces constraints such as harmonic validity, motif consistency, and physical playability (e.g., avoiding occluded or unreachable instrument parts).
3. Visuomotor Execution: The authors propose a Gaussian-Mixture Visuomotor Policy (GMP). Unlike diffusion-based policies that require iterative denoising—which can introduce latency—the GMP uses a conditional mixture-density policy to map visual context and target notes to robot actions in a single forward pass. This allows the robot to maintain multiple feasible action modes (like different striking postures) while ensuring real-time responsiveness.

Real-robot experiments involving chime performances demonstrate that Co-policy outperforms standard diffusion-policy baselines in intent alignment, execution accuracy, and interaction frequency. The modular design effectively balances the high-level semantic reasoning of foundation models with the low-level, high-speed requirements of physical robotic control. The authors show that their Guided Self-Attention (GSA) mechanism successfully couples global scene context with local, pixel-wise features, allowing the robot to accurately target specific instrument components during performance.