ROAD-VLA: Robust Online Adaptation via Self-Distillation for Vision-Language-Action Models

How can vision-language-action (VLA) models, which are typically pretrained on large-scale datasets, be effectively adapted to new robotic environments without relying on expensive expert demonstrations or suffering from the high variance of standard reinforcement learning (RL) methods?

The authors introduce ROAD-VLA, a self-distillation framework designed to stabilize online adaptation. Standard RL methods like PPO often struggle with VLA models because they treat sparse, scalar rewards as weak supervision for high-dimensional action tokens. ROAD-VLA addresses this by constructing a 'proximal teacher' directly from the policy itself. It uses calibrated advantage estimates to perturb the policy's own action-token logits, effectively turning sparse rewards into dense, token-level supervision. This teacher is kept close to the current policy via a KL-regularized objective, ensuring stable updates without needing external expert models or additional demonstration data.

ROAD-VLA consistently outperforms standard PPO baselines across seven robotic manipulation environments. The authors demonstrate that their approach is robust to various distribution shifts, including visual changes (e.g., unseen backgrounds, sensor noise), compositional reasoning tasks (e.g., disambiguating distractors), and execution-level perturbations. The theoretical analysis further supports this, providing a policy-improvement lower bound that confirms the effectiveness of distilling toward an advantage-guided proximal teacher.

This work provides a practical solution for deploying general-purpose robot foundation models in real-world settings where environments differ from pretraining data. By enabling efficient, stable online adaptation using only self-generated feedback, ROAD-VLA reduces the reliance on costly, task-specific expert data, making robotic systems more versatile and reliable in dynamic, unpredictable environments.