RECALL: Recovery Experience Collection for Active Lifelong Learning in Vision-Language-Action Models

Research Approach

This paper investigates how to improve the fine-tuning of Vision-Language-Action (VLA) models, which are typically adapted to new tasks using inefficient passive imitation learning. The authors propose an active, continual learning paradigm that uses uncertainty quantification—specifically the INSIGHT method—to identify states where the robot is likely to fail. By collecting expert demonstrations only at these high-uncertainty states, the researchers aim to maximize the information gain of each new demonstration. They evaluate this approach against passive baselines and test various continual learning strategies, including data replay and Elastic Weight Consolidation (EWC), to prevent the model from forgetting its original capabilities.

Key Findings

• Efficiency of Active Learning: Demonstrations collected from high-uncertainty states are significantly more effective than those collected from task initial states, even when the total volume of data is controlled.
• Online vs. Offline Collection: Collecting data from the first high-uncertainty state in a rollout (online) is as effective as collecting from every high-uncertainty state (offline), making the active learning process more efficient.
• The Forgetting Problem: Fine-tuning a VLA model exclusively on new recovery data leads to severe catastrophic forgetting. While regularization techniques like EWC or using lower learning rates can mitigate this, they often limit the model's ability to learn from the new data.
• Replay as the Best Strategy: Replay-based data mixing—where new recovery data is combined with a buffer of original training data—consistently provides the best balance between adapting to new tasks and retaining prior performance.

As VLA models are increasingly deployed in robotics, the ability to adapt them to new environments without full retraining is critical. This study demonstrates that active learning can reduce the burden on human demonstrators by focusing their efforts on the most informative states. However, it also highlights that for large-scale robot policies, simple fine-tuning is insufficient; robust continual learning mechanisms are essential to ensure that new knowledge does not come at the expense of previously mastered skills.