FORCE: Efficient VLA Reinforcement Fine-Tuning via Value-Calibrated Warm-up and Self-Distillation

Vision-Language-Action (VLA) models are often limited by an 'imitation ceiling'—a performance plateau caused by the sub-optimal nature of the human demonstration data used for pre-training. While Reinforcement Learning (RL) can theoretically surpass this ceiling, applying it to real-world robotics is notoriously difficult due to sample inefficiency and 'initial unlearning,' where the policy collapses during the transition from offline training to online interaction. The authors introduce FORCE, a three-stage framework designed to stabilize this transition without requiring costly human-in-the-loop interventions.

FORCE operates through a structured pipeline:

1. Offline Pre-training: Uses Calibrated Q-Learning (Cal-QL) to establish a conservative, well-grounded critic that avoids overestimating out-of-distribution actions.
2. Distributional Warm-up: Bridges the gap between offline data and online behavior by collecting a small batch of on-policy rollouts and recalibrating the critic. This aligns the value function with the agent's actual visitation distribution, preventing the catastrophic performance drops typical of early training.
3. Value-Guided Policy Self-Distillation (VGPD): During online fine-tuning, the agent uses a dynamic filter to select only high-value actions for policy updates. By comparing action proposals against a state-dependent baseline, the model effectively ignores noisy exploration data and focuses on monotonic improvement.

By eliminating the need for human intervention and mitigating the instability of offline-to-online transfer, FORCE significantly lowers the barrier to deploying autonomous robotic agents. Experimental results across simulation and real-world tasks demonstrate that the framework achieves a 79% absolute improvement in success rates compared to baseline models, while also accelerating training speed by 32.5%. This makes it a highly efficient solution for refining VLA models in contact-rich, complex manipulation tasks.