RecoverFly: A Failure-Aware Reinforcement Learning Post-Training Framework for Aerial Vision-Language Navigation

Unmanned Aerial Vehicle (UAV) vision-language navigation (VLN) models often struggle with closed-loop execution. While end-to-end vision-language-action (VLA) policies avoid the complexity of modular systems, they are typically trained via behavior cloning, which fails to provide corrective feedback when the agent encounters errors like collisions or navigation drift. This paper asks how reinforcement learning (RL) can be effectively applied to these autoregressive policies to improve navigation success and generalization without degrading previously learned capabilities.

The authors propose RecoverFly, a post-training framework that adapts RL for end-to-end UAV-VLA policies. The method introduces three key innovations:

1. Token-Level RL: Instead of sequence-level optimization, it applies clipped surrogate objectives to individual action tokens, allowing for stable updates in the autoregressive action space.
2. Dynamic Failure Replay: The system maintains a pool of unresolved navigation failures, forcing the agent to repeatedly practice and learn from its mistakes rather than discarding them.
3. Two-Stage Curriculum & Regularization: A curriculum balances training across long-tailed scene distributions, while reference-policy KL regularization ensures the model does not deviate too far from its initial, stable behavior.

RecoverFly demonstrates significant improvements over the baseline AerialVLA model on the TravelUAV benchmark. Across seen, unseen-map, and unseen-object splits, the framework consistently achieves higher success rates. Specifically, the authors report success rate improvements of 3.12 to 8.37 percentage points compared to the initialization, even when constrained to a rollout budget of only 30% of the training-set size. These results confirm that the failure-aware replay and curriculum-based training effectively enhance the robustness and generalization of aerial navigation agents.