DriveVLA-M0: Failure-Aware Memory Augmentation for Autonomous Driving

Autonomous driving models often struggle with long-tail scenarios and repeat the same mistakes when encountering situations similar to past failures. Existing Vision-Language-Action (VLA) models lack an explicit mechanism to associate current driving scenes with historical errors. This paper asks: can we improve driving safety and robustness by explicitly storing past failure cases in a latent memory and using them to adapt the model's planning in real-time?

The authors introduce DriveVLA-M0, a framework that integrates a failure-aware latent memory with test-time adaptation. The system operates in two stages:

1. Memory Generation (Offline): The model identifies scenarios where the base VLA model underperforms using oracle simulation metrics. It stores these failure cases—including scene representations, agent interactions, and expert trajectories—into a latent memory pool.
2. Inference with Test-Time Training (Online): When the vehicle encounters a new scene, a dedicated 'Retrieve Model' identifies structurally similar failure cases from the memory. These retrieved cases are then used to fine-tune the model's action decoder via a lightweight, decoupled LoRA (Low-Rank Adaptation) mechanism. This allows the model to adjust its planning strategy specifically for the current context without needing to retrain the entire backbone.

The Retrieve Model is designed to decouple static road structure from dynamic agent interactions, ensuring that the retrieved memory cases are physically and structurally relevant to the current driving scene.

DriveVLA-M0 achieves state-of-the-art performance on the NAVSIMv1 and NAVSIMv2 benchmarks, reaching 94.1 PDMS (Planning-based Driving Metric Score) on Navtest and 47.0 EPDMS on Navhard. The system demonstrates that it can effectively scale with larger memory pools, providing performance gains without requiring offline retraining. The test-time training process is efficient, adding only 26.44 ms of latency, making it suitable for real-time autonomous driving applications.