Self-Adaptive Anomaly Detection with Reinforcement Learning and Human Feedback in Connected Vehicles

Connected vehicles operate as complex cyber-physical systems (CPS) where software updates, configuration changes, and shifting workloads create non-stationary data environments. Traditional, static anomaly detection methods struggle in these settings because they cannot adapt to the evolving definition of normal behavior, leading to silent performance degradation. This paper addresses this by introducing a supervisory loop that combines automated model selection, statistical drift detection, and human-in-the-loop retraining.

The core of the framework is a factorized deep Q-network (F-DQN) with self-attention. Unlike standard RL agents that would struggle with the high-dimensional action space of selecting detectors for multiple services, this architecture uses a shared encoder and independent per-service heads. The self-attention mechanism allows the agent to account for inter-service dependencies, such as cascading failures across a microservice topology.

To manage non-stationary data, the system employs an ensemble of three statistical drift detectors: the Page-Hinkley test (for mean shifts), the Kolmogorov-Smirnov test (for marginal distribution changes), and a Mahalanobis-distance outlier-rate test (for structural dependency shifts). By requiring all three detectors to concur before raising an alarm, the system prioritizes high precision, ensuring that the operator is only alerted when a genuine distributional shift occurs.

When the system detects drift or flags an anomaly, it provides an interface for human experts to validate the findings. To prevent catastrophic forgetting—where the model loses its ability to handle previous data distributions after learning from new ones—the framework uses a prioritized replay buffer. This buffer maintains a 60/40 mix of new and historical transitions, ensuring the agent retains its learned response to prior distributions while adapting to new operational conditions.

Evaluation on an automated valet parking testbed demonstrated that the attention-augmented agent significantly outperformed uniform detector application, achieving an F1 score of 0.69. Following a software update that induced concept drift, the system's performance dropped but successfully recovered to 0.65 after operator-triggered retraining, all while maintaining its performance on the prior distribution. This approach is significant because it provides a scalable, maintainable path for deploying diagnostic AI in safety-critical, evolving environments where manual oversight is limited but necessary.