TOUR: A Trajectory-Level Unlearning Benchmark for Offline Reinforcement Learning

Offline reinforcement learning (RL) policies are trained on fixed datasets of sensitive behavioral trajectories. When specific trajectories must be removed—due to privacy regulations or data rights—it is unclear how to verify that the model has truly 'forgotten' the data. The authors investigate how to evaluate trajectory-level unlearning without confusing genuine privacy improvements with policy collapse (where the model loses all useful behavior) or residual memorization that remains hidden from simple metrics.

The authors introduce TOUR (Trajectory-level memOrization and Unlearning in offline RL), a comprehensive evaluation framework. TOUR partitions datasets into forget, retain, and matched non-member sets. It uses a multi-attack auditing protocol that combines likelihood-based membership inference, retraining references, and retained-performance anchors. By testing various policy architectures (Decision Transformers, LSTMs, and MLPs) across D4RL locomotion tasks, the authors assess whether deletion baselines—such as gradient ascent with refitting (GA+Refit) or TrajDeleter—actually remove trajectory influence while maintaining task utility.

The study reveals that privacy-utility trade-offs are highly environment-dependent and that single-score metrics (like a simple likelihood-based AUC) are insufficient for auditing. For instance, some methods appear to achieve privacy by significantly degrading the model's performance, effectively causing a 'utility collapse' rather than selective unlearning. Furthermore, the authors find that conclusions about unlearning effectiveness are unstable under single-score auditing; they depend heavily on the choice of matched non-member controls, the specific attack family used, and the calibration against retraining references. The results highlight that a robust unlearning audit must be a profile of evidence rather than a single number.