Conservative Query and Adaptive Regularization for Offline RL under Uncertainty Estimation

Offline reinforcement learning (RL) is often limited by the distribution shift between the static dataset and the learned policy. While action preference queries can help improve policies without environment interaction, existing methods suffer from query shift—where queried actions fall outside the dataset distribution—and static constraints that fail to adapt as the policy improves. This paper investigates how to better select and utilize these preferences to enhance policy performance in value regularization frameworks.

The authors propose a lightweight framework that integrates uncertainty estimation into the preference query process. First, they use a Morse neural network to quantify the epistemic uncertainty of state-action pairs relative to the offline dataset. This uncertainty score serves two purposes:

1. Conservative Querying: Instead of selecting actions based solely on distance, the method filters candidate actions using a dataset-aware threshold, ensuring that only actions near the dataset distribution are queried to prevent unstable Bellman updates.
2. Adaptive Regularization: The framework replaces fixed regularization coefficients with an uncertainty-aware, non-linear scaling function. This dynamically adjusts the penalty strength based on the action's uncertainty, applying stronger constraints to out-of-distribution actions while allowing more flexibility for reliable, in-distribution actions.

The authors integrated their framework into the Conservative Q-Learning (CQL) algorithm and evaluated it on the D4RL benchmark. The results indicate that the proposed method consistently achieves superior or competitive performance compared to baseline approaches. By mitigating query shift and balancing pessimistic constraints with optimistic Bellman updates, the framework provides a more stable and effective way to incorporate expert preferences into offline RL.