RENEW: Towards Learning World Models and Repairing Model Exploitation from Preferences

Overview

World models in reinforcement learning allow agents to plan by simulating future states. However, these models often suffer from 'model exploitation,' where an agent discovers and exploits inaccurate, physically impossible transitions (like walking through walls) to achieve artificially high rewards. While traditional methods address this by collecting more expert demonstrations or constraining the agent's behavior, these approaches are often expensive or limit the agent's ability to generalize.

This paper introduces Dynamics Learning from Human Feedback (DLHF), a framework that allows humans to supervise a world model's dynamics directly. Instead of providing expert demonstrations, humans simply compare two imagined trajectory segments and indicate which one is more physically realistic. The authors then introduce RENEW, an algorithm that uses epistemic uncertainty to focus these human preference queries on the specific transitions where the model is most likely to be wrong.

RENEW operates by treating the world model's transition dynamics as the target of optimization. It uses a Bradley-Terry preference loss, where the probability of preferring one trajectory over another is proportional to the log-likelihood of the transitions under the learned dynamics model.

To make this process efficient, RENEW employs an active learning loop:

1. Uncertainty Estimation: The model identifies regions of high epistemic uncertainty—areas where the model is unsure about the dynamics due to limited training data.
2. Targeted Querying: The system generates trajectory pairs specifically from these uncertain regions.
3. Human Feedback: Annotators label which trajectory is more physically plausible.
4. Finetuning: The model is updated to increase the likelihood of the preferred, physically realistic transitions.