Learning Safe Agent Behaviour from Human Preferences and Justifications via World Models

In safety-critical environments where environment dynamics are unknown and no reward function exists, how can we train and deploy agents safely without requiring constant, burdensome human oversight or risking real-world damage during the learning process?

The authors introduce DROPJ (Dream-World Reward Learning from One-Shot Human Preferences and Justifications). The process follows four steps:

1. World Model Learning: A dynamics model is trained on an offline dataset of reward-free real-world trajectories to create a 'dream' simulator.
2. One-Shot Data Collection: A human interacts with this simulator to generate diverse, informative trajectories, allowing for safe exploration of failure states.
3. Reward Modeling: The system samples trajectory segments from the simulator and asks the human for preferences accompanied by specific safety justifications. These are used to train a reward model that captures both performance goals and safety constraints.
4. Deployment: The agent is deployed in the real world using Model Predictive Control (MPC), utilizing the learned reward model to plan actions that maximize reward while avoiding unsafe states.

The study demonstrates that generating informative simulated trajectories through a human-in-the-loop 'one-shot' process significantly reduces computational costs compared to iterative, automated query-generation methods. Furthermore, the inclusion of safety justifications allows the reward model to better align with user-prescribed safety requirements, leading to safer deployment performance. The authors show that using preferences over trajectory segments is more effective for performance than other forms of feedback in this simulated training context.

This research provides a scalable way to align autonomous agents with human values in complex, safety-critical domains like robotics or autonomous driving. By moving the training process into a learned simulator and using human justifications to define safety, the framework minimizes the risk of real-world accidents and reduces the time-intensive burden on human supervisors.