Themis: An explainable AI-enabled framework for Reinforcement Learning with Human Feedback

Overview

Reinforcement Learning (RL) agents often struggle with reward hacking, where they exploit poorly defined reward functions to maximize returns while failing to perform the intended task. While Reinforcement Learning from Human Feedback (RLHF) is a powerful tool for aligning agents with human values, existing frameworks are often proprietary, narrow in scope, or lack the infrastructure to handle large-scale human feedback collection. Themis addresses these gaps by providing a modular, XAI-enabled framework that combines RL training with a scalable, cloud-based crowdsourcing platform.

Themis is designed as a two-part system: an RLHF training engine and a human-facing interface. The training engine supports over 200 environments (via the Gym API) and utilizes the Soft Actor-Critic (SAC) algorithm to learn from both environment rewards and human preferences. To improve transparency, Themis integrates explainable AI (XAI) methods—such as Integrated Gradients, Kernel SHAP, and TracInCP—allowing human teachers to better understand agent behaviors before providing feedback. The crowdsourcing platform manages the logistics of these experiments, enabling researchers to upload video clips of agent trajectories, collect human preferences, and manage multiple concurrent experiments through a unified web interface.

Findings and Impact

The authors demonstrate that Themis can effectively distill human preferences into reward models that approximate or outperform ground-truth environment rewards. Furthermore, the platform is built for high scalability, with tests confirming it can support up to 1,000 users in back-to-back experiments on modest hardware. By providing an open-source, highly configurable tool, Themis lowers the barrier to entry for researchers looking to incorporate human-in-the-loop alignment and explainability into their RL pipelines.