INVERSE RL HELPS ALIGN AI BY IMITATING HUMANS

Traditional language model alignment relies on supervised fine-tuning (SFT) or reinforcement learning from human feedback (RLHF). While effective, these methods often treat expert demonstrations as static targets for imitation rather than as sources of an underlying, optimizable objective. This paper asks whether one can recover an explicit, inspectable reward function directly from demonstrations—without needing preference annotations—to enable more flexible alignment, such as contextual adaptation for different audiences.

To address this, the authors propose Projected Alignment Reward Estimated from Demonstrations (PARED). The method operates by mapping both expert demonstrations and policy-generated responses into a compact, practitioner-defined feature space (e.g., helpfulness, harmlessness, and topic distributions). A lightweight logistic discriminator is then trained to distinguish between expert and policy trajectories within this space. The resulting "expert-likeness" score serves as a scalar reward function. This reward is used in two ways:

1. Inference-time reranking: Selecting the best response from a pool of candidates without modifying model weights.
2. On-policy RL: Using the reward to drive policy updates via reinforcement learning, allowing the model to improve beyond the initial imitation of demonstrations.

PARED demonstrates that demonstrations contain a rich optimization signal that is often underutilized by standard SFT. The authors show that the recovered reward improves base models significantly, even when used as a post-hoc optimization step after SFT. Furthermore, the method enables contextual alignment: a single policy can be conditioned on different audience labels (e.g., adult vs. child), with PARED learning distinct rewards for each to tailor responses accordingly. The approach achieves high win rates in automated side-by-side evaluations against baseline models, confirming that the inferred reward successfully captures the desired behavioral nuances.

Why It Matters

AI-generated third-party summary by ResearchPod. Not official content or an endorsement by the paper authors or affiliated organizations.