Learning User Simulators with Turing Rewards

How can we train large language models (LLMs) to accurately simulate specific human users in interactive settings? Existing approaches typically focus on response matching—training models to replicate a single ground truth response via log-probability maximization or similarity rewards. The authors argue that this is suboptimal because a single context can have many plausible human responses. Instead, they propose a framework that optimizes for indistinguishability, mirroring the classic Turing Test.

Approach: Turing-RL

The authors introduce Turing-RL, a reinforcement learning (RL) framework for training user simulators. The process begins with supervised fine-tuning (SFT) on user data augmented with chain-of-thought reasoning. Following this, the model is trained using Group Relative Policy Optimization (GRPO) with a discriminative Turing reward. In this setup, an LLM judge compares a model-generated response against a real human response, scoring which is more likely to have been written by the human. This reward signal encourages the simulator to produce responses that are human-like in style, motive, and context, rather than just matching content.

Findings

Across two distinct domains—multi-turn chat and Reddit forum discussions—Turing-RL consistently outperformed baselines (including similarity-based and log-probability-based rewards) on both LLM-as-a-judge and human evaluation metrics. Notably, the authors found that optimizing for indistinguishability did not sacrifice content alignment; the models remained as similar to ground truth as models explicitly trained for similarity. Furthermore, qualitative analysis suggested that larger, general-purpose models often fail to simulate specific users effectively because they default to overly verbose, assistant-like personas, whereas Turing-RL models successfully adopt more naturalistic, user-specific behaviors.

This research provides a scalable way to build user simulators, which are critical for training AI assistants, testing personalization systems, and creating social world models. By shifting the objective from "matching the ground truth" to "being indistinguishable from the user," the authors demonstrate a more effective path toward creating AI agents that can accurately reflect individual human preferences and communication styles.