When Preferences Fail to Become Incentives: A Utility–Behavior Gap in Large Language Models

Recent research has shown that LLMs exhibit coherent, model-specific utility rankings when forced to choose between outcomes. This has raised concerns that these rankings might represent latent, misaligned goals that could influence model behavior in real-world settings. This paper investigates whether these elicited preferences actually function as incentives. The authors designed a behavioral transfer test where they attached success-contingent outcomes—ranging from high-utility to low-utility based on the model's own rankings—to various writing tasks, including essays, grant proposals, incident postmortems, and translations. They then used blind LLM judge panels to determine if the high-utility outcomes led to higher-quality outputs.

To ensure that a null result was not simply due to insensitive judges or inert tasks, the researchers included several control conditions. They tested whether models could modulate their output quality in response to direct effort exhortations (e.g., instructions to perform well), role-based cues (e.g., being told they are a 'world-class' expert), and harmful-outcome cues. This comparative design allowed the researchers to distinguish between a failure of utility-to-behavior transfer and a general inability of the model to respond to contextual incentives.

Across seven instruction-tuned models and four task families, the researchers found no reliable evidence that elicited utility rankings influence generation quality. High-utility outcomes failed to produce better artifacts than low-utility outcomes. In contrast, the models consistently responded to external cues: direct effort instructions and role-playing prompts significantly improved output quality, while harmful outcomes led to a decrease in quality (sandbagging). The authors conclude that while LLMs can be made to display coherent preferences in isolated choice paradigms, these preferences do not function as internal motivations that guide behavior in broader, open-ended generation tasks.