ODYSSE: Episode-wise Policy Optimization for Personalized Agentic Reasoning

Modern agentic systems often struggle with ambiguous user requests that require multi-step reasoning to resolve. Existing models typically optimize for individual actions or final outcomes, which fragments the reasoning process and fails to capture the long-term dependencies necessary for personalized service delivery. This paper asks: how can we optimize agents to coherently decode ambiguous user requests into personalized actions across long-horizon interactions?

The authors introduce ODYSSE, a framework for personalized agentic reasoning that utilizes Episode-wise Group Relative Policy Optimization (ESPO). Unlike standard reinforcement learning methods that treat each action step in isolation, ESPO groups entire interaction trajectories into unified training batches. The framework employs a Chain-of-User-Thought (COUT) reward mechanism that evaluates two critical components: Intent Confidence (how well GUI actions support intent prediction) and Intent Contribution (how well the inferred intent leads to a successful personalized decision). By broadcasting these episode-level rewards back to individual action steps, the agent learns to prioritize actions that contribute to the overall goal of resolving user ambiguity.

Experimental results on long-horizon personalized GUI reasoning tasks demonstrate that ODYSSE consistently outperforms both specialist GUI agents and general-purpose Large Vision-Language Models (LVLMs). The study shows that by explicitly modeling the cross-step dependencies between GUI execution, intent prediction, and final decision-making, the agent becomes more effective at navigating open-ended solution spaces and delivering services that align with specific user preferences.