Reward-Driven LLM Agent Workflows: Synthesizing POMDP Routing and Self-Correction for Autonomous Decision-Making

This paper addresses the limitations of current Large Language Model (LLM) agents, which often struggle with long-horizon planning and cascading errors due to a lack of formal decision-making objectives. The authors propose a new architecture, the Reward-Driven LLM Agent Workflow (RLAW), which treats the agent as a policy function within a Partially Observable Markov Decision Process (POMDP). By integrating reinforcement learning principles—specifically value function approximation and proximal policy optimization—the agent gains a formal mechanism to evaluate and correct its reasoning paths before committing to actions.

At the heart of the RLAW framework is a self-correcting critique module. Unlike previous methods that rely on heuristic-based self-reflection or simple prompt-based scoring, this module acts as an internal 'Critic' that evaluates candidate trajectories proposed by the 'Actor' (the LLM). This loop allows the agent to filter out suboptimal or hallucinated reasoning steps, effectively mitigating the error accumulation that typically plagues long-sequence tasks. The system also utilizes graph-based memory to maintain a structured representation of the environment, helping the agent track state changes over time.

To validate the framework, the authors conducted experiments on the ALFWorld embodied simulation and the WebShop navigation benchmark. The results show a 24.5% absolute improvement in task success rates and trajectory efficiency over the standard ReAct framework. Ablation studies confirm that the reward-driven critique module is the primary driver of this performance, significantly reducing hallucination rates. The authors argue that this framework provides a scalable, reliable reference for deploying autonomous agents in complex, multi-step systems.