Recently, LLM-based agents have become increasingly popular across many applications, including complex sequential decision-making problems. However, they inherit the tendency of LLMs to hallucinate, leading to incorrect decisions. In sequential settings, even a single mistake can irreversibly degrade the trajectory, making hallucinations an even bigger problem. Although larger LLMs hallucinate less, they incur a significantly higher per-token cost. In this paper, we address this tradeoff by proposing ReDAct (Reason-Defer-Act). In ReDAct, an agent is equipped with two LLMs: a small, cheap model used by default, and a large, more reliable but expensive model. When the predictive uncertainty of the small model exceeds a calibrated threshold, the decision is deferred to the large model. We evaluate our approach in text-based embodied environments such as ALFWorld and MiniGrid and show that deferring only about 15% of decisions to the large model can match the quality of using it exclusively, while significantly reducing inference costs.
Alex: Welcome to another episode of ResearchPod.
Sam: Today we're looking at a paper called ReDAct: Uncertainty-Aware Deferral for LLM Agents. It addresses a problem with AI agents powered by large language models: they sometimes hallucinate, making up wrong facts, and in step-by-step tasks, one error can derail the whole sequence.
Alex: So these are AIs that handle chains of decisions, like navigating a virtual house? The challenge is knowing when to switch from a cheap small AI to a pricier large one?
Sam: Yes. Picture a warehouse robot picking items: a wrong grab early on ruins the delivery. Large AIs make fewer mistakes but cost more per step, so the paper uses a small AI for most decisions and switches to the large one only when unsure.
Alex: Like a driver using a basic map but pulling up a premium one for tricky spots. How does the small AI spot its own uncertainty?
Sam: It checks simple scores from its predictions, like how spread out its guesses are for the next word or step. If doubt passes a limit, it defers. In ALFWorld tests—virtual household chores where errors compound—this matches full large-AI performance by deferring just 15% of steps, at much lower cost.
Alex: Okay, so it suggests an action, scores its confidence, then decides. But where in the process do they check—during planning or at the action itself?
Sam: They tested both. Internal planning is vague and wordy, so doubt scores there mix good and bad ideas. Action-picking is concrete, like "pick up cup," so scores like word-spread or prediction surprise clearly flag shaky choices. Labeled ALFWorld steps confirmed this.
Alex: So action-stage checks work better, like judging a basketball shot by the release, not team huddle talk.
Sam: Right. They tuned cutoffs on validation data to average five large-AI calls per task. This beats random deferrals.
Alex: How does that perform on chores?
Sam: Perplexity-guided deferrals matched large-AI success rates, calling it for 15% of steps. All uncertainty methods beat the small AI alone; random helped less. Slight call-count shifts occur in real runs as fixes alter later doubts, but it works in practice.
Alex: Like a filter: cheap routine, expert fixes where needed.
Sam: Trade-off plots show these methods balance wins and calls best, even over tuned random.
Alex: What about real costs?
Sam: For ALFWorld, full large-AI runs cost over $500; small AI alone, $9 to $45. Guided deferrals hit $15 to $30 while matching success—a clear saving.
Alex: Cheaper than full large, pricier than small alone. Why over random?
Sam: Guided calls cluster late in failing runs, with longer prompts from error history, raising token counts slightly. Still, low per-token prices keep differences small.
Alex: Like a student asking for help deepest in a mess, with a longer question.
Sam: Yes. Overall, action uncertainty like Perplexity separates good from bad moves effectively.
Alex: Any catches, like needing model internals?
Sam: It needs token probabilities from the small AI, available in most APIs but not all. Tested on models over 70 billion parameters, as smaller ones need tuning. A meaningful step for efficient agents.
Alex: How did they prove these scores flag errors well?
Sam: They used Prediction-Rejection Ratio, or PRR—like sorting test papers by your weakest grading gut first and checking if the rest improve steadily. Over 2,400 labeled ALFWorld steps showed info-based scores like word-spread and surprise about twice as sharp as others.
Alex: Like a referee ejecting worst players first for a stronger game.
Sam: Rejection plots confirmed: accuracy rose steadily as they skipped more doubtful steps, closer to ideal than rivals.
Alex: Simple scores prove reliable for smart deferrals.
Sam: The paper positions this as practical for tasks where slips cascade, blending cheap and expert models.
Alex: It turns pricey AIs into on-call consultants for critical fixes.
Sam: Pareto charts confirm: guided deferrals lead on cost-success trade-offs, matching full large-AI results at 15% calls across chores and mazes.
Alex: Clear efficiency. Limitations?
Sam: Needs token probabilities, not always in APIs. Tuning overlooks how fixes shift later uncertainties slightly.
Alex: Fair hurdles, but a template for hybrids.
Sam: It suggests scaling to sequential tasks, cutting costs significantly.
Alex: Thanks for breaking down ReDAct, Sam. That's it for this episode of ResearchPod.