Han Luo, Bingbing Wen, Lucy Lu Wang
4 min
Abstract
LLM agents are expected to act over multiple turns, using search, browsing interfaces, and terminal tools to complete user goals. Yet not every goal is well specified or achievable in the available environment. In such cases, a reliable agent should recognize that further interaction is unlikely to help and abstain from additional tool calls. We define Agentic Abstention, the problem of deciding when an agent should stop acting under uncertainty. Unlike standard LLM abstention, which is usually evaluated as a single-turn answer-or-abstain decision, agentic abstention is a sequential decision problem: an agent can answer, abstain, or gather more information at each turn, and the need to abstain may only become clear after interacting with the environment. We study this problem across web shopping, terminal environments, and question answering, evaluating 13 LLM-as-agent systems and 2 agent scaffolds on more than 28,000 tasks. Our results show that the main challenge is not only whether agents can abstain, but also when they abstain. Some agents never abstain when they should, while others do so only after many unnecessary interactions. This gap is especially large on tasks where the instruction appears feasible until the environment reveals otherwise (e.g., no valid result matches the instruction). We further find that model scale, reasoning, and agent scaffolding affect abstention in different ways, where larger or more capable models sometimes perform worse at timely abstention. Finally, we introduce CONVOLVE, a context engineering method for improving agentic abstention that distills full interaction trajectories into reusable stopping rules. On WebShop, CONVOLVE substantially improves timely abstention without updating model parameters, raising Llama-3.3-70B's timely recall rate from 26.7 to 57.4. Our dataset and code are available at https://lhannnn.github.io/agentic-abstention
Alex: No, that would be completely unworkable. Instead, the system uses what the paper calls a "Reflection Agent"—a separate, specialized component whose only job is to study past failures. It goes back through the full record of previous interactions—every step the agent took, every dead end it hit—and asks: at what point should it have stopped? It then distills those lessons into a concise, reusable set of guidelines that get added to the agent's instructions going forward. Like a coach reviewing game footage and writing up a new playbook before the next match.
Sam: So the agent is essentially learning from its own mistakes, just not in the way we usually mean. It's not retraining its internal knowledge—it's building up a set of hard-won "do not proceed" rules that get passed forward.
Alex: Exactly. The internal workings of the model stay completely unchanged. What changes is the context the model receives—the framing and the rules it's given before it starts. That's why the researchers describe this as "context engineering" rather than model training. It's a meaningful distinction, because it means the approach can be applied to existing AI systems without any of the expense or complexity of retraining.
Sam: And does it actually work?
Alex: The paper reports that for one of the models they tested—a large language model called Llama-3.3-70B—this approach more than doubled the rate of timely recall on web shopping tasks. That's a substantial shift. The research suggests the bottleneck isn't the AI's raw reasoning ability, but how its decision-making process is structured in the first place.
Sam: It's a meaningful reframe. The question isn't just "can this agent do the task?"—it's "does this agent know when the task is undoable?" Teaching that second skill, without touching the model itself, seems like a practical and underexplored direction.
Alex: It really does. And it points to something broader: a lot of AI reliability problems may not require more powerful models to fix. Sometimes, the answer is better-structured guidance about when not to act. Thanks for listening to ResearchPod.