LLM agents are expected to act over multiple turns, using search, browsing interfaces, and terminal tools to complete user goals. Yet not every goal is well specified or achievable in the available environment. In such cases, a reliable agent should recognize that further interaction is unlikely to help and abstain from additional tool calls. We define Agentic Abstention, the problem of deciding when an agent should stop acting under uncertainty. Unlike standard LLM abstention, which is usually evaluated as a single-turn answer-or-abstain decision, agentic abstention is a sequential decision problem: an agent can answer, abstain, or gather more information at each turn, and the need to abstain may only become clear after interacting with the environment. We study this problem across web shopping, terminal environments, and question answering, evaluating 13 LLM-as-agent systems and 2 agent scaffolds on more than 28,000 tasks. Our results show that the main challenge is not only whether agents can abstain, but also when they abstain. Some agents never abstain when they should, while others do so only after many unnecessary interactions. This gap is especially large on tasks where the instruction appears feasible until the environment reveals otherwise (e.g., no valid result matches the instruction). We further find that model scale, reasoning, and agent scaffolding affect abstention in different ways, where larger or more capable models sometimes perform worse at timely abstention. Finally, we introduce CONVOLVE, a context engineering method for improving agentic abstention that distills full interaction trajectories into reusable stopping rules. On WebShop, CONVOLVE substantially improves timely abstention without updating model parameters, raising Llama-3.3-70B's timely recall rate from 26.7 to 57.4. Our dataset and code are available at https://lhannnn.github.io/agentic-abstention
Alex: Welcome to another episode of ResearchPod. Today, we're discussing a paper on "Agentic Abstention"—the challenge of teaching AI agents when to stop trying to solve an impossible task.
Sam: So this is basically asking why AI agents act like they have a "sunk cost fallacy"—burning through computing power on tasks that simply have no solution?
Alex: Exactly. Think of it this way: when you ask a person to find something that doesn't exist, they usually figure it out pretty quickly and give up. AI agents, on the other hand, are very good at following instructions—but they often struggle to recognize when a request is fundamentally impossible. Instead of stopping, they just keep trying, wasting time and resources on actions that can never succeed.
Sam: And the failure isn't sudden. They fail slowly, continuing to poke around in the environment long after any reasonable observer would have called it quits.
Alex: Right. The researchers call this the "Agentic Abstention" problem. At every step, an agent faces a choice: take another action, give an answer, or stop entirely. The tricky part is that the right moment to stop isn't always obvious from the start—it usually only becomes clear after the agent has already gathered some information from the world around it.
Sam: That makes sense. It's like a shopper looking for a specific pink pillow in a store that simply doesn't carry that product. If the agent doesn't recognize that the store is empty of what it needs, it just keeps searching every aisle, over and over, for ten turns.
Alex: That's the perfect illustration. The paper introduces a specific term for the skill the agents are missing: "timely recall." That means stopping at the earliest possible moment when the agent has gathered enough evidence to know the task cannot be completed—not ten turns later, but right when the writing is on the wall.
Sam: So if an agent is trained by watching humans do tasks, it might not have ever seen a human model the logic of when to quit. It just learned to keep going. How does the research actually fix that—without retraining the entire model from scratch?
Alex: That's the key question, and the answer is a method the researchers call CONVOLVE. Rather than rebuilding the AI from the ground up, CONVOLVE works by changing what the agent is told before it starts a task. Think of it like a "stop-loss" order in stock trading—a pre-set rule that says "if things go this badly, stop automatically" so you don't keep throwing money at a losing position.
Sam: So it's injecting "stopping rules" into the agent's instructions. But where do those rules come from? Does a researcher have to write them manually for every possible type of failure?
Alex: No, that would be completely unworkable. Instead, the system uses what the paper calls a "Reflection Agent"—a separate, specialized component whose only job is to study past failures. It goes back through the full record of previous interactions—every step the agent took, every dead end it hit—and asks: at what point should it have stopped? It then distills those lessons into a concise, reusable set of guidelines that get added to the agent's instructions going forward. Like a coach reviewing game footage and writing up a new playbook before the next match.
Sam: So the agent is essentially learning from its own mistakes, just not in the way we usually mean. It's not retraining its internal knowledge—it's building up a set of hard-won "do not proceed" rules that get passed forward.
Alex: Exactly. The internal workings of the model stay completely unchanged. What changes is the context the model receives—the framing and the rules it's given before it starts. That's why the researchers describe this as "context engineering" rather than model training. It's a meaningful distinction, because it means the approach can be applied to existing AI systems without any of the expense or complexity of retraining.
Sam: And does it actually work?
Alex: The paper reports that for one of the models they tested—a large language model called Llama-3.3-70B—this approach more than doubled the rate of timely recall on web shopping tasks. That's a substantial shift. The research suggests the bottleneck isn't the AI's raw reasoning ability, but how its decision-making process is structured in the first place.
Sam: It's a meaningful reframe. The question isn't just "can this agent do the task?"—it's "does this agent know when the task is undoable?" Teaching that second skill, without touching the model itself, seems like a practical and underexplored direction.
Alex: It really does. And it points to something broader: a lot of AI reliability problems may not require more powerful models to fix. Sometimes, the answer is better-structured guidance about when not to act. Thanks for listening to ResearchPod.