ResearchPod Summary
As personal AI agents increasingly persist across sessions, they are expected to accumulate experience—such as user preferences, task histories, and learned skills—to improve future performance. However, it remains unclear whether these agents actually leverage this retained experience to evolve, or if performance gains are merely artifacts of base model capabilities or prompt shortcuts. This paper introduces PAST-Bench to systematically evaluate and diagnose this "online self-evolution."
PAST-Bench evaluates agents across 26 scenarios and 204 episodes, focusing on four key capabilities: memory, procedural reuse, information gathering, and update. The benchmark uses a trajectory-level evaluation method where agents run through sequences of fresh-session tasks. To isolate the impact of persistence, the authors compare performance under "with-persistence" and "without-persistence" (matched control) conditions. They also track "mechanism evidence" to verify if the agent actually follows the intended save, retrieve, and update pathway, rather than relying on superficial cues.
Testing seven base models and four agent frameworks, the authors find that while agents do improve, the gains are inconsistent across different capabilities. Crucially, they discover that two agents can achieve the same headline performance gain while differing significantly in their mechanism evidence, meaning one agent may be "guessing" correctly while the other is actually retrieving the intended information. To address these failures, the authors developed Hermes+, which incorporates five targeted interventions across the agent loop (Plan, Render, Route, Gate, Close). Hermes+ demonstrates higher average gains and provides more reliable evidence of the intended persistence mechanisms, particularly in tasks requiring the replacement of outdated information.
Alex: Welcome to another episode of ResearchPod. Today, we're looking at a new way to test if AI agents are actually learning from their experiences.
Sam: We're discussing a paper on "PAST-Bench." The core puzzle is this: we see AI agents that seem to "remember" things, but we have no way to prove if they're truly learning or just guessing correctly because the task is easy.
Alex: So this paper is basically asking whether an AI's performance gain is actually caused by its memory, or whether it's just a coincidence?
Sam: Exactly. If an agent does better on a task today than it did yesterday, we need to know if that's because it saved and used a useful lesson, or if it's just luck—or simply a more capable base model to begin with.
Alex: That makes sense. It sounds like you're trying to separate the "smartness" the AI started with from the "learning" it does over time. How do you actually test that?
Sam: Think of it like testing a student's memory by letting them take an exam twice—once with their notes, and once without. The difference in their scores tells you exactly how much the notes actually helped. In the paper, they call this "matched ablation." You run the same AI on the same tasks, but sometimes you let it access its memory, and sometimes you wipe it clean first.
Alex: So the wipe is the control condition. Without it, you'd never know if the agent was genuinely using what it stored.
Sam: Right. They call that wipe a "context-clearing protocol." By removing the agent's temporary working memory between tasks, you force it to rely only on the persistent storage it was supposed to build up over time. If it performs better with that storage turned on, the improvement is real and traceable.
Alex: And what are these agents actually saving? Is it just simple notes, or something more structured?
Sam: It's a mix. An agent might save a specific procedural routine—a series of steps for filing an expense report, say—or a piece of information it gathered earlier that might be useful later. The paper tests four distinct dimensions: storing facts in memory, reusing step-by-step procedures, gathering new information, and updating old data when things change.
This work shifts the focus of agent evaluation from one-shot task success to longitudinal, trajectory-based improvement. By providing a diagnostic framework that separates outcome gains from the underlying mechanisms, PAST-Bench allows developers to identify exactly where an agent's self-evolution loop is breaking down, providing a clearer path toward building truly persistent and adaptive AI agents.
AI-generated third-party summary by ResearchPod. Not official content or an endorsement by the paper authors or affiliated organizations.
Alex: So it's not just "did it get the right answer?" It's "did it use the right tool to get there?"
Sam: That's the key insight. They look at what they call "mechanism evidence"—the agent's internal logs—to check whether it actually wrote something to storage, retrieved it, and applied it. Getting the answer right isn't enough. The agent has to show its work.
Alex: Right. Because if it gets the right answer but never actually looked at its notes, then it didn't really learn anything—it just got lucky.
Sam: Exactly. And that's a real problem they found. Some agents score well on paper but show almost no evidence of using their memory at all. That's why the paper also introduces a framework called Hermes+. Think of it as adding a structured checklist to the agent's routine—five specific steps, including planning ahead and reviewing what it already knows—so it's forced to actually engage with its own stored experience rather than skip past it.
Alex: So it's less about building smarter AI and more about building AI that actually uses what it already knows.
Sam: That's a good way to put it. The study found that improvement is real, but uneven. Some agents show gains in their scores, but when you look at the mechanism evidence, the support for those gains is surprisingly thin. They improved, but not necessarily because they learned.
Alex: Which means without this kind of diagnostic tool, you could easily mistake a lucky result for genuine progress.
Sam: Precisely. And that matters a lot when you're trying to build systems that reliably improve over time. If you can't tell which part of the system is responsible for a gain, you can't deliberately make it better.
Alex: So where does the benchmark fall short? What can't it measure yet?
Sam: The main limitation is that PAST-Bench currently uses controlled, isolated task families. It tests whether an agent can remember a routine for a specific type of job, but it doesn't yet capture what happens over longer timeframes or across very different domains. In the real world, an agent might need to take a lesson from filing an expense report and apply it to something completely different, like scheduling a meeting. The benchmark doesn't yet measure that kind of transfer.
Alex: It's like testing whether a student can remember a math formula, but not whether they can actually use it to solve a problem they've never seen before.
Sam: That's the gap. And the next challenge the paper points toward is what they call dynamic routing—the idea that future agents won't just dump everything into one big memory bank. They'll learn to sort their own experience: deciding what belongs in long-term storage, what becomes a reusable skill, and what can safely be discarded.
Alex: So instead of just remembering more, they'd learn how to organise what they remember.
Sam: Exactly. It's a shift from simple storage to something more like a self-managed filing system. And you can only build toward that if you have a way to measure whether the filing is actually working—which is precisely what PAST-Bench is designed to provide.
Alex: That's a useful place to land. The benchmark isn't a finished answer—it's a foundation for asking better questions. Thanks for walking us through it, Sam, and thanks to everyone listening to ResearchPod.