Shuhan Xue, Zixin Ding, Yichen Shen, Yinjie Wang, Zhenfei Yin, Yingcheng Wu, Yuxin Chen, Mengdi Wang, Ling Yang
5 min
As personal AI agents increasingly persist across sessions, they are expected to accumulate experience—such as user preferences, task histories, and learned skills—to improve future performance. However, it remains unclear whether these agents actually leverage this retained experience to evolve, or if performance gains are merely artifacts of base model capabilities or prompt shortcuts. This paper introduces PAST-Bench to systematically evaluate and diagnose this "online self-evolution."
PAST-Bench evaluates agents across 26 scenarios and 204 episodes, focusing on four key capabilities: memory, procedural reuse, information gathering, and update. The benchmark uses a trajectory-level evaluation method where agents run through sequences of fresh-session tasks. To isolate the impact of persistence, the authors compare performance under "with-persistence" and "without-persistence" (matched control) conditions. They also track "mechanism evidence" to verify if the agent actually follows the intended save, retrieve, and update pathway, rather than relying on superficial cues.
Testing seven base models and four agent frameworks, the authors find that while agents do improve, the gains are inconsistent across different capabilities. Crucially, they discover that two agents can achieve the same headline performance gain while differing significantly in their mechanism evidence, meaning one agent may be "guessing" correctly while the other is actually retrieving the intended information. To address these failures, the authors developed Hermes+, which incorporates five targeted interventions across the agent loop (Plan, Render, Route, Gate, Close). Hermes+ demonstrates higher average gains and provides more reliable evidence of the intended persistence mechanisms, particularly in tasks requiring the replacement of outdated information.
This work shifts the focus of agent evaluation from one-shot task success to longitudinal, trajectory-based improvement. By providing a diagnostic framework that separates outcome gains from the underlying mechanisms, PAST-Bench allows developers to identify exactly where an agent's self-evolution loop is breaking down, providing a clearer path toward building truly persistent and adaptive AI agents.
Recursive self-improvement requires agents to turn accumulated experience into better future behavior. Personal AI agents offer a concrete setting for studying this capability because they retain preferences, task histories, tool routines, and learned skills across sessions. Yet whether retained experience actually improves them over time has not been systematically tested. We introduce PAST-Bench, a benchmark designed to isolate this question. Each agent runs through ordered sequences of fresh-session tasks under matched conditions that turn retained experience on and off. It spans 26 scenarios and 204 episodes across memory, procedural reuse, information gathering, and update. We report both later-task gains and whether those gains follow the intended save, retrieve, and update pathway. Across seven base models and four agent frameworks, improvement is real but uneven across capabilities. Agents with the same headline gain can differ markedly in whether that gain is supported by evidence of the intended pathway. Guided by these findings, we develop Hermes+, which extends Hermes with five targeted interventions across stages of the agent loop. Hermes+ raises the average gain from retained experience and provides clearer pathway evidence, with its strongest improvement on tasks requiring outdated state to be replaced, although the effect remains capability- and model-dependent. Together, PAST-Bench and Hermes+ provide an evaluation and diagnostic foundation for studying how persistent agents can progress from retaining experience to systematically improving through it. Code: https://github.com/Gen-Verse/PAST-Bench
Sam: Exactly. And that's a real problem they found. Some agents score well on paper but show almost no evidence of using their memory at all. That's why the paper also introduces a framework called Hermes+. Think of it as adding a structured checklist to the agent's routine—five specific steps, including planning ahead and reviewing what it already knows—so it's forced to actually engage with its own stored experience rather than skip past it.
Alex: So it's less about building smarter AI and more about building AI that actually uses what it already knows.
Sam: That's a good way to put it. The study found that improvement is real, but uneven. Some agents show gains in their scores, but when you look at the mechanism evidence, the support for those gains is surprisingly thin. They improved, but not necessarily because they learned.
Alex: Which means without this kind of diagnostic tool, you could easily mistake a lucky result for genuine progress.
Sam: Precisely. And that matters a lot when you're trying to build systems that reliably improve over time. If you can't tell which part of the system is responsible for a gain, you can't deliberately make it better.
Alex: So where does the benchmark fall short? What can't it measure yet?
Sam: The main limitation is that PAST-Bench currently uses controlled, isolated task families. It tests whether an agent can remember a routine for a specific type of job, but it doesn't yet capture what happens over longer timeframes or across very different domains. In the real world, an agent might need to take a lesson from filing an expense report and apply it to something completely different, like scheduling a meeting. The benchmark doesn't yet measure that kind of transfer.
Alex: It's like testing whether a student can remember a math formula, but not whether they can actually use it to solve a problem they've never seen before.
Sam: That's the gap. And the next challenge the paper points toward is what they call dynamic routing—the idea that future agents won't just dump everything into one big memory bank. They'll learn to sort their own experience: deciding what belongs in long-term storage, what becomes a reusable skill, and what can safely be discarded.
Alex: So instead of just remembering more, they'd learn how to organise what they remember.
Sam: Exactly. It's a shift from simple storage to something more like a self-managed filing system. And you can only build toward that if you have a way to measure whether the filing is actually working—which is precisely what PAST-Bench is designed to provide.
Alex: That's a useful place to land. The benchmark isn't a finished answer—it's a foundation for asking better questions. Thanks for walking us through it, Sam, and thanks to everyone listening to ResearchPod.