Tianyi Guan, Yiding Wang, Haotong Yang, Siyuan Cao, Shirui Liu, Yi Hu, Jiaqi Li, Muhan Zhang
6 min
Abstract
Modern agent frameworks equip large language models with external skill libraries to solve complex tasks. However, it remains unclear whether these systems can effectively evolve their skills and whether the resulting skills improve task-solving capabilities. To bridge this gap, we introduce ContinualSkillBench, a dynamic evaluation framework for in-context continual skill learning. It covers five representative domains, each containing 100 interconnected subtasks ordered by increasing difficulty and opportunities for cross-task skill reuse. Our experiments show that sequential execution generally improves performance, but the gains vary substantially across models and domains. Moreover, in-context learning performs comparably to explicit skill maintenance on average, suggesting that much of the improvement arises from adaptation to prior context and feedback rather than reusable skill abstraction alone. Explicit skills nevertheless provide selective benefits for tasks requiring reusable procedures or precise outputs. We further find that less capable models tend to accumulate larger, more fragmented collections of task-specific skills. These findings show that current in-context skill evolution mechanisms can support continual adaptation, but still struggle to consistently consolidate experience into robust and transferable skills.
Sam: So the researchers are checking whether the agent can keep its notes organized without accidentally deleting the early chapters.
Alex: That's a good way to put it. To test this, they compared two types of agents. The first type—they call them "sequential agents"—actively maintain a skill library. After each task, they update their stored procedures and try to build on what they've already learned. The second type, the "independent baselines," start every single task from scratch with no memory of what came before.
Sam: And the sequential ones come out ahead?
Alex: Generally, yes—but the gap is smaller and more complicated than you might expect. Here's the part that caught the researchers' attention: in many cases, a much simpler approach worked just as well as maintaining a full skill library. If you simply show the agent the feedback from its most recent task and let it adjust on the fly, it often performs comparably to an agent with a carefully curated library of stored skills.
Sam: So the agent doesn't need a formal library at all? It's just reading the room based on what just happened?
Alex: That's what the evidence suggests. And it raises a real question about what "learning" is actually happening. If an agent improves mainly because it's paying close attention to immediate feedback—rather than because it's built up a deep, reusable understanding—then what looks like growth might really just be sophisticated short-term adaptation.
Sam: Like the difference between a student who genuinely understands algebra, versus one who is very good at spotting patterns in the last few practice problems.
Alex: That's exactly the distinction the paper is drawing out. And it gets more nuanced when you look at how different models handle their libraries. Stronger, more capable models tend to build compact, well-organized libraries—they extract the general principle from a task and store that. Weaker models tend to hoard details: they save long, task-specific instructions that don't transfer well to anything new.
Sam: So a cluttered library can actually slow you down. You spend so much effort searching through irrelevant notes that you'd have been better off with nothing.
Alex: That's a real risk the paper identifies. When the library fills up with narrow, one-off procedures, the agent gets bogged down sifting through instructions that don't apply, rather than quickly finding a useful, general skill. The hard problem, it turns out, isn't just learning—it's learning how to summarize what you've learned into something you can actually use later.
Sam: Does the paper point toward what would need to change for agents to get better at that?
Alex: It points more toward diagnosis than prescription, which is honest given where the field is. The benchmark itself is the contribution—a structured way to measure whether skill accumulation is actually happening, rather than just assuming it is. The finding that immediate feedback often matches explicit skill libraries suggests the field may be overestimating how much those libraries currently help. That's a useful thing to know before investing heavily in more complex library-building systems.
Sam: So the value of the study is in revealing a gap we didn't have a clean way to measure before.
Alex: That's a fair reading. CONTINUALSKILLBENCH gives researchers a concrete tool to track whether agents are genuinely accumulating knowledge across tasks—and right now, the honest answer is that most of them aren't doing it as well as we might hope. That's not a failure of the research; it's exactly the kind of clear-eyed finding that points toward where the work needs to go next.
Sam: It's a bit like discovering that what you thought was a library is actually just a very good short-term memory.
Alex: A well-put summary. Thanks for listening to ResearchPod.