Jiayin Zhu, Kelong Mao, Yudong Guo, Dengbo He, Sulong Xu, Simiu Gu, Yutao Yue
6 min
Abstract
Skills are becoming a reusable operational layer for LLM agents, encoding SOPs, domain rules, tool workflows, scripts, and validation routines. In realistic skill repositories, overlapping skills make reliable skill-use difficult. Final verifier success is too coarse for both evaluation and training, since an agent may pass through trial and error while selecting distractor skills, skipping required steps, composing workflows incorrectly or omitting final checks. We introduce SkillCoach, a self-evolving rubric framework for evaluating and enhancing agentic skill-use. SkillCoach derives skill-grounded process rubrics from real rollouts and evaluates trajectories along four dimensions: skill selection, skill following, skill composition, and skill-grounded reflection. It keeps the external verifier as a separate outcome signal, allowing process quality to be distinguished from accidental task success. The evolved rubrics further serve as process supervision for selecting high-quality training trajectories. Experiments show that evolved rubrics substantially improve evaluation quality, expose failures hidden by final accuracy, and provide stronger supervision signals than outcome-only filtering for enhancing agentic skill-use.
Alex: That last one is interesting. Checking your own work is something a lot of humans skip too.
Sam: It is. And it matters because an agent that never reflects on its own steps can't catch its own errors. The rubric treats that self-checking behavior as a core skill, not an optional extra.
Alex: Does all of this just change how agents are graded, or does it actually change how they're trained?
Sam: It does both. By using these rubrics as a filter, researchers can identify the high-quality attempts—the runs where the agent genuinely followed the right steps—and use only those to train the model further. It's a way of making sure the training data itself is clean. The agent learns the process, not just how to land on a correct-looking answer.
Alex: So there's also a separate challenge about what happens when the agent has access to a very large library of tools. How does that complicate things?
Sam: The paper calls this the "distractor-boundary" problem. Imagine a toolbox with ten tools—it's straightforward to pick the right one. Now imagine ten thousand tools. The research shows that as the number of irrelevant tools grows, models eventually hit a point where they can no longer reliably pick the correct one at all.
Alex: Like trying to find a specific wrench in a warehouse where every aisle looks identical.
Sam: That's a good way to put it. And the type of distraction matters as much as the quantity. The most damaging case is what the paper calls "near-neighbor" distractors—tools that are very similar to the correct one. Think of two slightly different versions of the same compliance form. The model sees them as nearly identical and picks the wrong one. That kind of mistake can break an entire workflow downstream.
Alex: So it's not just about having too many options. It's about having options that are hard to distinguish.
Sam: Right. And the paper makes a point that's easy to miss: measuring whether an agent triggers any skill is not the same as measuring whether it triggers the right skill. The researchers call the ability to make that distinction "boundary control"—knowing exactly when to use a tool, which one to pick, and crucially, when to stay silent and use nothing at all.
Alex: So the system needs to be just as good at saying "none of these apply" as it is at picking the correct one.
Sam: Precisely. The study suggests that stronger models don't necessarily avoid these mistakes entirely, but they degrade much more slowly as the library grows more complex. They hold their accuracy longer before hitting that collapse point.
Alex: That reframes how we should think about evaluating these agents. It's not just about whether they get the right answer—it's about how gracefully they handle a messy, crowded environment.
Sam: That's the core insight. SkillCoach uses this understanding to move beyond simple success metrics. By grading the process and stress-testing the agent's ability to navigate a noisy tool library, it tries to distinguish between a model that is genuinely capable and one that is getting lucky.
Alex: Are there limits to what this approach can tell us?
Sam: The paper flags two main ones. First, the test environments used in this research are smaller and more controlled than the massive, constantly changing tool repositories you'd find in a large enterprise. Whether the findings scale up to that level is still an open question. Second, the training method is what researchers call "offline"—the agent learns from a fixed dataset of past attempts. It hasn't been tested in a live setting where the agent learns from its own real-time mistakes as they happen. Both of those are meaningful gaps that future work would need to address.
Alex: So SkillCoach is a meaningful step toward building AI agents we can actually trust in high-stakes situations—not because it solves every problem, but because it asks a more honest question. Not just "did it get the right answer?" but "did it earn that answer?"
Sam: That's a fair summary. And in domains where the wrong step can have real consequences—emergency response, compliance, infrastructure monitoring—that distinction matters quite a lot.
Alex: Thanks for listening to ResearchPod.