Skills are becoming a reusable operational layer for LLM agents, encoding SOPs, domain rules, tool workflows, scripts, and validation routines. In realistic skill repositories, overlapping skills make reliable skill-use difficult. Final verifier success is too coarse for both evaluation and training, since an agent may pass through trial and error while selecting distractor skills, skipping required steps, composing workflows incorrectly or omitting final checks. We introduce SkillCoach, a self-evolving rubric framework for evaluating and enhancing agentic skill-use. SkillCoach derives skill-grounded process rubrics from real rollouts and evaluates trajectories along four dimensions: skill selection, skill following, skill composition, and skill-grounded reflection. It keeps the external verifier as a separate outcome signal, allowing process quality to be distinguished from accidental task success. The evolved rubrics further serve as process supervision for selecting high-quality training trajectories. Experiments show that evolved rubrics substantially improve evaluation quality, expose failures hidden by final accuracy, and provide stronger supervision signals than outcome-only filtering for enhancing agentic skill-use.
Alex: Welcome to another episode of ResearchPod. Today, we're looking into why AI agents might pass a test but fail when you actually put them to work.
Sam: That's the central puzzle. We're discussing a system called SkillCoach, designed to fix what the researchers call the "accidental success" trap. The core claim is that checking whether an agent got the right final answer isn't enough to know if it actually did the work correctly.
Alex: So this is asking how we tell the difference between an AI that's genuinely following the right steps and one that's just guessing its way to a correct answer?
Sam: Exactly. Think about an AI agent that monitors flood levels. It needs to follow a specific set of step-by-step instructions—what engineers call a Standard Operating Procedure. If it ignores those steps but happens to guess the right number, it passes the final check. But that behavior isn't reliable. Next time, the guess might be wrong.
Alex: That's like a student guessing the right answer on a math test instead of showing their work. If they don't know the steps, they'll fail the next, harder problem.
Sam: Perfect analogy. The researchers found that relying only on the final outcome is what they call a "coarse" signal—it's too blurry to tell you much. It masks the fact that the agent might have picked the wrong tools or stumbled into the answer by luck. SkillCoach is designed to fix that by looking at the process itself.
Alex: How does it actually see what the agent is doing?
Sam: It uses something the paper calls "process-level supervision." Think of a driving instructor who doesn't just care that you arrived at your destination. They grade you on whether you checked your mirrors, signaled at the right times, and followed the speed limit. SkillCoach creates a kind of grading rubric that evaluates each step of the process, not just the final result.
Alex: So the rubric is the key piece. But how does it know what the "right" process looks like for every possible task?
Sam: That's where the "self-evolving" part comes in. It starts with a basic set of rules, then watches the agent work and gradually refines those rules based on what it observes. The paper describes this refinement process using the term "validation-gated local patches"—which just means it checks whether each update to its own rules actually makes it a better judge before keeping the change. It learns from its own mistakes.
Alex: Like an instructor who updates their grading sheet after each class. What specific things does the rubric actually look for?
Sam: The paper breaks it down into four dimensions. First, skill selection—did the agent pick the right tool for the job? Second, skill following—did it use that tool correctly? Third, skill composition—did it combine multiple tools in the right order? And fourth, skill-grounded reflection—did it stop and check its own work along the way? The system also deliberately includes "distractor skills," which are irrelevant tools mixed in with the correct ones, specifically to test whether the agent can tell the difference.
Alex: That last one is interesting. Checking your own work is something a lot of humans skip too.
Sam: It is. And it matters because an agent that never reflects on its own steps can't catch its own errors. The rubric treats that self-checking behavior as a core skill, not an optional extra.
Alex: Does all of this just change how agents are graded, or does it actually change how they're trained?
Sam: It does both. By using these rubrics as a filter, researchers can identify the high-quality attempts—the runs where the agent genuinely followed the right steps—and use only those to train the model further. It's a way of making sure the training data itself is clean. The agent learns the process, not just how to land on a correct-looking answer.
Alex: So there's also a separate challenge about what happens when the agent has access to a very large library of tools. How does that complicate things?
Sam: The paper calls this the "distractor-boundary" problem. Imagine a toolbox with ten tools—it's straightforward to pick the right one. Now imagine ten thousand tools. The research shows that as the number of irrelevant tools grows, models eventually hit a point where they can no longer reliably pick the correct one at all.
Alex: Like trying to find a specific wrench in a warehouse where every aisle looks identical.
Sam: That's a good way to put it. And the type of distraction matters as much as the quantity. The most damaging case is what the paper calls "near-neighbor" distractors—tools that are very similar to the correct one. Think of two slightly different versions of the same compliance form. The model sees them as nearly identical and picks the wrong one. That kind of mistake can break an entire workflow downstream.
Alex: So it's not just about having too many options. It's about having options that are hard to distinguish.
Sam: Right. And the paper makes a point that's easy to miss: measuring whether an agent triggers any skill is not the same as measuring whether it triggers the right skill. The researchers call the ability to make that distinction "boundary control"—knowing exactly when to use a tool, which one to pick, and crucially, when to stay silent and use nothing at all.
Alex: So the system needs to be just as good at saying "none of these apply" as it is at picking the correct one.
Sam: Precisely. The study suggests that stronger models don't necessarily avoid these mistakes entirely, but they degrade much more slowly as the library grows more complex. They hold their accuracy longer before hitting that collapse point.
Alex: That reframes how we should think about evaluating these agents. It's not just about whether they get the right answer—it's about how gracefully they handle a messy, crowded environment.
Sam: That's the core insight. SkillCoach uses this understanding to move beyond simple success metrics. By grading the process and stress-testing the agent's ability to navigate a noisy tool library, it tries to distinguish between a model that is genuinely capable and one that is getting lucky.
Alex: Are there limits to what this approach can tell us?
Sam: The paper flags two main ones. First, the test environments used in this research are smaller and more controlled than the massive, constantly changing tool repositories you'd find in a large enterprise. Whether the findings scale up to that level is still an open question. Second, the training method is what researchers call "offline"—the agent learns from a fixed dataset of past attempts. It hasn't been tested in a live setting where the agent learns from its own real-time mistakes as they happen. Both of those are meaningful gaps that future work would need to address.
Alex: So SkillCoach is a meaningful step toward building AI agents we can actually trust in high-stakes situations—not because it solves every problem, but because it asks a more honest question. Not just "did it get the right answer?" but "did it earn that answer?"
Sam: That's a fair summary. And in domains where the wrong step can have real consequences—emergency response, compliance, infrastructure monitoring—that distinction matters quite a lot.
Alex: Thanks for listening to ResearchPod.