Chenhao Dang, Siyuan Xiong, Conghui He, Weijia Li
8 min
As autonomous agents become more prevalent, the use of standardized "skills"—packages of procedural knowledge and tool-use conventions—has become essential for constraining model behavior. However, these skills are often designed for massive, expensive closed-source models. This paper asks how we can automatically generate effective, task-specific skills for smaller, open-source models (e.g., Qwen3.5-9B/4B) to reduce deployment costs without sacrificing performance.
The authors propose SKILLER, a reinforcement learning framework that treats the textual skill itself as an optimizable policy. Instead of updating neural network weights, SKILLER uses a strong frontier model (like GPT-5.5 or Claude) as both an actor and a critic. The environment consists of the small-model agent loop. The framework propagates reinforcement learning signals—such as states, diagnostic rewards, and policy updates—entirely through natural language. By comparing the small model's execution trajectory against a reference trajectory, the critic identifies causal errors and provides specific, localized feedback, which the actor then uses to perform bounded edits on the skill artifact.
SKILLER significantly improves the performance of compact models across five diverse benchmarks, including SkillsBench and EarthBench. For the Qwen3.5-9B model, SKILLER achieved absolute gains of 4.3 to 20.4 percentage points over existing skill-generation methods. Remarkably, these small models, when equipped with SKILLER-generated skills, matched the performance of strong closed-source models on single-skill tasks, demonstrating that well-tailored procedural constraints can bridge the capability gap between compact and frontier-scale models.
This work provides a practical pathway for deploying high-performance autonomous agents on consumer-grade hardware. By shifting the computational burden to a one-time skill-generation phase, developers can drastically lower the inference costs of real-world agentic applications while maintaining high reliability and task-specific precision.
Agent skills represent a standardized format for packaging procedural knowledge and domain expertise, serving within agent harness systems as an essential mechanism to continually constrain a language model's behavior space for repeatable, high-quality task execution. However, because strong closed-source models entail high inference costs, current popular agent harnesses, such as Codex and OpenClaw, remain prohibitively expensive when deploying these skills to accomplish real-world tasks. The rapid capability enhancement of open-source models deployable on consumer-grade GPUs presents a compelling opportunity to drastically reduce these costs by leveraging skill-based behavioral constraints. Nevertheless, automatically generating effective skills tailored specifically for such compact models remains a significant practical challenge. To address this, we propose SKILLER, a natural-language-driven reinforcement learning framework designed to automatically generate executor-specific skills for small models, which employs a strong model as the actor and critic, treats the small-model agent system as the environment, and propagates all reinforcement learning signals entirely via natural language. Extensive experimental evaluations across five relevant benchmarks using Qwen3.5-9B and Qwen3.5-4B demonstrate that SKILLER outperforms three open-source and one closed-source skill generation or evolution methods, achieving absolute gains ranging from 4.3 to 20.4 percentage points for the 9B model and 1.8 to 13.3 points for the 4B model, while remarkably matching the performance of strong closed-source models on single-skill tasks in SkillsBench. The project is available at https://github.com/DANG-ai/SKILLER.
Alex: So the big model is coaching the small model, but through language rather than by changing its brain.
Sam: That's the mechanism. And it runs in a loop. The small model tries the task, the system records every action it takes—researchers call this an execution trajectory—and then an official verifier checks whether the final result was correct.
Alex: Like a referee at the end of a practice run.
Sam: Right. The verifier produces a score and diagnostic error messages. Those get combined with a recording of a successful reference run into what the paper calls a structured state—essentially a side-by-side comparison of what the model did versus what a successful attempt looked like.
Alex: So you can see exactly where the paths diverged.
Sam: Precisely. That structured state goes to a critic module—powered by the large frontier model—which identifies the earliest point where things went wrong and generates natural-language suggestions for fixing the instructions.
Alex: And then someone rewrites the manual based on those suggestions?
Sam: Not someone—something. An actor module, also powered by the frontier model, takes those suggestions and applies targeted edits: inserting a missing step, deleting an ineffective rule, or replacing a vague instruction with a precise one. The skill evolves in small, controlled steps rather than being rewritten from scratch.
Alex: That's a meaningful distinction. Small surgical edits rather than starting over every time.
Sam: And the system keeps a structured memory of past failures, diagnoses, and accepted edits—a kind of journal of what worked and what didn't. That prevents the system from repeating the same mistakes and gives the actor and critic relevant context for each new round of revision.
Alex: So it genuinely learns from its own history.
Sam: It does. And it can run these optimization loops across batches of tasks simultaneously, aggregating evidence from multiple failures before committing to a single skill update. That makes each revision more robust.
Alex: How well does this actually work when tested?
Sam: On software engineering tasks—where the benchmarks strictly penalize unverified file edits and out-of-order tool use—SKILLER outperforms both open-source and closed-source baseline methods by substantial margins. Those are exactly the kinds of structured, rule-sensitive tasks where the critic-actor loop has the most to fix.
Alex: What about more open-ended tasks, like searching for information?
Sam: It consistently matches or surpasses the strongest baselines there too. The optimized instructions act as guardrails, regulating the search process and enforcing systematic verification even when the task doesn't have a single right answer.
Alex: And do these skills hold up on problems the system has never seen before?
Sam: On held-out test sets—problems kept completely separate from the optimization process—SKILLER still outperforms all baselines. That suggests the loop is extracting genuinely reusable procedural rules, not just memorizing solutions to specific examples.
Alex: There's one result I want to make sure we discuss. You mentioned the small model can actually beat a larger model?
Sam: Yes, and it's worth understanding why. A smaller model equipped with optimized skills can achieve a higher pass rate than a larger model using unoptimized skills. The reason is that for structured, rule-sensitive tasks, a well-crafted behavioral policy—one that specifically addresses the model's known failure modes—is more valuable than simply having more parameters. Disciplined guidance outweighs raw scale.
Alex: So it's not just about how big the model is. It's about how well the instructions match the model's actual capabilities.
Sam: That's the central finding. And you can see it in the structure of the instructions themselves. Baseline methods tend to produce long, repetitive prompts that overwhelm compact models. SKILLER produces concise instructions that closely resemble human-authored guidelines—short enough to fit within the limited context windows of small models, precise enough to prevent the most common errors.
Alex: Context window being the amount of text the model can hold in its working memory at once.
Sam: Exactly. By keeping instructions compact and offloading complex procedural reasoning to external tools, the framework avoids overloading the model. And that structural efficiency also keeps costs down—SKILLER achieves its performance advantage while remaining substantially more cost-efficient than methods that rely on exhaustive text generation.
Alex: Are there real limits to this approach? Things that could cause it to break down?
Sam: There are, and the paper is candid about them. The framework depends on two things that aren't always available: a strong frontier model to run the actor-critic loop, and a reliable verifier that can produce clear, diagnostic feedback. Without those two components, you can't reliably isolate what went wrong or generate precise edits. The optimization loop simply stalls.
Alex: So it's a powerful approach within a specific set of conditions, not a universal fix.
Sam: That's the honest boundary. Where those conditions are met—structured tasks with clear success criteria and access to a capable frontier model—the framework offers a practical path to making lightweight systems significantly more capable without any retraining.
Alex: And looking further out, this suggests that smaller devices—phones, local assistants—could potentially run optimization loops in the background, tailoring their own instruction sets to the tasks their users actually care about.
Sam: That direction follows directly from the research. By treating the text of an instruction as something that can be systematically improved, you decouple capability from model size in a meaningful way. The skill becomes the lever, not the hardware.
Alex: That's a genuinely practical shift in how we think about deploying AI. Thanks for walking through it, Sam.
Sam: Thanks, Alex. And thanks to everyone listening to ResearchPod.