Agent systems increasingly use textual skills to encode reusable task procedures, but injecting these skills into the prompt at every step incurs substantial context overhead and exposes skill content as plaintext. We present LatentSkill, a framework that converts textual skills into plug-and-play LoRA adapters through a pretrained hypernetwork. LatentSkill stores skill knowledge in weight space rather than context space, removing per-step skill tokens while preserving modular loading, scaling, and composition. On ALFWorld and Search-QA, LatentSkill outperforms the corresponding in-context skill baseline while using substantially fewer prefill tokens: it improves ALFWorld success by 21.4 and 13.4 points on the seen and unseen splits with 64.1% fewer prefill tokens, and improves Search-QA exact match by 3.0 points with 72.2% lower skill-token overhead. Further analysis shows that generated skill LoRAs form a structured semantic geometry, can be precisely controlled via the LoRA scaling coefficient, and can be composed through parameter-space arithmetic when skill components are aligned. These findings suggest that weight-space skills provide an efficient, modular, and less exposed substrate for extending LLM agents.
Alex: Welcome to another episode of ResearchPod.
Sam: Today we're looking at a paper called LatentSkill — and it tackles a surprisingly fundamental problem with how AI agents currently work. Right now, when an AI agent needs to do something specific — search the web, navigate a virtual environment, complete a multi-step task — it reads a text document that tells it how. Every single time. Those documents can be long, and reading them over and over consumes a huge portion of the system's working memory.
Alex: So the central question is: can we move those instructions out of the reading pile and bake them directly into how the model thinks?
Sam: That's exactly the puzzle. And LatentSkill proposes a concrete answer. Instead of handing the agent a manual to read each time, the system compiles those instructions into a tiny set of mathematical adjustments applied directly to the model's internal settings. Think of it like the difference between reading a recipe every time you cook versus having made that dish so many times that the steps are just part of how your hands move.
Alex: So the knowledge shifts from something the model reads to something the model *is* — at least temporarily.
Sam: That's a good way to put it. The mechanism that makes this work is called a hypernetwork. It's a second, smaller neural network — a kind of specialist — whose only job is to read a skill document and produce those mathematical adjustments. The adjustments themselves are applied using a technique called Low-Rank Adaptation, or LoRA.
Alex: Can you break LoRA down for someone who hasn't encountered it?
Sam: Sure. Imagine you have a large, complex machine — the main AI model — and you don't want to rebuild it from scratch every time you need it to do something new. A LoRA adapter is like clipping a small, specialised attachment onto that machine. It's a compact set of extra settings that nudges the machine's behaviour in a specific direction, without touching the original machinery at all. So the hypernetwork reads the skill document and generates one of these attachments on the fly — which is why it's more efficient than simply pasting the instructions into the prompt.
Alex: And once the skill lives in those internal settings, you no longer need to include the instructions in the text the model reads at all.
Sam: Exactly. That saves computing resources, and it also means the instructions are hidden from the model's visible text stream entirely — which turns out to matter for security, but we'll get to that.
Alex: You mentioned these skills can be combined. How does that work?
Sam: Because skills are now just sets of numbers, you can do arithmetic with them. You can take the weights for one skill, add them to the weights for another, and get a hybrid behaviour. It's a bit like mixing paint — you're not choosing between red and blue, you're producing something in between. And it works at a finer level too: if two different tasks share a common sub-step, you can isolate just that component and reuse it, rather than encoding the same behaviour twice.
Alex: So it's not just combining whole skills — it's more like building from shared parts.
Sam: Right. The agent doesn't relearn the basics every time. It snaps the right components together.
Alex: How does the system actually learn to do this translation — turning a text document into a useful set of weights?
Sam: There are two training phases. The first is a kind of reading comprehension test: the compiler learns to generate an adapter that can reconstruct the original skill document. If it can reproduce the document from the weights alone, it's demonstrated that it understood the content.
Alex: And the second phase?
Sam: That's more like a practical exam. They show the compiler recordings of an expert agent actually completing tasks — step by step — and train it to generate adapters that make the model replicate that behaviour. So it's not just understanding what the document says; it's learning what *doing the skill* looks like in practice.
Alex: One phase for comprehension, one for execution. Does this hold up when instructions are messy or poorly written?
Sam: That's where the weight-space approach shows a clear advantage. The researchers tested what happens when you paraphrase instructions, strip out formatting, or introduce noise. Traditional prompt-based systems saw noticeable performance drops. The latent skill version stayed stable, because it captures the underlying meaning of the instructions rather than reacting to surface-level word choices.
Alex: And I'd imagine that connects to the security point you flagged earlier — prompt injection attacks, where someone slips malicious instructions into the text the model reads.
Sam: It does. Because LatentSkill stores expertise as mathematical weights rather than plain text, there's no visible instruction string for an attacker to override. The paper shows that when researchers attempted to hijack the agent through the prompt, the text-based version's performance dropped sharply, while the weight-space version held up. The paper also tested performance on two standard benchmarks — environments where agents navigate rooms and answer questions — and LatentSkill outperformed the traditional text-based approach on both, while using meaningfully fewer computational resources.
Alex: What are the limitations worth flagging?
Sam: A few. The hypernetwork is trained for a specific base model architecture. If you swap out the underlying model, you'd likely need to retrain the compiler from scratch — it's not universally portable yet. And the quality of the whole system depends on how carefully the skills are structured. If the components aren't well-aligned, combining them can produce unexpected behaviour. It's a precise engineering process, not a plug-and-play solution.
Alex: So it offers real gains in efficiency and security, but it asks for careful design work in return.
Sam: That's the trade-off. The results suggest it's a worthwhile one — but it's not a shortcut. As AI agents take on more complex, multi-step tasks, having a principled way to package and combine expertise becomes increasingly important. LatentSkill is one concrete proposal for how to do that.
Alex: A clear look at a meaningful shift in how AI agents might manage knowledge. Thanks for walking us through it, Sam — and thank you for listening to ResearchPod.