Aofan Yu, Chenyu Zhou, Tianyi Xu, Zihan Guo, Rong Shan, Zhihui Fu, Jun Wang, Weiwen Liu, Yong Yu, Weinan Zhang, Jianghao Lin
5 min
Abstract
Agent systems increasingly use textual skills to encode reusable task procedures, but injecting these skills into the prompt at every step incurs substantial context overhead and exposes skill content as plaintext. We present LatentSkill, a framework that converts textual skills into plug-and-play LoRA adapters through a pretrained hypernetwork. LatentSkill stores skill knowledge in weight space rather than context space, removing per-step skill tokens while preserving modular loading, scaling, and composition. On ALFWorld and Search-QA, LatentSkill outperforms the corresponding in-context skill baseline while using substantially fewer prefill tokens: it improves ALFWorld success by 21.4 and 13.4 points on the seen and unseen splits with 64.1% fewer prefill tokens, and improves Search-QA exact match by 3.0 points with 72.2% lower skill-token overhead. Further analysis shows that generated skill LoRAs form a structured semantic geometry, can be precisely controlled via the LoRA scaling coefficient, and can be composed through parameter-space arithmetic when skill components are aligned. These findings suggest that weight-space skills provide an efficient, modular, and less exposed substrate for extending LLM agents.
Alex: So it's not just combining whole skills — it's more like building from shared parts.
Sam: Right. The agent doesn't relearn the basics every time. It snaps the right components together.
Alex: How does the system actually learn to do this translation — turning a text document into a useful set of weights?
Sam: There are two training phases. The first is a kind of reading comprehension test: the compiler learns to generate an adapter that can reconstruct the original skill document. If it can reproduce the document from the weights alone, it's demonstrated that it understood the content.
Alex: And the second phase?
Sam: That's more like a practical exam. They show the compiler recordings of an expert agent actually completing tasks — step by step — and train it to generate adapters that make the model replicate that behaviour. So it's not just understanding what the document says; it's learning what *doing the skill* looks like in practice.
Alex: One phase for comprehension, one for execution. Does this hold up when instructions are messy or poorly written?
Sam: That's where the weight-space approach shows a clear advantage. The researchers tested what happens when you paraphrase instructions, strip out formatting, or introduce noise. Traditional prompt-based systems saw noticeable performance drops. The latent skill version stayed stable, because it captures the underlying meaning of the instructions rather than reacting to surface-level word choices.
Alex: And I'd imagine that connects to the security point you flagged earlier — prompt injection attacks, where someone slips malicious instructions into the text the model reads.
Sam: It does. Because LatentSkill stores expertise as mathematical weights rather than plain text, there's no visible instruction string for an attacker to override. The paper shows that when researchers attempted to hijack the agent through the prompt, the text-based version's performance dropped sharply, while the weight-space version held up. The paper also tested performance on two standard benchmarks — environments where agents navigate rooms and answer questions — and LatentSkill outperformed the traditional text-based approach on both, while using meaningfully fewer computational resources.
Alex: What are the limitations worth flagging?
Sam: A few. The hypernetwork is trained for a specific base model architecture. If you swap out the underlying model, you'd likely need to retrain the compiler from scratch — it's not universally portable yet. And the quality of the whole system depends on how carefully the skills are structured. If the components aren't well-aligned, combining them can produce unexpected behaviour. It's a precise engineering process, not a plug-and-play solution.
Alex: So it offers real gains in efficiency and security, but it asks for careful design work in return.
Sam: That's the trade-off. The results suggest it's a worthwhile one — but it's not a shortcut. As AI agents take on more complex, multi-step tasks, having a principled way to package and combine expertise becomes increasingly important. LatentSkill is one concrete proposal for how to do that.
Alex: A clear look at a meaningful shift in how AI agents might manage knowledge. Thanks for walking us through it, Sam — and thank you for listening to ResearchPod.