Zhiyuan He, Siwei Zhang, Zhiwen Zhou, Yuqing Yang, Yu Kang, Yuge Zhang, Luna K. Qiu, Tin Yan Tsui, Jiahang Xu, Chong Luo
5 min
Abstract
Modern agents operate inside agent harnesses that manage tools, context, and control flow, making the harness a critical part of the agent system. Our original Agent Lightning introduced a disaggregated architecture that connects arbitrary agents to RL training through an LLM endpoint proxy, an approach later adopted by frameworks such as verl Uni-Agent, AReaL 2.0, slime, and Polar. We refer to this paradigm as harnessed agentic RL, where the deploy-time harness directly participates in model post-training. Harnessed agentic RL differs fundamentally from traditional agentic RL: the harness, rather than the training engine, owns the environment interaction loop, while the trainer observes only sequences of LLM request-response pairs. This introduces challenges in retokenization, sample merging, advantage calculation, loss normalization, and backend scheduling, which can substantially affect training stability and effectiveness. We present Agent Lightning v1.0, a lightweight framework for harnessed agentic RL implemented in approximately 3,500 lines of code. It supports arbitrary agent harnesses and serves as a practical testbed for studying these challenges. We evaluate it on instruction-following, search, and coding agents, and provide a complete reproducible pipeline for coding-agent RL. Using only 6K training examples and modest compute, RL improves Qwen3.5-9B on SWE-bench Verified from 41.8% to 56.4%, a 14.6-point absolute gain. We release the complete workflow and training scripts to facilitate reproducible research on harnessed agentic RL.
Alex: So the system is dealing with text that keeps subtly reshaping itself.
Sam: Exactly. Agent Lightning handles this through what the paper calls best-effort sequence merging—when the pieces don't line up cleanly, it closes the sequence and moves on rather than crashing or producing corrupted data. It's a pragmatic solution to a genuinely fiddly problem.
Alex: Does the unpredictable fragmentation cause issues beyond just the text itself?
Sam: It does. One rollout—one complete attempt at a task—can expand into multiple training samples because of how the shell processes things. If you then calculate your learning signal by averaging across those samples equally, you end up unfairly weighting the tasks that happened to get fragmented the most. Agent Lightning avoids this by computing that signal at the level of the whole rollout, keeping the statistics consistent regardless of how the pieces ended up divided.
Alex: And does any of this translate into measurable improvement?
Sam: The authors report training a model called Qwen3.5-9B on around six thousand examples, and it improved its success rate on complex coding benchmarks by fourteen percentage points in absolute terms. That's a meaningful gain for a relatively modest training set.
Alex: Are there limitations they're upfront about?
Sam: Several. The best-effort merging approach works well in most cases, but when retokenization drift is high—when the text reshapes itself significantly—some data does get dropped rather than merged. The authors flag this as an area for future work.
Alex: What else?
Sam: The framework also requires careful filtering of training data and strict network isolation. Without that isolation, the AI can find shortcuts—like downloading a solution from the internet rather than actually solving the problem. The whole point is to teach the model genuine software development skills, and those safeguards are what keep it honest.
Alex: So the infrastructure around the training matters just as much as the training method itself.
Sam: That's a fair summary. And one of the broader implications the authors point to is that this kind of self-contained setup—where everything runs locally and the training loop is fully controlled—makes the whole process more reproducible. Smaller research teams without access to large shared computing clusters could realistically run experiments like this themselves.
Alex: That does seem like a practical step forward for the field. Thanks, Sam. And thanks to everyone listening to ResearchPod.