Post-training is essential for refining the domain-specific capabilities of large language models (LLMs), yet existing reward optimization and distribution matching methods tightly couple policy exploration with distribution alignment. This coupling forces expensive exploration directly on the policy model and severely hinders the asynchronous generation, reuse, and cross-model transfer of optimization signals. In this paper, we propose Proxy-guided Update Signal Transfer (PUST), a novel post-training framework that fundamentally decouples update-signal exploration from distribution alignment. Instead of utilizing the primary model for costly exploration, PUST employs a lightweight proxy model as an efficient testbed to discover high-reward behaviors. We extract the relative improvement signal between the proxy's initial and optimized states, transferring this directional update to the primary model to guide its policy alignment. This decoupled pipeline, comprising proxy exploration, update-signal extraction, and signal transfer, significantly reduces computational overhead and enables optimization signals to be asynchronously generated, cached, and reused. Crucially, by transferring relative improvements rather than absolute policy distributions, PUST naturally supports weak-to-strong improvement and seamless cross-model transfer. Systematic evaluations on Qwen3-family models across math and code domains demonstrate that update signals extracted from substantially weaker proxies can robustly and adjustably enhance stronger primary models. Ultimately, PUST transforms post-training from a monolithic online optimization process into a highly modular, reusable, and cost-efficient paradigm.
Alex: Welcome to another episode of ResearchPod. Today, we're looking at a new approach to training large language models — specifically a framework called Proxy-guided Update Signal Transfer, or PUST.
Sam: So what's the core problem this is trying to solve? Is it basically that training these massive models is just too expensive?
Alex: That's right. Think about how a large language model learns. Right now, the model has to do two very different jobs at once. First, it has to explore — trying out lots of different responses to figure out which ones are actually good. Then it has to align — adjusting its own behaviour based on what it learned. Doing both at the same time, with a model that might have billions of internal settings, is slow and enormously costly. And if you want to upgrade to a newer model, you have to start that whole expensive process from scratch.
Sam: So the paper is asking: what if we split those two jobs apart? Do the expensive exploration once, and then somehow hand the lessons over to the bigger model?
Alex: Exactly. The idea is to use a small, cheap "proxy" model as a kind of scout. It does the hard work of exploring the problem space — figuring out which directions lead to better answers. Then, instead of throwing that work away, you capture what it learned and pass it on to the larger, more powerful model.
Sam: But how does that handoff actually work? Does the proxy just give the big model its best answers and say "copy these"?
Alex: That's where it gets interesting. Simply copying the proxy's final answers wouldn't work well, because the proxy is a weaker model — its specific answers might not be right for a much more capable system. Instead, the proxy captures something more abstract: the direction of its improvement. Imagine you're learning to shoot free throws in basketball. A coach doesn't just show you a perfect shot and say "do that." They tell you: "you were releasing too early, shift your elbow slightly left." That directional guidance — the change — is what transfers, not the specific shot.
Sam: So the proxy is producing something like a map of "this way is better, that way is worse," rather than a fixed answer.
Alex: Precisely. It calculates the difference between how it was making decisions before training and how it makes them after. That difference — that shift in which options it considers more or less likely — becomes the signal that gets handed to the primary model.
Sam: And because it's a relative signal rather than an absolute answer, does that mean the size gap between the proxy and the main model doesn't matter much?
Alex: That's the key finding the researchers highlight. Even a considerably smaller, weaker proxy can produce a directional signal that meaningfully improves a much larger model. The signal isn't saying "be like me." It's saying "move in this direction."
Sam: But if the big model is just following the proxy's directions, how does it avoid picking up the proxy's bad habits along the way?
Alex: Good question. The primary model uses what you might think of as an anchor. At every step, it compares where it currently is to where it started — its original, stable foundation. If it starts drifting too far from that foundation, a built-in penalty pulls it back. So it's genuinely learning from the proxy's guidance, but it never loses sight of its own solid base. Think of it like a ship using a compass heading from a smaller scout vessel, while still checking its own charts to make sure it hasn't wandered into shallow water.
Sam: So the big model isn't just blindly following — it's using the proxy's signal as a nudge, while staying grounded in its own starting point.
Alex: Exactly. And because the signal is separable from the model that generated it, it can be stored and reused. You run the proxy's exploration once, save the resulting signal, and apply it to different primary models without repeating the costly exploration phase each time.
Sam: That does sound like a meaningful efficiency gain. Are there any real limitations, or does it work cleanly in practice?
Alex: The authors are careful to note one important caveat. You have to calibrate how strongly the primary model responds to the proxy's signal. There's a scaling factor that controls the intensity of the update — essentially, how much weight the big model gives to the proxy's coaching. Set it too high, and the model overcorrects and becomes unstable. Set it too low, and you don't get much benefit. Finding that balance is a genuine practical challenge, and the paper doesn't suggest it resolves itself automatically.
Sam: It's a bit like turning up the volume on advice. Too quiet and you can't hear it. Too loud and it drowns out your own judgement.
Alex: That's a fair way to put it. The framework gives you the tools, but using them well still requires careful tuning.
Sam: So stepping back — what's the broader significance here? Is this mainly a cost-saving measure, or does it change how we think about model training more fundamentally?
Alex: Both, in a way. On the practical side, it reduces the computational cost of improving large models, which matters a great deal given how expensive this work has become. But conceptually, it's a shift in how we think about what "training" produces. Instead of treating a training run as something you do to a specific model and then discard, this approach treats the lessons learned as a portable asset — something you can extract, store, and apply elsewhere. That reframing could have real implications for how model development pipelines are structured going forward, though the paper is appropriately cautious about how broadly that will generalise.
Sam: So the value isn't just in the model you end up with — it's in the signal itself, which you can keep using.
Alex: That's the core insight. By separating exploration from alignment, and treating improvement as something transferable rather than locked inside a single training run, the researchers have proposed a more modular way to build and upgrade these systems. It's a measured but meaningful step toward making large model training less of a one-time, all-or-nothing process. Thanks for listening to ResearchPod.