Violet Xiang, Amrith Setlur, Chase Blagden, Nick Haber, Aviral Kumar
5 min
Abstract
Sparse reward reinforcement learning (RL) has become a standard tool for improving LLM reasoning, but its success depends critically on the coverage present in the base model. In practice, models are often primed for RL through \emph{mid-training} on curated reasoning traces that teach useful primitive skills such as decomposition, verification, or self-correction. Although effective, this strategy requires manually specifying what the model should learn, and it remains unclear whether such primitive coverage is enough for much harder problems, which require combining these skills into broader solution strategies. We study a more automated approach: \emph{RL-based mid-training} using large corpora of human-written question-answer data. Rather than treating reference solutions as targets to imitate, our method, ExpRL, uses them as \emph{reward scaffolds}: references are hidden from the policy and used only to construct problem-specific grading rubrics for judging on-policy reasoning traces. The policy samples from the original problem prompt, while an LLM judge compares the sampled reasoning trace against the reference solution and assigns outcome-level or process-level dense rewards. This lets ExpRL reinforce partial progress, useful intermediate reductions, and productive reasoning behaviors that sparse final-answer rewards often fail to upweight. On challenging math reasoning tasks, ExpRL yields stronger RL priming than SFT, sparse-reward GRPO, and self-distillation, and provides a better initialization for subsequent sparse-reward RL. Additional mixed-domain experiments further suggest that ExpRL can extend beyond the original math-only setting.
Alex: How exactly do they measure whether a step is good or bad?
Sam: They break the solution into chunks—segments of reasoning. Each chunk gets compared to the corresponding part of the reference solution. If a segment moves the model's thinking closer to the reference, it gets a boost. If it drifts further away, it gets a penalty. The paper calls this a "segment-level advantage." The model learns, over many attempts, which kinds of moves tend to lead somewhere useful.
Alex: That's much more informative than a single right-or-wrong signal. It's not just about the destination—it's about the quality of the path you take to get there.
Sam: Right. And this has a practical consequence. By training this way first, the model gets much better at generating a wide variety of valid approaches to a problem. Researchers measure this with something called "Pass@k"—the idea being: if you give the model several attempts, what's the chance that at least one of them is correct? ExpRL raises that ceiling significantly, which means the final stage of training has much more to work with.
Alex: Does this approach work across different subjects, or is it specific to one area?
Sam: The study tested it on math, science, and coding. For math and science, having a reliable reference solution is essential—without it, the automated judge can't tell which steps are genuinely productive, and the rewards become noisy and misleading. Coding is a somewhat different case, because you can just run the code and see if it works. The environment itself provides the feedback.
Alex: So in coding, the test results do the judging. But in math and science, you need that reference path to make the whole system function.
Sam: That's a fair summary. The method works best in domains where the reasoning process matters as much as the final answer—and where you have a reliable way to evaluate each step along the way. That's the condition ExpRL is designed to exploit.
Alex: It's a meaningful shift in how we think about training these systems. Instead of just rewarding the right answer, you're building a model that understands how to think its way toward one.
Sam: And that distinction turns out to matter quite a lot. A model that has learned to navigate the reasoning process—not just memorize outcomes—is in a much stronger position when it encounters problems it has never seen before. That's ultimately what this line of research is trying to build toward.
Alex: Thanks for walking us through that. And thanks to everyone listening to ResearchPod.