Sparse reward reinforcement learning (RL) has become a standard tool for improving LLM reasoning, but its success depends critically on the coverage present in the base model. In practice, models are often primed for RL through \emph{mid-training} on curated reasoning traces that teach useful primitive skills such as decomposition, verification, or self-correction. Although effective, this strategy requires manually specifying what the model should learn, and it remains unclear whether such primitive coverage is enough for much harder problems, which require combining these skills into broader solution strategies. We study a more automated approach: \emph{RL-based mid-training} using large corpora of human-written question-answer data. Rather than treating reference solutions as targets to imitate, our method, ExpRL, uses them as \emph{reward scaffolds}: references are hidden from the policy and used only to construct problem-specific grading rubrics for judging on-policy reasoning traces. The policy samples from the original problem prompt, while an LLM judge compares the sampled reasoning trace against the reference solution and assigns outcome-level or process-level dense rewards. This lets ExpRL reinforce partial progress, useful intermediate reductions, and productive reasoning behaviors that sparse final-answer rewards often fail to upweight. On challenging math reasoning tasks, ExpRL yields stronger RL priming than SFT, sparse-reward GRPO, and self-distillation, and provides a better initialization for subsequent sparse-reward RL. Additional mixed-domain experiments further suggest that ExpRL can extend beyond the original math-only setting.
Alex: Welcome to another episode of ResearchPod. Today, we're looking at a new way to train large language models—the AI systems that power things like chatbots—to solve complex reasoning problems.
Sam: We're discussing a paper on a method called Exploratory Reinforcement Learning, or ExpRL. The central puzzle is that these models often fail at hard problems because they don't know how to explore different ways to find a solution.
Alex: So this paper is basically asking how we can get these models to explore more effectively before they even start their final training?
Sam: Exactly. Standard training relies on what researchers call "sparse rewards." Think of it like a teacher who only tells a student if their final answer is correct or wrong, without ever looking at their work. If the student doesn't know how to get to the answer, they just guess randomly—and the teacher's feedback doesn't actually help them learn.
Alex: That makes sense. It's like trying to solve a maze where you only find out if you made it to the exit after you've already walked into a dead end. You don't learn anything from the wrong turns.
Sam: That's a good way to put it. The researchers noticed something important: while models are bad at generating correct answers from scratch, they're actually quite good at checking whether a step is on the right track. The paper calls this the "verification-generation gap"—the model is a better judge than it is a solver.
Alex: So instead of just showing the model the final answer, they use that answer as a guide to grade the steps along the way?
Sam: Exactly. They use the reference solution to build what they call a "reward scaffold." Imagine a tutor who watches you solve a math problem and gives you credit for setting up the equation correctly, even if your final calculation is off. That's the idea behind "dense rewards"—a continuous signal that guides the model toward productive reasoning, rather than a single thumbs up or thumbs down at the end.
Alex: So the model isn't just copying the answer. It's learning the *process* of how to reach it, because it gets rewarded for making progress along the way.
Sam: Precisely. In ExpRL, the model tries out many different ways to solve a problem. An automated judge then compares those attempts against the reference solution, step by step. If the model makes a good intermediate move, it gets a positive score. This reinforces useful behaviors—things like catching its own mistakes or breaking a big problem into smaller parts.
Alex: Wait—so the failure mode in previous methods was that the model was essentially blind to its own partial progress?
Sam: That's the key insight. When feedback only comes at the very end, the model never learns the "how." It's like only ever being told whether you passed or failed a test, with no comments on your answers. By rewarding the process, ExpRL gives the model a map instead of just a final grade.
Alex: How exactly do they measure whether a step is good or bad?
Sam: They break the solution into chunks—segments of reasoning. Each chunk gets compared to the corresponding part of the reference solution. If a segment moves the model's thinking closer to the reference, it gets a boost. If it drifts further away, it gets a penalty. The paper calls this a "segment-level advantage." The model learns, over many attempts, which kinds of moves tend to lead somewhere useful.
Alex: That's much more informative than a single right-or-wrong signal. It's not just about the destination—it's about the quality of the path you take to get there.
Sam: Right. And this has a practical consequence. By training this way first, the model gets much better at generating a wide variety of valid approaches to a problem. Researchers measure this with something called "Pass@k"—the idea being: if you give the model several attempts, what's the chance that at least one of them is correct? ExpRL raises that ceiling significantly, which means the final stage of training has much more to work with.
Alex: Does this approach work across different subjects, or is it specific to one area?
Sam: The study tested it on math, science, and coding. For math and science, having a reliable reference solution is essential—without it, the automated judge can't tell which steps are genuinely productive, and the rewards become noisy and misleading. Coding is a somewhat different case, because you can just run the code and see if it works. The environment itself provides the feedback.
Alex: So in coding, the test results do the judging. But in math and science, you need that reference path to make the whole system function.
Sam: That's a fair summary. The method works best in domains where the reasoning process matters as much as the final answer—and where you have a reliable way to evaluate each step along the way. That's the condition ExpRL is designed to exploit.
Alex: It's a meaningful shift in how we think about training these systems. Instead of just rewarding the right answer, you're building a model that understands how to think its way toward one.
Sam: And that distinction turns out to matter quite a lot. A model that has learned to navigate the reasoning process—not just memorize outcomes—is in a much stronger position when it encounters problems it has never seen before. That's ultimately what this line of research is trying to build toward.
Alex: Thanks for walking us through that. And thanks to everyone listening to ResearchPod.