Caijun Xu, Changyi Xiao, Zhongyuan Peng, Yixin Cao
5 min
Abstract
Reinforcement learning has become a central paradigm for advancing reasoning in large language models, yet most existing methods still depend on stronger teacher models or heavily curated difficult datasets, limiting scalable capability improvement. In this paper, we introduce DenoiseRL, a reinforcement learning framework that substitutes external supervision with recovery-oriented optimization over failures from weak models. Instead of relying on stronger supervision or carefully engineered data, DenoiseRL learns directly from incorrect reasoning traces by converting them into opportunities for improvement, making training more scalable and less dependent on external resources. This yields a richer and more diverse learning signal, improving exploration efficiency from imperfect model behavior. As a result, DenoiseRL improves reasoning performance and overall training efficiency while reducing the need for expensive data curation or stronger teacher models. Empirically, DenoiseRL consistently outperforms strong on-policy RL baselines across competitive mathematical and general reasoning benchmarks and promotes stronger self-corrective behavior as training difficulty increases, highlighting an effective and scalable alternative pathway for improving reasoning in large language models.
Alex: So it has to recognize the bad steps and then reason its way around them?
Sam: Exactly. And in practice, the model learns to do something quite specific. It doesn't just blindly follow the error. It actually pauses and writes out a pivot—something like, "Wait, this approach isn't working, let me try a different method." The researchers call this "reasoning recovery." In one example from the paper, a model was given a math problem with a starting step that required finding a very difficult square root. Instead of getting stuck, the model recognized the path was too hard and switched to a different algebraic approach that solved the problem directly. It's the difference between a student who keeps erasing the same wrong answer and one who recognizes they need a different formula entirely.
Alex: That sounds like a much more human way of thinking. But I imagine there's a limit to how much "mess" you can give the model before it just gets confused.
Sam: That is a sharp observation. The study found that if you give the model too much incorrect information—making the majority of its starting steps wrong—it can start to "overthink." It becomes so skeptical of its own logic that it begins doubting perfectly correct steps, leading to very long and inefficient reasoning chains.
Alex: So it becomes a nervous student who double-checks everything ten times, even when they already have the right answer.
Sam: Precisely. There is a sweet spot for this kind of training. The researchers also use a specific grading method to make it work fairly. The model looks at a whole group of attempts for the same problem at once and figures out which ones are better than the average. This way, it learns that reaching the right answer is valuable even if it started from a bad position—not just when everything goes smoothly from the start.
Alex: It's like a teacher grading on a curve. If you started with a messy desk but still finished the project, you get a higher score than someone who started clean and gave up.
Sam: That is the idea. And the evidence shows this approach works significantly better than standard practice. On some of the hardest math competition problems, models using this recovery training were roughly twice as effective as those that trained the conventional way.
Alex: So by spending time cleaning up flawed logic from a weaker model, it actually became better at solving hard problems from scratch. That's a counterintuitive result.
Sam: It is. The researchers found that the "denoising" process didn't just help the model fix errors—it deepened its overall understanding of the reasoning. By repeatedly seeing where logic goes wrong, the model learned to navigate correct paths with much more precision. It developed, in a sense, a stronger internal compass—because it had spent so much time practicing what to do when it was lost.
Alex: It's a grounded way to think about progress. We don't always need a smarter teacher. Sometimes we just need to be better at catching our own mistakes.
Sam: That is the core contribution here. It suggests a path toward models that are more resilient and reliable—not by feeding them more perfect examples, but by turning failure into a structured tool for growth.
Alex: A meaningful shift in perspective. Thanks for listening to ResearchPod.