Liangchen Luo, Yinxiao Liu, Rosanne Liu, Samrat Phatale, Meiqi Guo, Harsh Lara, Yunxuan Li, Lei Shu, Yun Zhu, Lei Meng, Jiao Sun, Abhinav Rastogi
4 min
How can we scale the collection of process supervision data for training Process Reward Models (PRMs) without the prohibitive costs of human annotation? The authors seek to automate the generation of fine-grained feedback for intermediate reasoning steps to improve the performance of Large Language Models (LLMs) on complex multi-step mathematical problems.
The authors propose OmegaPRM, a divide-and-conquer Monte Carlo Tree Search (MCTS) algorithm. Unlike previous methods that rely on expensive human labeling or brute-force Monte Carlo estimation, OmegaPRM uses a binary search strategy to efficiently locate the first error in a Chain of Thought (CoT) sequence. By building a state-action tree, the algorithm collects over 1.5 million process supervision annotations. These annotations are used to train PRMs, which are then combined with a weighted self-consistency decoding strategy to rerank model outputs during inference.
OmegaPRM successfully automates the creation of a large-scale, high-quality dataset for process supervision. The resulting PRMs significantly boost the reasoning capabilities of existing models. For instance, the instruction-tuned Gemini Pro model saw its success rate increase from 51% to 69.4% on the MATH500 benchmark and from 86.4% to 93.6% on GSM8K. Similarly, the Gemma2 27B model showed substantial improvements, rising from 42.3% to 58.2% on MATH500 and from 74.0% to 92.2% on GSM8K. The entire pipeline functions without human intervention, offering a cost-effective alternative to manual annotation.
This work addresses a critical bottleneck in LLM reasoning: the scarcity of high-quality, step-by-step feedback. By demonstrating that automated process supervision can outperform human-labeled datasets in terms of scale and efficiency, the authors provide a scalable path toward more reliable and accurate reasoning models. This approach bridges the gap between simple outcome-based verification and labor-intensive process supervision, enabling broader application of PRMs in complex domains like mathematics and code generation.
Complex multi-step reasoning tasks, such as solving mathematical problems or generating code, remain a significant hurdle for even the most advanced large language models (LLMs). Verifying LLM outputs with an Outcome Reward Model (ORM) is a standard inference-time technique aimed at enhancing the reasoning performance of LLMs. However, this still proves insufficient for reasoning tasks with a lengthy or multi-hop reasoning chain, where the intermediate outcomes are neither properly rewarded nor penalized. Process supervision addresses this limitation by assigning intermediate rewards during the reasoning process. To date, the methods used to collect process supervision data have relied on either human annotation or per-step Monte Carlo estimation, both prohibitively expensive to scale, thus hindering the broad application of this technique. In response to this challenge, we propose a novel divide-and-conquer style Monte Carlo Tree Search (MCTS) algorithm named \textit{OmegaPRM} for the efficient collection of high-quality process supervision data. This algorithm swiftly identifies the first error in the Chain of Thought (CoT) with binary search and balances the positive and negative examples, thereby ensuring both efficiency and quality. As a result, we are able to collect over 1.5 million process supervision annotations to train Process Reward Models (PRMs). This fully automated process supervision alongside the weighted self-consistency algorithm is able to enhance LLMs' math reasoning performances. We improved the success rates of the instruction-tuned Gemini Pro model from 51\% to 69.4\% on MATH500 and from 86.4\% to 93.6\% on GSM8K. Similarly, we boosted the success rates of Gemma2 27B from 42.3\% to 58.2\% on MATH500 and from 74.0\% to 92.2\% on GSM8K. The entire process operates without any human intervention or supervision, making our method both financially and ...
Alex: Right. They used it to train what you might call a "judge" model. When the AI works through a new problem, this judge evaluates each step and gives more weight to solution paths that follow sound logic. It's steering the model away from plausible-sounding but flawed reasoning.
Sam: So the model isn't just guessing and hoping—it has a trained critic riding alongside it, nudging it toward the more trustworthy path.
Alex: That's a good way to put it. And because the whole labeling process is automated, it's also significantly more cost-effective than approaches that rely on human annotators. You get a smarter, more targeted search strategy rather than brute-force checking.
Sam: Which means it can handle a much wider range of problems—not just the ones a small team of human reviewers could get through.
Alex: Precisely. The results from the paper suggest that this kind of step-by-step supervision produces a meaningful improvement over systems that only check whether the final answer is correct. The mechanism matters: catching errors early, before they compound, turns out to be far more effective than waiting until the end.
Sam: It's a bit like the difference between a coach who corrects your technique during practice versus one who only tells you the final score after the game.
Alex: That's a fair analogy. And it points to something worth sitting with: the bottleneck in making AI systems more reliable isn't always raw computing power. Sometimes it's about building better tools for self-correction—and finding ways to do that without requiring a human expert at every step.
Sam: So the real contribution here isn't just a performance gain. It's a more scalable method for teaching a model to reason carefully.
Alex: That's the paper's central claim. Whether this approach extends cleanly beyond mathematical reasoning to other domains is still an open question—but as a proof of concept for automated process supervision, the evidence is worth paying attention to. Thanks for listening to ResearchPod.