Lovish Madaan, Aniket Didolkar, Suchin Gururangan, John Quan, Ruan Silva, Ruslan Salakhutdinov, Manzil Zaheer, Sanjeev Arora, Anirudh Goyal
11 min
Abstract
Reasoning training incentivizes LLMs to produce long chains of thought (long CoT), which among other things, allows them to explore solution strategies with self-checking. This results in higher accuracy, but inflates context length, token/compute cost, and answer latency. We ask: Can current models leverage their metacognition to provide other combinations on this Pareto frontier, e.g., better accuracy with lower context length and/or latency? Abstractly, we view the model as an improvement operator on its own "thoughts" with a continuum of possible strategies. We identify an interesting inference family Parallel-Distill-Refine (PDR), which performs the following: (i) generate diverse drafts in parallel; (ii) distill them into a bounded, textual workspace; and (iii) refine conditioned on this workspace, producing an output that seeds the next round. Importantly, context length (hence compute cost) is controllable via degree of parallelism, and is no longer conflated with the total number of generated tokens. We report PDR instantiations of current models that give better accuracy than long CoT while incurring lower latency. Setting degree of parallelism to 1 yields an interesting subcase, Sequential Refinement (SR) (iteratively improve a single candidate answer) which provides performance superior to long CoT. Success of such model orchestrations raises the question whether further training could shift the Pareto frontier. To this end, we train an 8B thinking model with Reinforcement Learning (RL) to make it consistent with PDR as the inference method. On math tasks with verifiable answers, iterative pipelines surpass single-pass baselines at matched sequential budgets, with PDR delivering the largest gains (e.g., +11% on AIME 2024 and +9% on AIME 2025).
Alex: Okay, but how do they make sure the model actually gets good at this iterative stuff? Doesn't training usually focus on one long ramble?
Sam: Great question—that's where they tackle a sneaky train-test mismatch. Imagine practicing basketball by watching full games on tape, but playing one-on-one in real matches; you'd fumble the transitions. Standard RL trains on long chains with rewards at the end, but PDR uses short bursts: generate, summarize, refine. So they invented operator-consistent RL to fix it—unrolling the exact PDR process during training, sampling short drafts, distilling a compact summary, then refining based on that, all with verifiable rewards. It's like drilling the specific play patterns you'll use in the game.
Alex: So, does that mean the model learns these meta-skills you mentioned earlier, like verifying errors or compressing info?
Sam: Exactly. This approach teaches verification—spotting mistakes by cross-checking drafts—refinement using feedback, compression to keep summaries tight, and diversification to avoid groupthink. They mixed it with standard RL on an 8B model, netting about 5% gains on AIME tasks while keeping long CoT skills intact. The paper suggests this alignment shifts the Pareto frontier, making iterative methods more efficient overall.
Alex: Fascinating—sounds like it could scale to other reasoning tasks beyond math.
Sam: Yeah, it could extend to puzzles, coding, even scientific reasoning—anywhere you need step-by-step problem-solving without the AI getting bogged down.
Alex: But let's dig into how PDR actually works in practice. You mentioned parallelism and distillation, but walk me through a real example, like on a math problem. Why not just keep refining one idea sequentially?
Sam: Fair point—sequential refinement, or SR, is a simpler version of PDR where you stick to one draft and improve it round by round without any workspace, basically like editing a single essay over and over. It beats long chains of thought by keeping contexts short, but it can get stuck in ruts, like fixating on one wrong approach. PDR fixes that by adding parallelism: imagine tackling a math problem, say proving a geometry theorem. In the first round, the AI generates, say, three diverse solution drafts at once—each exploring a different angle, like using coordinates versus vectors. That's the parallel step, sparking broader exploration without bloating a single chain.
Alex: Okay, so multiple paths at once. But how do you avoid a mess of conflicting ideas?
Sam: That's where distillation comes in. The model then creates a compact textual workspace—a short summary, limited to a fixed number of tokens, that pulls out the agreements, like "All drafts agree the angle is 90 degrees," contradictions, such as "One uses area, another perimeter—resolve which fits," and open subgoals, like "Verify the hypotenuse." This isn't replaying everything; it's a fresh, bounded note that retires outdated info and flags uncertainties. Finally, it refines by generating the next set of drafts based on that summary, keeping each round's context short and focused.
Alex: So it's like a team meeting: brainstorm ideas, condense into bullet points, then build on those—without the full transcript dragging everyone down.
Sam: Exactly. This read-write-compress cycle decouples total compute— all those parallel drafts add up in the background—from the sequential budget, which is just the tokens along the main path users see, proxying for wait time. On math tasks, it dodges long-context forgetfulness and anchoring biases, where early mistakes snowball.
Alex: Got it—smarter iteration without the endless ramble. Does SR have any edge, or is PDR always better?
Sam: SR shines when latency is ultra-tight since there's no parallel overhead, but PDR pulls ahead on accuracy by injecting diversity, especially as rounds increase. The paper shows both shift the Pareto frontier, trading off cost for better results under matched budgets.
Alex: So if PDR edges out SR on accuracy thanks to that diversity kick, how do we know it's not just burning more compute? Like, does it really deliver at the same overall cost?
Sam: That's a sharp question—it's all about those budgets we keep circling back to. Picture this: you're on a road trip with a gas limit. Long CoT is like driving straight through, guzzling fuel on a single winding path that might hit detours or forget the map halfway. SR is smarter, stopping to tweak the route in short bursts, but it might loop back on the same bad turn. PDR, though, scouts multiple side roads at once in the background—using extra gas there—but the main highway you travel stays efficient and short. The sequential budget, or B_seq, is that main path's length: tokens in the accepted chain plus any quick summary notes, acting as a stand-in for wait time since users only see the streamlined steps. Total budget, B_total, counts everything, including those discarded scout paths, for overall compute cost.
Alex: Okay, so B_seq proxies latency—fair. But in tests, does PDR actually shift things so you get more bang without exploding the bill?
Sam: Yeah, the experiments nail this on AIME math problems, using models like o3-mini and gemini-2.5-flash. They compared SR and PDR to long CoT at matched B_seq levels, from about 8K to 24K tokens. Both iterative methods beat the single long ramble on accuracy, but PDR pulls further ahead by channeling parallel compute into better exploration without bloating per-step waits. For instance, at comparable latency proxies, PDR converts that background work into higher scores, dodging forgetfulness in long contexts. It's like upgrading your trip: same drive time, but smarter planning gets you there fresher and faster overall.
Alex: Wait, so practically, for building AI tools, this means we could swap endless thinking traces for these quick team huddles and cut user frustration?
Sam: Spot on—it suggests iterative setups like PDR could make reasoning AI snappier for real apps, trading smart parallelism for reliability without the old pitfalls.
Alex: But if PDR is so efficient, how do they pick which drafts to carry forward in that distillation step? Does the model just guess, or is there a smart way to avoid carrying junk?
Sam: That's a key detail—they test different distillation strategies to see what synthesizes the best workspace without bloating it. Imagine your team brainstorming: you don't just toss all ideas into the meeting notes; you filter for the strongest ones to keep everyone focused. The paper compares a few approaches, like random-k, where they simply sample a few drafts at random to forward, or top-k, where the model itself scores and picks the best ones using a grading prompt—either a shared rubric across all or tailored per problem. There's also a global-summary that condenses everything into one tight overview, highlighting agreements and contradictions.