Sample More, Reflect Less: Self-Refine and Reflexion Lose to Repeated Sampling at Equal Token Cost, from 1.5B to 7B

Many modern techniques aim to improve language model reasoning without retraining, such as asking the model to plan, critique its own work, reflect on mistakes, or debate with itself. However, these methods inherently increase the number of tokens generated per question. This study investigates whether the observed performance gains from these methods are truly due to the proposed reasoning strategies or simply a byproduct of the increased compute budget.

The author conducts a controlled experiment comparing seven reasoning methods against a baseline of simple repeated sampling (self-consistency). The study uses Qwen2.5 models (1.5B, 3B, and 7B parameters) on two mathematics benchmarks (GSM8K and MATH-500). Crucially, the study measures the exact token cost of every method—including critiques, reflections, and debate turns—and compares each against the sampling baseline at an identical token budget. All comparisons are paired by question and include statistical significance testing with multiplicity correction.

No method consistently beats the simple sampling baseline at an equal token cost. In fact, ten of the tested configurations were found to be reliably worse. The study identifies a clear divide: methods that involve self-inspection (critiquing or rewriting) consistently underperform compared to simple majority voting. While "Best-of-N" selection (where the model picks the best of several samples) was significantly worse than majority voting at 1.5B parameters, this performance gap narrowed as the model size increased to 7B, where the difference became statistically indistinguishable from zero. Furthermore, the study notes that the original Reflexion method often failed to trigger its own retry mechanism on smaller models, effectively collapsing into a single chain of thought.

These findings suggest that much of the perceived benefit of complex test-time reasoning strategies may be an artifact of increased compute rather than the sophistication of the methods themselves. For practitioners, this implies that simply sampling more and taking the majority vote is a more efficient and reliable way to boost performance than implementing complex, self-reflective prompting pipelines, particularly for smaller models.