Who Thinks Best Depends on How Long You Let Them: Budget-Dependent Rankings in LLM Evaluation

Standard LLM evaluation typically assumes that model rankings are stable regardless of the inference-time constraints. This paper investigates whether this assumption holds by systematically varying the token generation budget (the maximum number of tokens a model is allowed to produce) across seven levels, ranging from 64 to 4,096 tokens.

The authors evaluated four open-weight models (ranging from 8B to 70B parameters) on three reasoning benchmarks: GSM8K, MATH-500, and GPQA. They conducted over 56,000 inferences using greedy decoding to ensure determinism. The analysis focused on three areas: classifying how model accuracy changes with budget (the behavioral taxonomy), testing for statistically significant ranking reversals, and evaluating the potential for a budget-aware router to select the best model for a given query.

The study reports four primary results. First, performance is not always monotonic; up to 19% of items show a 'non-monotone' pattern where accuracy decreases as the budget increases, a phenomenon the authors identify as model-specific 'overthinking.' Second, model rankings are unstable, with the 'best' model changing significantly across different budget levels on all benchmarks. Third, an oracle analysis shows that models are highly complementary, particularly at constrained budgets, where an ensemble could outperform the best single model by up to 27.8 percentage points. Finally, the authors demonstrate that a budget-aware routing system can capture a portion of this oracle gap, though the effectiveness of these routing features is largely domain-specific.

These findings challenge the current practice of using static leaderboards to determine model superiority. If a model's rank depends on the compute budget, then 'state-of-the-art' status is context-dependent rather than an intrinsic property. The results suggest that future evaluation protocols must explicitly account for token budgets to provide a realistic assessment of model capabilities in deployment scenarios where compute constraints vary.