Coding Agents Have Converged: Why the SWE-bench Leaderboard Can No Longer Order Its Top Entries, and What to Measure Instead

Coding agent leaderboards, particularly SWE-bench, are frequently used to rank systems for model selection and research. However, this paper demonstrates that the top-performing agents have reached a state of convergence. By auditing 254 submissions, the authors show that the top ten agents share 285 successful resolutions and 51 failures out of 500 instances, leaving only 164 instances that actually distinguish their performance. This high degree of overlap means that the small differences in aggregate scores are often statistically insignificant.

Beyond the lack of resolution, the authors find that a leaderboard score is not a pure reflection of the underlying model. Instead, it is a product of the model combined with its scaffold—the specific agent loop, tool set, and retrieval policy. The study reveals that within-model scaffold variation can reach 29.8 percentage points, which is significantly larger than the 8.8-point spread observed among the top thirty entries. Furthermore, the relative ranking of scaffolds often reverses depending on the model being used, suggesting that the leaderboard is measuring a complex, confounded interaction rather than a stable capability of the model alone.

To address these issues, the authors propose a five-step audit protocol that shifts the focus from strict rankings to descriptive tiers. Instead of interpreting small score gaps as meaningful differences, researchers should use paired statistical tests to group systems into tiers that are actually distinguishable. The authors also argue for recording model-scaffold provenance in machine-readable formats and curating new benchmark instances based on their ability to break the observed nesting of solution sets, rather than simply increasing the total number of tasks.