Challenging repository-level benchmarks remain limited for many programming language ecosystems, where filtering scarce candidates with ambiguous descriptions or defective tests can discard valuable engineering problems. We introduce Longtail-SWE, a benchmark of 202 issue-resolution tasks across 13 languages with limited coverage in existing repository-level benchmarks: Clojure, Dart, Elixir, Erlang, F#, Haskell, Julia, Kotlin, Lua, OCaml, R, Scala, and Swift. Its construction uses a solver--judge--repairer pipeline: a solver attempts each candidate task, and a judge reviews the task together with the solver's trajectory and test results to guide revisions to tests and task descriptions. These revisions aim to correct tests that reject valid alternative implementations or accept incorrect solutions while retaining challenging tasks and preserving their intended requirements. We evaluate nine model configurations, including frontier models, and analyze task-solving performance, consistency across repeated attempts, and resource usage. We release all benchmark instances, executable environments, complete construction and evaluation tooling, and agent trajectories, together with an additional set of easier tasks to support evaluation of smaller models.
Alex: A coding agent can fail because the benchmark is wrong. Longtail-SWE shows how repairing flawed tests and unclear instructions can preserve scarce coding challenges and change measured performance.
Sam: If those scores help researchers choose coding agents, that sounds consequential. What kind of evidence are we talking about?
Alex: It’s a benchmark and methods preprint from Vladimir Dobrovolskii and colleagues at Nebius. Their benchmark contains 202 tasks drawn from real software issues. It covers thirteen programming languages with limited representation in existing evaluations.
Sam: So this matters both to people choosing agents and to people building the tests. Let’s separate what makes these tasks more useful from what makes their scores trustworthy.
Alex: We’ll follow how the authors recover tasks, examine their strongest controlled comparison, and ask what remains uncertain. The central distinction is simple: did the agent write bad code, or did the benchmark judge it badly?
Sam: Start with what’s being tested. Are these short programming exercises, or work inside an existing project?
Alex: Existing projects. This is repository-level evaluation: an agent gets a software issue and the codebase before its fix. It must produce a code change that resolves the issue without breaking designated existing behavior. The tasks come from public issues linked to merged pull requests.
Sam: The merged change gives the authors a known fix. Why doesn’t checking that fix against the tests settle whether the task works?
Alex: That original change is the reference patch. Showing that it passes only establishes that the tests accept that implementation. They might reject another valid implementation, or accept incorrect code because they never check an important requirement.
Sam: There’s also a specification problem. A test can demand something the agent was never told to provide.
Alex: That’s one of the defects the authors explicitly target. Hidden tests may require particular interface names, signatures, or exact outputs absent from the instructions. An interface is how other code accesses a component. Unless the requirement is available in the task context, failing that check need not demonstrate poor engineering.
Sam: Previous benchmarks already screened for unclear issues and unsuitable tests. What changes here beyond doing more screening?
Alex: The prior SWE-rebench V2 approach retained candidates classified as well specified. Longtail-SWE also keeps categories judged potentially repairable. That matters when suitable issues are scarce: throwing away every imperfect task can discard difficult engineering work and favor smaller changes.
Sam: So the new contribution isn’t simply adding more languages. It’s treating some benchmark defects as repairable rather than disqualifying. How does that repair happen?
Alex: The authors call it a solver–judge–repairer pipeline. A solver agent attempts the issue. A judge examines its actions, code changes, and test results alongside the instructions and reference fix. A separate repair agent then addresses defects the judge identifies.
Sam: Take that example of an undocumented interface name. What can the repairer change without quietly giving away the solution?
Alex: It can append a clarification of expected behavior, while keeping the reference implementation undisclosed. Or it can revise tests that enforce incidental implementation choices rather than actual requirements. It can also strengthen tests when incorrect code passes. The goal runs both ways: accept valid alternatives and reject genuine mistakes.
Sam: That still gives the repairer considerable power over the exam. What prevents it from making a troublesome task easy?
Alex: The reference patch stays unchanged, and repairs must preserve the intended requirements. Revised tests must still expose a failure before that fix and pass afterward. The repairer cannot slip production-code fixes into the test changes, and acceptance requires agreement from several judge models.
Sam: If the instructions change, an old failed attempt no longer tells you how the agent would respond. Do they rerun it?
Alex: They do. Changed instructions require new solver attempts. Test-only changes allow existing code patches to be reevaluated. They also rebuild and validate executable environments, then conduct another audit after evaluation. That later review still removes tasks that cannot be repaired without changing their intent.
Sam: Let’s distinguish recovering more tasks from proving those tasks are better. What’s the clearest evidence for each?
Alex: For recovery, only sixty-eight tasks in the final benchmark would remain available through the original construction path. The expanded screening and repair process therefore retains substantially more usable candidates. But that comparison combines several changes; it doesn’t isolate the contribution of each stage.
Sam: And a higher success rate after repair could mean better judging, or just easier judging. Is there a comparison that holds the agent’s work fixed?
Alex: Yes. On a subset whose descriptions stayed unchanged, the authors scored the same Opus patches against the old and revised tests. Average single-attempt success rose from about twenty-nine percent to about sixty-seven percent. The code had not improved; the evaluator had changed.
Sam: That isolates why the score moved. It doesn’t, by itself, establish that the new tests are correct.
Alex: Correctness still depends on the review and validation behind those revisions. My reading is that this comparison strongly demonstrates sensitivity to test design, not independent proof that every revised verdict is right. It’s also limited to the unchanged-description subset, so it cannot validate description repairs.
Sam: How well did the automated judges agree with people inspecting these defects?
Alex: The authors audited sixty-five tasks sampled across languages and pipeline stages. Engineers first assessed them independently, then reconciled disagreements. The judge protocol caught nearly all the tasks their consensus identified as defective, but initial human agreement was low.
Sam: Catching defects is useful, but overcalling them could send acceptable tasks into needless repair. Did that happen?
Alex: The protocol rejected half the tasks humans ultimately considered acceptable. That adds construction cost and shows how difficult these judgments are. Most importantly, this audit does not establish the error rate of the final released benchmark. The authors explicitly call for an independent audit of that set.
Sam: What do the resulting agent scores tell us, once we keep that uncertainty in view?
Alex: Opus had the highest observed single-attempt success, roughly four in five. Repeated attempts improved coverage, but did not solve everything. Performance also varied across language subsets, and the authors caution against reading those differences as inherent language difficulty.
Sam: There’s another possible dependency: models helped construct the benchmark and then were evaluated on it. How much should that qualify the ranking?
Alex: Some evaluated models also participated in construction, which may favor their interpretations of tasks. Difficulty selection also used Qwen attempts, potentially biasing that model’s reported results, despite using fresh attempts for evaluation. And public issues may overlap training data. These are configuration-specific results, not a clean measure of universal coding ability.
Sam: Given those qualifications, who should read the full document, and where should they begin?
Alex: Benchmark builders should start with Solver-Guided Task Diagnosis and Repair, then the pre-versus-post-repair comparison. That pairing shows both the procedure and its effect on identical submissions. Coding-agent evaluators should read Experimental Setup alongside Threats to Validity before interpreting rankings. The release also includes environments, tooling, and agent histories for closer inspection.
Sam: And for researchers who won’t build a coding benchmark, what’s the line to remember?
Alex: A failed test can expose a bad solution—or a bad test. Before trusting an agent’s score, ask how the benchmark learned to tell the difference.
Sam: That’s a question to carry into the next evaluation you read.