The evaluation of coding agents generally strives to be as realistic as possible. In our study, we collect 4,782 agent sessions of real software engineers in JetBrains IDEs, which we call Production Sessions. Since our subject is interactive agents, we study the sessions with at least three user messages (33% of the sample). These long sessions differ from issue-derived benchmark tasks in two ways: (i) user requests span a far wider mix of task types - questions about the project's code, planning, review, refactoring, execution - and (ii) users switch between types throughout a session. Long-session samples from three public interaction corpora exhibit markedly different Task Flows (the distributions of session lengths, task types, and type-to-type transitions), so no single interaction distribution is universally realistic: benchmarks should name a target use case and calibrate to measurements from it. We present SWE-TaskFlow, an approach for transforming any issue-derived benchmark: it preserves the verified tasks and tests while steering the interaction toward a target Task Flow through prompt splitting and verifiable repository QA, with a TaskFlow Alignment Score (TFAS) for selecting among generated trajectories. In a pilot on 700 SWE-Bench Pro tasks, solving the task sequentially in several steps approximately doubles agent cost without a stable change in resolve rate: the interaction protocol itself is an important dimension of evaluation.
Sam: Real developers don't talk to coding agents the way benchmarks do. They ask, plan, change their minds, and ask again, while benchmarks hand over one polished issue and wait for a patch. The argument here is that a benchmark should copy how a specific group of users actually works, because no single version of "realistic" exists.
Alex: Before we take that on, place it for me. Whose work is this, and what kind of paper are we talking about?
Sam: It's from JetBrains Research, a short opinion paper for a NeurIPS workshop on evaluating interactive agents. It's a position piece, but it comes with real usage data, a method for transforming existing benchmarks, and a small pilot experiment.
Alex: Plenty of people choose models by benchmark scores. Why should someone outside agent evaluation care about how the user messages are arranged?
Sam: Because the arrangement turns out to change what gets measured. In their pilot, delivering the same task in pieces roughly doubled what the agent spent. So we'll cover what real sessions look like, how they rebuild a benchmark to match them, and how far to trust the results.
Alex: Start with the real sessions, then. What did they actually look at?
Sam: They collected 4,782 agent sessions from engineers working in JetBrains IDEs, the code editors, in everyday use. They call these Production Sessions. Only the user's side was analysed, roughly fourteen thousand messages, as anonymized aggregates. About half the sessions are a single message. But about one third have at least three, and since the subject is interactive agents, that long slice is what they study.
Alex: And how do they compare those sessions to a benchmark? The two look very different on the surface.
Sam: They use one idea they call a Task Flow. It has three parts: how long sessions run, what type of request each message is, and how one type tends to follow another. That last part keeps the order, which a simple count of labels would throw away. An open-weight language model labels every message into one of sixteen types, like bug fix, new feature, explain, planning, code review, or run a command.
Alex: So the benchmark side gets labelled the same way. What's the gap?
Sam: The comparison is SWE-Bench Pro, an issue-derived benchmark, meaning tasks built from real repository issues and scored by tests. There, new features and bug fixes together make up about ninety-four percent of tasks. In the long production sessions, those two cover well under half the messages. The rest is people asking how their own code works, general programming questions, planning, review, refactoring, running things.
Alex: And the second gap is the switching, I'd guess. A benchmark task is one message, so there's nothing to switch between.
Sam: Right, and that's their second observation. A session that opens with a feature request often moves to a question about existing behaviour, then a planning step, then a review request. The single-issue format measures building from an upfront spec, not adapting as requirements unfold.
Alex: Couldn't someone just say, fine, measure real usage once and make that the standard benchmark?
Sam: That's exactly what they argue against. They ran the same labelling on three public corpora of coding-agent conversations, a thousand long sessions each. All three differ from the production data and from each other. One is dominated by running commands and short operational turns. Another leans on execution and planning. The closest one still has more planning. Their conclusion: a distribution measured on one product doesn't transfer to another. So you name your target users and measure them.
Alex: Then here's the practical worry. Building a new interactive benchmark for every user group sounds expensive. How do they avoid that?
Sam: They don't build new tasks. Their approach, called SWE-TaskFlow, keeps the repository, the requested change, the environment, and the final tests fixed. Only the user messages get regenerated. They do that two ways. The first is prompt splitting: one polished issue becomes three ordered user turns, because real users reveal requirements gradually.
Alex: Rewriting an issue with a model could quietly drop or add a requirement. How do they guard against that?
Sam: The splitter has a contract. It must keep every original requirement, add none, keep technical details word for word, and make each turn workable from the state earlier turns left behind. Issues that can't be split cleanly stay single-turn. Seven hundred of the public tasks survived. They also ran a control called Concat, which sends the three rewritten pieces as one message. That separates the effect of rewriting from the effect of sequential delivery.
Alex: And the second augmentation?
Sam: Repository questions, to inject the explain requests benchmarks lack. A model reads a past trajectory that solved the issue, plus the live code, and proposes up to three questions about how the code behaves. Each needs a hidden reference answer, must stay valid before and after the fix, and must not leak the solution. And each has to pass a proof script, actually run on a clean copy of the repository. So a question about the rationale for a design choice gets excluded, because there's nothing observable to check.
Alex: So the pieces are splits and questions. How do they decide where the questions go?
Sam: With an alignment score, the TaskFlow Alignment Score. For each of the three parts, lengths, types and transitions, they measure how far the benchmark is from the target, using Jensen-Shannon divergence, a bounded distance between distributions. They combine the three into one score out of a hundred. Then a search picks how many questions each task keeps and where they sit. Pure splitting scores about sixty. The optimized version reaches about seventy-eight, mostly by improving the transitions.
Alex: Moving questions around to raise a score could make tasks harder or easier without anyone noticing. Did they check?
Sam: They did, on a smaller set of sixty-four tasks. Putting all questions first, all last, or randomly interleaved showed no consistent effect on resolve rate or answer accuracy across models. Differences were comparable to run-to-run variation. That's their licence to let the optimizer move questions freely, within the range they tested.
Alex: Now the headline. On the seven hundred tasks, what happened when agents got the split version?
Sam: Four models, one common agent scaffold. Splitting approximately doubled agent cost. Resolve rate moved in mixed directions across models, largely within run-to-run noise. So the authors say the runs establish no ranking of success. What they do show is that sequential delivery reliably changes how much effort the agent spends. Adding the calibrated questions added moderate cost on top.
Alex: And the questions themselves give you a second score, I assume, beyond pass or fail on the patch.
Sam: Yes. Answer accuracy ranged from about seventy to eighty-three percent across the four models. And a split trajectory can be checked after each turn, against only the requirements revealed so far. Those intermediate results climbed steadily as turns accumulated. So one task yields several verified signals instead of one terminal verdict.
Alex: How much weight should we put on all this?
Sam: Treat the resolve-rate findings as directional. Each condition ran three times, with min-to-max ranges, no confidence intervals and no significance tests, as the authors themselves say. The bigger caveat for me is the labelling. Everything rests on a model classifying intent, and there's no human-labelled accuracy set. Three different models agreed on about seventy-six percent of labels. That's robustness, not accuracy. And production sessions are just one use case.
Alex: And presumably a lot of what real users do still isn't covered.
Sam: Most of it, by their own account. Planning, code review, refactoring, debugging without a fix, general questions, they all still need a way to generate the turn and verify the answer. It's unclear how to grade a planning turn automatically. They also note the fixed-message design can't capture feedback that reacts to what the agent did. They see reactive benchmarks with simulated users as a complement.
Alex: So who should actually sit down with the full paper, and where should they start?
Sam: Anyone building or choosing benchmarks for coding agents. Start with the appendix on calibrating to a new target. It's a step-by-step recipe: name your users, measure their flow, find the gaps, and report rare but risky intents as separate stress tests. Then read the pilot, especially the Concat control, which separates rewriting from delivering in pieces.
Alex: And for everyone else, the one line to carry around?
Sam: There's no realistic benchmark in general. There's only realistic for whom, and the benchmark should say who.
Alex: Worth asking of the next leaderboard you check.