Standard coding-agent benchmarks primarily evaluate binary task success, treating execution as a black box and obscuring the behaviors that produce those outcomes. We analyze a large corpus of 143,209 execution trajectories from 125 agent configurations spanning 117 unique models. To enable comparisons across heterogeneous agent interfaces, we develop a rule-based parser that maps raw execution traces to high-level actions such as repository search, file editing, and test execution, and audit its predictions against a six-model consensus reference on 500 events. We find that agents with similar solve rates can nevertheless differ substantially in how they explore, modify and verify code. Moreover, without using task outcomes, token counts, or tool names as features, aggregate behavioral profiles retrieve the corresponding agent stack across disjoint repository sets and programming languages, although task composition is not fully controlled in the cross-repository setting. Through analyses of frontier models and agent scaffolds, we further show that scaffold choice and reasoning effort systematically shift agents' search, editing, and testing behavior. We release the deterministic parser together with the full annotation and analysis pipeline alongside an annotated trajectory subset.
Alex: Two coding agents can solve equally many problems and still work very differently. This study finds reproducible differences in how they search, edit, and check their code.
Sam: So a success ranking can hide different working habits. Whose study is this, and what did they examine?
Alex: It’s a methods-and-analysis preprint by Ibragim Badertdinov and Alexander Golubev at Nebius. They examined about a hundred and forty-three thousand recorded coding-agent runs. Their central claim is that success rates leave out a useful dimension of evaluation: observable behavior.
Sam: That matters if you build these systems or compare them for research. You want to know not just whether they succeed, but what their workflow involves.
Alex: We’ll unpack how the authors made different systems comparable, why they trust the differences, and where interpretation should stop.
Sam: Researchers already inspect agent logs. What does this add beyond reading a few successful and unsuccessful runs?
Alex: Earlier work has developed workflow labels, process representations, and failure diagnostics. This study adds scale and comparisons across deployed systems. Crucially, it compares agents with nearly equal success rates, then asks whether their behavioral differences exceed ordinary variation between runs.
Sam: But different systems expose different tools. How do you avoid mistaking the interface for the behavior?
Alex: That’s the central measurement problem. An execution trajectory is the recorded sequence of tool calls and environment responses. The authors build a deterministic parser: fixed rules that translate those records into shared action categories.
Sam: Give me an example of what those rules preserve and what they set aside.
Alex: Reading a file counts as reading, whether it happens through a shell command or a dedicated file tool. The parser also distinguishes searching, modifying files, running formal tests, and inspecting workspace changes. It records additional details, including whether a modification targets a conventional test file.
Sam: So the comparison moves from button names to actions. But a single command can still contain a lot of work.
Alex: Yes, and that matters when interpreting action counts. In a selected task, both models succeeded on every repeated run. Claude Opus used many bounded searches and reads, while GPT Sol used broader commands across fewer interactions. A shorter trace didn’t mean identical exploration, just compressed into fewer steps.
Sam: That makes the labeling rules central to the result. How did the authors check them?
Alex: They audited five hundred actions from held-out runs after finalizing the parser. The reference was agreement among a panel of leading language models, not human annotations. For the main action category, parser agreement was about eighty-two percent.
Sam: That leaves meaningful disagreement, and the reference models could share mistakes. Which findings deserve the most caution?
Alex: Agreement was lower for file roles and kinds of modification. The authors therefore treat measures such as non-test file writes as diagnostic proxies, not exhaustive filesystem facts. The audit also checked isolated labels, not the validity of whole sequences, such as repeated failures or trailing actions.
Sam: With that measurement uncertainty in mind, what is the strongest comparison?
Alex: The outcome-matched analysis. They selected sixty pairs using the same scaffold, meaning the software harness that provides tools and structures interaction. Within each pair, the underlying models differed, but their aggregate solve rates were nearly identical on shared tasks.
Sam: Holding the harness fixed helps separate model differences from interface differences. What did the workflows show?
Alex: Almost ninety-seven percent of the pairs differed on several workflow measures at once. Those measures included total attempted actions, the share devoted to exploration, and formal-test invocations. So the distinction wasn’t just an isolated difference in one obscure command category.
Sam: Repeated runs can diverge substantially even for the same model. How much separation would they get just by comparing a system with itself?
Alex: They split repeated runs of identical systems into independent groups and repeated that comparison. Separation across the same number of workflow measures occurred in less than one percent of those same-system comparisons. That makes ordinary run-to-run variation an unlikely explanation for the much broader differences between models.
Sam: Equal overall success still doesn’t mean they solved the same tasks. Could task difficulty explain some of this?
Alex: It could, and the authors explicitly acknowledge that. Matching each task’s outcome would be stricter. But selected case studies also show different editing and verification patterns on tasks where both agents succeeded in every repeated run.
Sam: What did those differences look like beyond the number of commands?
Alex: In one example, Opus continued after its final edit with a targeted test, a full regression check, and workspace inspection. Sol passed a targeted test and submitted. That illustrates different stopping behavior in those runs; it doesn’t establish that either policy is generally better.
Sam: The harness could encourage those habits too. Did they examine what happens when the model stays the same but its surroundings change?
Alex: They did. Under Claude Code, the displayed Opus configuration relied heavily on native reading and editing tools. Under the generic harness, it used more shell-level mechanisms. That’s why the paper talks about an agent stack: the model, harness, reasoning-effort setting, and decoding settings together.
Sam: And when they changed the configured reasoning effort, did the extra activity mostly become extra testing?
Alex: Not in their detailed Claude Code comparison. Higher effort mainly brought more exploration and reading, alongside more workspace inspection after editing. Formal-test volume showed no clear increase, and neither did the measured unproductive tail—the actions after the last counted write or passing test.
Sam: So more configured effort changed the external workflow, but we still can’t infer what happened inside the model.
Alex: The authors make that boundary explicit. They did not inspect internal reasoning traces. These are observations of tool use and environment interaction, not direct measurements of thought.
Sam: They also ask whether behavior can identify a system elsewhere. How does that test work?
Alex: They summarize each stack’s action mix and sequence patterns, then match profiles across separate code repositories. The matching features exclude task success, token volume, total action counts, and native tool names. The full profile identified the exact stack in about eighty-five percent of queries, far above chance.
Sam: If different stacks encountered different kinds of tasks, that could help identify them. Does a repository split remove that problem?
Alex: No. The authors say task composition remains a possible contributor, so this does not establish task-independent signatures. A separate programming-language analysis, using a different profile, reached about ninety percent correct matching. Much of that signal came from action composition, not richer sequence dynamics.
Sam: That supports a recognizable operational style in these evaluations, rather than a fingerprint of deep problem-solving strategy. And all the runs still come from one benchmark regime.
Alex: That’s the right scope. The profiles may partly reflect task familiarity or benchmark-specific policies. The study supports behavior as a complement to success, not a replacement for it, and not a proven guide to which workflow is cheaper or better.
Sam: Who should read the full document, and where would you send them first?
Alex: Researchers building coding agents or evaluation harnesses should start with the outcome-matched comparisons and the same-stack null control. Then read the parser audit and limitations before adopting the metrics. If behavioral identification is your interest, read the retrieval protocol closely, especially the task-composition caveat.
Sam: And what should everyone else carry away?
Alex: Matching success does not mean matching behavior. To understand a coding agent, look at the route as well as the result.
Sam: Thanks for listening.