Ibragim Badertdinov, Alexander Golubev
7 min
Abstract
Standard coding-agent benchmarks primarily evaluate binary task success, treating execution as a black box and obscuring the behaviors that produce those outcomes. We analyze a large corpus of 143,209 execution trajectories from 125 agent configurations spanning 117 unique models. To enable comparisons across heterogeneous agent interfaces, we develop a rule-based parser that maps raw execution traces to high-level actions such as repository search, file editing, and test execution, and audit its predictions against a six-model consensus reference on 500 events. We find that agents with similar solve rates can nevertheless differ substantially in how they explore, modify and verify code. Moreover, without using task outcomes, token counts, or tool names as features, aggregate behavioral profiles retrieve the corresponding agent stack across disjoint repository sets and programming languages, although task composition is not fully controlled in the cross-repository setting. Through analyses of frontier models and agent scaffolds, we further show that scaffold choice and reasoning effort systematically shift agents' search, editing, and testing behavior. We release the deterministic parser together with the full annotation and analysis pipeline alongside an annotated trajectory subset.
Alex: They audited five hundred actions from held-out runs after finalizing the parser. The reference was agreement among a panel of leading language models, not human annotations. For the main action category, parser agreement was about eighty-two percent.
Sam: That leaves meaningful disagreement, and the reference models could share mistakes. Which findings deserve the most caution?
Alex: Agreement was lower for file roles and kinds of modification. The authors therefore treat measures such as non-test file writes as diagnostic proxies, not exhaustive filesystem facts. The audit also checked isolated labels, not the validity of whole sequences, such as repeated failures or trailing actions.
Sam: With that measurement uncertainty in mind, what is the strongest comparison?
Alex: The outcome-matched analysis. They selected sixty pairs using the same scaffold, meaning the software harness that provides tools and structures interaction. Within each pair, the underlying models differed, but their aggregate solve rates were nearly identical on shared tasks.
Sam: Holding the harness fixed helps separate model differences from interface differences. What did the workflows show?
Alex: Almost ninety-seven percent of the pairs differed on several workflow measures at once. Those measures included total attempted actions, the share devoted to exploration, and formal-test invocations. So the distinction wasn’t just an isolated difference in one obscure command category.
Sam: Repeated runs can diverge substantially even for the same model. How much separation would they get just by comparing a system with itself?
Alex: They split repeated runs of identical systems into independent groups and repeated that comparison. Separation across the same number of workflow measures occurred in less than one percent of those same-system comparisons. That makes ordinary run-to-run variation an unlikely explanation for the much broader differences between models.
Sam: Equal overall success still doesn’t mean they solved the same tasks. Could task difficulty explain some of this?
Alex: It could, and the authors explicitly acknowledge that. Matching each task’s outcome would be stricter. But selected case studies also show different editing and verification patterns on tasks where both agents succeeded in every repeated run.
Sam: What did those differences look like beyond the number of commands?
Alex: In one example, Opus continued after its final edit with a targeted test, a full regression check, and workspace inspection. Sol passed a targeted test and submitted. That illustrates different stopping behavior in those runs; it doesn’t establish that either policy is generally better.
Sam: The harness could encourage those habits too. Did they examine what happens when the model stays the same but its surroundings change?
Alex: They did. Under Claude Code, the displayed Opus configuration relied heavily on native reading and editing tools. Under the generic harness, it used more shell-level mechanisms. That’s why the paper talks about an agent stack: the model, harness, reasoning-effort setting, and decoding settings together.
Sam: And when they changed the configured reasoning effort, did the extra activity mostly become extra testing?
Alex: Not in their detailed Claude Code comparison. Higher effort mainly brought more exploration and reading, alongside more workspace inspection after editing. Formal-test volume showed no clear increase, and neither did the measured unproductive tail—the actions after the last counted write or passing test.
Sam: So more configured effort changed the external workflow, but we still can’t infer what happened inside the model.
Alex: The authors make that boundary explicit. They did not inspect internal reasoning traces. These are observations of tool use and environment interaction, not direct measurements of thought.
Sam: They also ask whether behavior can identify a system elsewhere. How does that test work?
Alex: They summarize each stack’s action mix and sequence patterns, then match profiles across separate code repositories. The matching features exclude task success, token volume, total action counts, and native tool names. The full profile identified the exact stack in about eighty-five percent of queries, far above chance.
Sam: If different stacks encountered different kinds of tasks, that could help identify them. Does a repository split remove that problem?
Alex: No. The authors say task composition remains a possible contributor, so this does not establish task-independent signatures. A separate programming-language analysis, using a different profile, reached about ninety percent correct matching. Much of that signal came from action composition, not richer sequence dynamics.
Sam: That supports a recognizable operational style in these evaluations, rather than a fingerprint of deep problem-solving strategy. And all the runs still come from one benchmark regime.
Alex: That’s the right scope. The profiles may partly reflect task familiarity or benchmark-specific policies. The study supports behavior as a complement to success, not a replacement for it, and not a proven guide to which workflow is cheaper or better.
Sam: Who should read the full document, and where would you send them first?
Alex: Researchers building coding agents or evaluation harnesses should start with the outcome-matched comparisons and the same-stack null control. Then read the parser audit and limitations before adopting the metrics. If behavioral identification is your interest, read the retrieval protocol closely, especially the task-composition caveat.
Sam: And what should everyone else carry away?
Alex: Matching success does not mean matching behavior. To understand a coding agent, look at the route as well as the result.
Sam: Thanks for listening.