When Does Restricting a Coding Agent to execute_code Help? A Regime× Agent-Design Ablation

Coding agents currently offer a variety of tool surfaces, ranging from specialized IDE primitives (like Edit and Write) to raw bash access and generic code-execution interfaces. Despite the proliferation of these tools, there is no consensus on which surface is most efficient. This paper investigates whether restricting an agent to a single code-execution tool (execute_code) improves performance or reduces costs compared to default, tool-rich environments, and how this choice interacts with different task types and agent designs.

To resolve conflicting claims in the field, the authors conducted a controlled, three-arm ablation study (baseline, bash_only, and code_only) across two distinct task regimes: synthetic computation tasks and SWE-bench Mini modification tasks. They evaluated two specific agents—Claude Code and OpenAI Codex CLI—while keeping the harness, prompts, and models fixed. The study uses a cache-adjusted cost methodology to account for vendor-side caching, ensuring that cost comparisons reflect the actual efficiency of the tool surface rather than artifacts of the caching system.

The study reveals that the "best" tool surface is jointly determined by the task regime and the agent design. In three out of four tested scenarios, restricting the agent to a single code-execution tool was either significantly cheaper than or statistically tied with the cheapest tool-rich alternative. The exception was the Claude agent on SWE-bench tasks, where the code-only approach was directionally more expensive due to increased edit friction on failed runs. Crucially, the authors found that pass rates remained invariant across all tool surfaces, indicating that tool choice primarily impacts the path taken and the associated costs, rather than the agent's ability to solve the task.

This research provides a clear framework for "cost-engineering" coding agents. By demonstrating that tool-rich interfaces do not necessarily improve success rates but do significantly impact operational costs, the authors provide a roadmap for developers to optimize their agent stacks. It shifts the focus from purely capability-based arguments to a more nuanced understanding of how tool-use taxes and failure-cost mechanisms influence the economics of automated software engineering.