Kishanthan Thangarajah, Boyuan Chen, Ahmed E. Hassan
6 min
Abstract
CLI-based software-engineering agents have matured rapidly, yet the open ecosystem has converged on a single training environment: trajectory datasets used to fine-tune open models are collected almost exclusively under OpenHands. Models fine-tuned on this data score well under OpenHands but degrade substantially when deployed under any non-training scaffold. Untrained base models do not show this divergence, indicating the gap is fine-tuning-induced and tied to the conventions of the training scaffold. We argue that a load-bearing scaffold-specific behavior is planning structure, in two senses this paper distinguishes: explicit planning, a pre-execution plan produced as a first-class artifact, and implicit planning, the structural conventions that shape execution throughout the agent loop. Under this hypothesis, closing the gap requires moving planning from a fixed scaffold artifact to a learned model capability. We introduce Decoupling CLI Agent Scaffolding (DCAS), a backend-substitution interception layer that routes API traffic between any CLI scaffold and any backend model without modifying the scaffold, enabling cross-scaffold evaluation and planning-aware trajectory collection. Using DCAS, a controlled plan-source intervention confirms planning quality is a high-leverage component, with gains exceeding the cross-scaffold drops we observe. A model fine-tuned on a small set of DCAS-collected planning-aware trajectories under a single scaffold gains consistently across non-training scaffolds, and the two senses of planning are empirically separable in training data.
Alex: That's the hypothesis. To test it, the researchers built a tool called DCAS—Decoupling CLI Agent Scaffolding. It acts as a middleman layer that lets you plug any AI model into any scaffold without modifying the scaffold itself. Think of it like a universal adapter that lets you run any appliance in any country, regardless of the local plug standard.
Sam: Why does that middleman matter so much?
Alex: Without it, you can't easily swap models or environments to isolate where the failure is coming from. Is it the model? Is it the scaffold? Is it the interaction between the two? DCAS lets the researchers hold one variable constant while changing the other. It also lets them collect what they call "planning-aware" data—recordings of exactly how a model behaves across different setups, with the planning decisions clearly labeled.
Sam: And when they ran those experiments, did planning quality actually turn out to be the key variable?
Alex: It did. When they controlled the source of the plan—essentially giving the agent a clearer, more structured blueprint to work from—the success rate went up meaningfully. Planning quality turned out to be a high-leverage component. Small improvements in the plan produced noticeable improvements in the outcome.
Sam: So the plan is like the foundation. If the foundation is solid, the agent can build on top of it regardless of which scaffold it's working in?
Alex: That's a useful way to think about it. And here's where it gets particularly interesting. They found that by training on a relatively small set of these planning-aware examples, the agent actually develops a transferable skill. It stops memorizing the environment and starts learning how to plan as a general capability—something it can bring to environments it has never encountered before.
Sam: That's a meaningful shift. It means you don't have to retrain these agents from scratch every time you move them to a new platform.
Alex: Correct. Once the model internalizes the underlying structure of planning, it can apply that skill in new contexts. The intelligence becomes portable rather than being locked to one specific harness.
Sam: Does this also change how we should be measuring these agents? Because if a model scores well on one scaffold but falls apart on another, that score doesn't tell us much.
Alex: That's a key takeaway from the paper. Reporting performance from a single environment is misleading—it hides how poorly the agent might perform elsewhere. The researchers argue that testing across multiple scaffolds should become the standard. It's the only way to distinguish between an agent that has genuinely learned to solve problems and one that has simply learned to navigate one specific tool.
Sam: Though I imagine the authors are careful not to claim this solves everything.
Alex: They are. They acknowledge that planning is a major piece of the puzzle, but it doesn't address every challenge. Things like how the agent handles very long sequences of information, or how it formats its tool requests, remain separate problems that need their own solutions.
Sam: So DCAS and planning-aware training represent a meaningful step toward agents that can work across different environments—but there's still work to do on the underlying plumbing.
Alex: Exactly. By treating planning as a structural skill rather than an environment-specific habit, we move closer to models that can operate reliably wherever they're deployed. That's a more honest and more useful goal than optimizing for one benchmark on one platform. Thanks for listening to ResearchPod.