Yubo Wang, Jiarong Liang, Yuxuan Zhang, Xuye Liu, Cong Wei, Yuyu Zhang, Ping Nie, Wenhu Chen
4 min
Coding agents rely on post-training pipelines to learn how to interact with tools and resolve software engineering tasks. However, standard pretraining on code often fails to provide the specific inductive biases required for the agentic action-observation-continuation loop. This paper investigates whether a dedicated mid-training stage, designed to align models with the structural logic of function calls, can improve agent performance and mitigate the capability erosion typically caused by agentic post-training.
The authors observe that a coding agent's step-by-step reasoning process is structurally isomorphic to a function call site: both involve context, a call/action, an externally computed return/observation, and a continuation. They introduce a function-aware fill-in-the-middle (FIM) objective that masks functions selected via program dependency graph analysis and a complexity-inferability criterion. This mid-training stage uses a 2.6B-token corpus of Python code, where models are trained to generate a chain-of-thought rationale followed by the function body, effectively embedding agent-like reasoning into the model's structural prior.
Mid-training consistently improves performance across multiple model sizes (Qwen2.5-Coder 7B/14B and Qwen3-8B) and various post-training pipelines (R2E-Gym, SWE-Smith, and SWE-Lego). On SWE-Bench-Verified, the method yields gains of +2.8 to +3.2 percentage points. Furthermore, the authors demonstrate that this mid-training mitigates the significant capability erosion—such as drops in general coding and tool-use benchmarks—that typically occurs during agentic post-training. Even though the mid-training corpus consists solely of Python code, the induced function-call bias transfers to non-coding tool-use benchmarks, providing evidence that the structural prior is robust and generalizable.
This work highlights a critical gap between standard pretraining and agent-specific post-training. By demonstrating that structural priors can be injected via self-supervised mid-training, the authors provide a scalable way to enhance agentic reasoning without requiring expensive, manually curated agent trajectories. This approach not only boosts performance on complex software engineering tasks but also helps preserve the model's general-purpose capabilities, making it a valuable strategy for building more versatile coding agents.
Coding agents must integrate external tool returns into ongoing reasoning - a capability that standard left-to-right pretraining on code exposes only in its forward direction. We observe that the action-observation-continuation loop of a coding agent is structurally isomorphic to a function call site, where a caller binds arguments, a callee returns a value computed elsewhere, and downstream code consumes that value. This conditioning structure exists at internet scale in ordinary code. We exploit it through function-aware fill-in-the-middle (FIM) mid-training: a self-supervised objective that masks functions selected via program dependency graph analysis and a complexity-inferability double criterion. We mid-train Qwen2.5-Coder-Instruct (7B/14B) and Qwen3-8B on a 2.6B-token decontaminated corpus drawn from 968 GitHub repositories, then apply existing agentic post-training pipelines. Mid-training improves SWE-Bench-Verified by +2.8/+3.0 at 7B/14B and by +3.2 on Qwen3-8B; SWE-Bench-Lite gains are +3.7/+4.0/+5.4 on the same models. The improvement holds across two post-training pipelines (R2E-Gym, SWE-Smith) and on a non-Qwen2.5 base (Qwen3-8B with SWE-Lego). Beyond in-domain gains, mid-training also mitigates the capability erosion that agentic post-training otherwise inflicts on non-agent coding (e.g., LiveCodeBench) and non-coding tool-use benchmarks (tau-bench, BFCL): although the mid-training corpus contains Python code only, the function-call inductive bias survives post-training and yields consistent gains.
Alex: That's where two pieces of the system work together. First, they build what's called a "Program Dependency Graph"—think of it as a map of the code that shows which functions rely on which other functions. This lets them target functions that sit at important junctions in the logic, rather than isolated, throwaway pieces.
Sam: And the second piece?
Alex: They score each candidate function on what they call "complexity-inferability." It's a balance. Too simple, and the model learns nothing—it's like blanking out the word "the" in a sentence. Too obscure, and there's no way to reason your way to the answer. They want the sweet spot: challenging enough to force real logical thinking, but solvable if you read the surrounding code carefully.
Sam: Does all of this actually translate to better performance on real-world tasks, like fixing bugs in a live codebase?
Alex: The results show a meaningful improvement. Models trained with this structural approach consistently resolved more real software issues than models trained without it. The paper suggests that by building this kind of logical awareness into the base model early—before any task-specific fine-tuning—the model becomes substantially more capable when it encounters complex, multi-step problems later.
Sam: That's a notable finding. Does the paper say anything about why this works better than just training on more synthetic data?
Alex: The core argument is that synthetic data, however carefully designed, is a simulation of reasoning. Real code is reasoning—it's the actual decisions programmers made when solving real problems, preserved in structure. When you train a model to reconstruct that structure, you're not teaching it to imitate reasoning. You're giving it practice doing the real thing.
Sam: That's a meaningful distinction. It's the difference between studying worked examples and actually working through problems yourself.
Alex: Exactly. And the implication is that the training signal researchers have been looking for was sitting in public repositories all along. The challenge wasn't finding more data—it was learning to read the data that already existed more carefully.
Sam: Thanks for walking through that. It's a clear-eyed look at how the structure of code itself can be a teacher.
Alex: Thanks for listening to ResearchPod.