Matryoshka Agent: Unfolding Sub-Agents for Long-Horizon Machine Learning Engineering

Machine learning engineering (MLE) tasks are notoriously difficult for autonomous agents because they require long-horizon decision-making, iterative debugging, and expensive environment interactions. Current monolithic agents often struggle to manage the resulting long, noisy contexts and vast solution spaces. This paper asks: can a hierarchical agent architecture, which separates strategic planning from concrete execution, improve performance and efficiency in these complex, long-horizon tasks?

The authors propose the Matryoshka Agent, a framework that decomposes problem-solving into three distinct layers:

• Orchestrator: A high-level decision-maker that maintains a compact state of previous attempts and outcomes, deciding which solution direction to refine next.
• Sub-Agents: Execution-focused modules that are instantiated with fresh contexts to implement, run, and debug specific code attempts based on the Orchestrator's instructions.
• Tool Interface: A mediation layer that translates high-level instructions into executable Sub-Agent contexts and returns structured summaries and scores to the Orchestrator.

To optimize this system, the authors introduce a training paradigm based on a "Solution Refinement Tree." This method samples alternative refinement paths, collects trajectory-level preference signals, and uses online reinforcement learning to improve the Orchestrator. Simultaneously, successful execution trajectories are used to fine-tune the Sub-Agents, creating a self-reinforcement loop.

The Matryoshka Agent demonstrates significant scalability and effectiveness across diverse MLE tasks. By decoupling strategic exploration from costly execution, the framework reduces the burden of long-context reasoning. Notably, the architecture allows smaller models, such as Qwen3-4B-Instruct, to achieve performance levels comparable to much larger models like o4-mini. Furthermore, applying the framework to Qwen3-30B-Coder resulted in a relative performance gain of up to 36.7%.