AutoTrainess: Teaching Language Models to Improve Language Models Autonomously

Training language models (LMs) is typically a labor-intensive, human-led process. While autonomous agents have shown promise in software engineering, they often struggle with the complex, long-horizon requirements of model training—such as iterative planning, data curation, and stable checkpoint management. AutoTrainess addresses this by providing a training-specialized Agent-Computer Interface (ACI) that guides the agent through structured workflows, reducing the uncertainty of raw command-line environments.

Instead of allowing the agent to operate in an unconstrained CLI environment, AutoTrainess introduces AutoTrainHub, a repository of reusable interfaces that organize training into a closed-loop system. This system consists of four primary modules:

• Data Processing: Explicit actions for selecting, constructing, and validating training data, ensuring alignment with benchmark-specific formats.
• Training: A stable, LlamaFactory-based backend that enforces consistent training practices and reduces engineering variance.
• Evaluation: A standardized pipeline that ensures comparable results and provides diagnostic feedback for subsequent iterations.
• Logging & Planning: A persistent memory mechanism that records experiment history and uses empirical evidence to define the objective of the next iteration.

AutoTrainess consistently outperforms CLI-only baselines across various models and benchmarks. Using GPT-5.4 (Codex), it achieved an average score of 26.94 on PostTrainBench, compared to 23.21 for the CLI-only baseline. The framework also demonstrated strong generalization, improving DeepSeek-V4-Flash from 12.13 to 19.58. Ablation studies revealed that each interface component provides complementary benefits: data tools improve the quality of training inputs, evaluation interfaces reduce orchestration failures, and planning tools sustain the iterative search process.

AI-generated third-party summary by ResearchPod. Not official content or an endorsement by the paper authors or affiliated organizations.