Zixiong Yu, Jun Rao, Guhan Chen, Songtao Tian, Bohan Li, Jiansheng Wei, Min Zhang, Xiaojun Meng
7 min
MathAgent tackles a critical bottleneck in training large language models (LLMs) for mathematical reasoning: the scarcity of high-quality training data. Traditional methods either mutate limited human-curated seeds (inherently bounded in diversity) or use zero-shot prompting (prone to mode collapse and logical errors). MathAgent reframes data synthesis as a hierarchical optimization problem over structured blueprints rather than direct text generation, enabling the creation of complex, diverse math problems without human priors or seed data.
The core innovation is the Legislator-Executor paradigm, decoupling logical structure design from natural language realization. This produces datasets that capture out-of-distribution (OOD) reasoning challenges, with models fine-tuned on just 1K MathAgent samples outperforming much larger mainstream datasets (e.g., LIMO, s1K) across eight math benchmarks like GSM8K and MATH.
At the heart are constraint graphs G=(V,E), where nodes V represent mathematical primitives (e.g., variables, operations like addition or geometry concepts) and edges E encode logical relations/constraints (e.g., 'depends on', 'must equal'). These graphs act as abstract skeletons specifying problem complexity without linguistic details.
Style tokens augment graphs with global attributes like difficulty or question type, providing coarse control. Unlike flat prompts, graphs enable systematic exploration of compositional structures—e.g., chaining inequalities with modular arithmetic—avoiding the 'superficial narrative imitation' of text-only synthesis.
The Legislator operates at a meta-level, adversarially evolving constraint graphs via a tri-agent system:
This unsupervised adversarial process iteratively optimizes for structural diversity and high difficulty, preventing mode collapse by prioritizing rare logical combinations absent in standard datasets. No seed problems needed—starts from basic primitives and evolves long-tail OOD samples.
Optimized graphs feed into the Executor, a conditional LLM that instantiates them into natural language question-answer pairs. By grounding generation in explicit constraints, the Executor produces coherent, diverse scenarios (e.g., word problems with embedded graphs) while preserving logical rigor.
This bi-level design ensures prioritized focus on reasoning structure over fluent text, yielding data that teaches core capabilities like multi-step deduction.
Fine-tuning Qwen/Llama/Mistral/Gemma models on 1K MathAgent samples yields superior OOD generalization on benchmarks testing extrapolation (e.g., harder variants of GSM8K). This demonstrates data quality > quantity, with implications for scaling reasoning in LLMs amid human data exhaustion.
MathAgent's seed-free, primitive-based approach generalizes beyond math to other reasoning domains, offering a blueprint for guided synthetic data at scale.
Synthesizing high-quality mathematical reasoning data without human priors remains a significant challenge. Current approaches typically rely on seed data mutation or simple prompt engineering, often suffering from mode collapse and limited logical complexity. This paper proposes a hierarchical synthesis framework that formulates data synthesis as an unsupervised optimization problem over a constraint graph followed by semantic instantiation, rather than treating it as a direct text generation task. We introduce a Legislator-Executor paradigm: The Legislator adversarially evolves structured generation blueprints encoding the constraints of the problem, while the Executor instantiates these specifications into diverse natural language scenarios. This decoupling of skeleton design from linguistic realization enables a prioritized focus on constructing complex and diverse logical structures, thereby guiding high-quality data synthesis. Experiments conducted on a total of 10 models across the Qwen, Llama, Mistral, and Gemma series demonstrate that our method achieves notable results: models fine-tuned on 1K synthesized samples outperform widely-used datasets of comparable scale (LIMO, s1K) across eight mathematical benchmarks, exhibiting superior out-of-distribution generalization.
Alex: So with just a thousand samples, it beats human-curated sets across different models and math tests—especially the tough ones. What makes you think it's really the structure driving that, not just luck?
Sam: They tested this with an ablation study—removing key parts one by one to see the impact. Without the constraint graph, forcing direct text generation instead, performance dropped. Skipping the adversarial rounds for a one-pass version cut performance further, as there's no iterative push to refine and stretch difficulty.
Alex: Right—so both the blueprint and the debate process are essential to avoid shallow copies. Does this hold up when they control for how answers are made?
Sam: Yes. They used the same answer generator for all datasets, including baselines like LIMO and new ones like NuminaMath. Even then, their 1K samples led to the highest scores, confirming the problems themselves are logically stronger—better at building reasoning skills without tricks in the answers.
Alex: And on really hard competition math, like olympiads? Plus, does specializing hurt general skills?
Sam: On intermediate and olympiad-level tasks, it outperformed even official instruction-tuned models from the makers—gains from capturing rare logical links that simple patterns miss. For cross-task checks on things like coding or general questions, forgetting was minimal compared to others, with a clear boost on code tasks sharing math logic. This points to robust transfer without much downside.
Alex: Huh. Structure-first data seems to teach deeper thinking that carries over. How does the Legislator actually refine those blueprints—walk me through the team behind it?
Sam: The Legislator relies on three specialized AI agents working together in rounds. One agent, the Proposer, starts with a basic idea and suggests changes to the graph—like adding new connections or concepts to make it more complex. Another, the Critic, reviews it closely for flaws, like logical gaps or mismatches with the desired difficulty, and proposes fixes. The third, the Moderator, weighs their input and decides whether to stop or loop back for another round.
Alex: So it's like a debate club: one pitches ideas, one pokes holes, and one calls time. Does a real example show how that builds something solid?
Sam: Yes, consider their case study on a saddle surface problem. The Proposer first links a basic shape to a real-world context, like a potato chip, and adds an integral for area. The Critic spots that without symmetry constraints, the math gets too hard—like elliptic integrals beyond medium difficulty—so it suggests fixes like equal parameters and partial derivatives. The Moderator agrees, refines once more, then approves the graph for the next phase.
Alex: Okay, that iterative push avoids weak spots early. Once approved, how does the Executor turn the abstract graph into readable problems?
Sam: The Executor takes the final graph and style details, then weaves them into fluent language—like turning nodes on surfaces and integrals into a story about calculating a chip's area, complete with bounds and symmetry steps. It follows a structured prompt to ensure every connection appears without robotic phrasing. This keeps the logic intact while making it natural for training.
Alex: Makes sense—blueprint to story without losing the frame. Did humans check the data quality?
Sam: Yes, they sampled 100 problems from MathAgent, Magpie, and OpenR1 for blinded human review on a five-point quality scale. MathAgent scored highest at 4.2 out of five. Experts noted MathAgent's problems have richer logical depth, while others repeat simple patterns—though automated judges sometimes overlooked flaws humans caught.
Alex: That aligns with the structure emphasis. Still, no method's perfect—what limits does the paper flag?
Sam: A few notable ones. It's tuned for math's strict logic, so adapting constraint graphs to fuzzy areas like law or stories needs work. The multi-round debates add compute time versus quick one-shot generators. And as problems get trickier, verifying exact solutions gets harder without extra checkers like math solvers.
Alex: Fair points—the gains are real but domain-bound and resource-heavy. Overall, a solid path for math data, with room to grow. Structure-first synthesis looks promising for pushing math AI forward.
Sam: Precisely. This work shows optimizing problem skeletons before text unlocks better training signals, with evidence from ablations, scaling, and human reviews. It points to a scalable path for reasoning AI, balanced by those practical hurdles.
Alex: Well put, Sam. Listeners, that's our dive into adversarial constraint graphs for math data synthesis—a step toward more capable AI tutors. Thanks for joining ResearchPod.