Youqing Fang, Zhouqi Hua, Yinhao Tang, Yuzhe Gu, Lingkai Kong, Bin Liu, Weiming Zhang, Kai Chen, Yanan Sun, Wenwei Zhang
8 min
Abstract
Mathematical supervision for language models is typically flattened into isolated problem-solution pairs, discarding the curricular organization and evidential structure of the original sources. We introduce TOME-Math (Textbook-Oriented Mathematical Evidence), a large-scale corpus that converts 2,118 textbooks spanning the core areas of university-level mathematics subjects into approximately 380K structured proof-supervision records while preserving chapter order, proposition-proof links, dependency references, and source provenance. Leveraging these structural signals, we explore curriculum learning across multiple mathematical subjects and characterize scaling patterns with supervision volume and source breadth. At full scale, fine-tuning on TOME-Math raises the unweighted mean score across our held-out evaluations and established public benchmarks by 16.7 percentage points over the base model, while exceeding all three public instruction-data baselines evaluated in aggregate. Our findings highlight structured textbooks as both a scalable source of mathematical supervision and an empirical framework for studying the impact of data organization on mathematical reasoning. We will release the corpus to facilitate further research on structured mathematical supervision.
Alex: But the reader still has to connect an even total with an even count of odd contributions.
Sam: The constructed answer spells that out. Even-degree vertices contribute nothing to the parity of the sum. Each odd-degree vertex flips it. For the total to be even, the number of those flips must be even. That illustrates the kind of intermediate reasoning the pipeline makes explicit.
Alex: How do they keep that process tied to the textbook, rather than asking a model to invent plausible mathematics?
Sam: They recover the table of contents and chapter boundaries from scanned textbook files. Then they separate exposition from native exercises. From the exposition, models extract propositions and proofs, resolve references to earlier results, and inspect proofs for missing steps. Candidate obligations are rewritten to declare the symbols and definitions they need.
Alex: That involves several opportunities for mistakes. What happens before a candidate reaches the dataset?
Sam: Automated checks assess completeness, accuracy, mathematical formatting, self-containment, and whether an item forms a coherent unit. Unsupported proof steps count as failures, not stylistic defects. But the later quality audit raises concerns about how well that source-grounding standard is achieved. We should keep that distinction in mind.
Alex: Let’s separate the headline result from the explanation for it. What did the main comparison establish?
Sam: All training runs start from the same Qwen base checkpoint. The authors compare TOME-Math with three public mathematical instruction datasets. Each gets a full training pass under shared optimization settings. TOME-Math has the highest unweighted average across six evaluations, leading on everything except the AIME competition set.
Alex: Those datasets differ in content and length, though. Winning that comparison would not establish that chapter order caused the improvement.
Sam: The authors explicitly say so. These runs have different total training-token counts, despite sharing the training setup. The comparison establishes the usefulness of the whole package. To isolate organization, they run more controlled experiments.
Alex: Before those, how do they test whether the model can handle unfamiliar textbook problems?
Sam: Their long-form evaluation contains three hundred problems drawn from books excluded from training. It covers several mathematical subjects and is proof-heavy, rather than exclusively proofs. Native exercises are kept separate from the proof-extraction stream, and all training records from the reserved books are removed.
Alex: Now suppose I already have a model and cannot retrain it. Does textbook structure still help?
Sam: In their retrieval experiment, yes. Retrieval means supplying selected source material alongside the question. The frozen model’s single-answer success estimate rises from about thirteen percent without context to about seventeen percent with structured propositions and explicit proof-gap tasks. All context-supplying conditions get the same token allowance.
Alex: So the useful comparison is not just context versus no context. It is carefully selected evidence versus random or nearby material from the same book.
Sam: And the structured policy does better than those alternatives. They remove the target exercise and material overlapping its reference answer from the retrieval pool. Random same-book context and chapter-local context sometimes fail to help particular subjects. Access alone is not the same as useful evidence selection.
Alex: What happens when they change the learning order instead of the evidence supplied at test time?
Sam: That is curriculum learning: deliberately ordering training examples. Within each subject and training budget, the schedules use identical examples. They compare shuffled order, backward textbook progression, and forward progression. They also test whether presenting local gap tasks before the remaining proof examples helps.
Alex: Which part of that recipe carries the result?
Sam: Forward progression with gap tasks first wins in every subject-and-budget combination they test. In these subject-specific experiments, chapter direction contributes most, with gap-first ordering adding a smaller gain. In the full-corpus ablation, gap-first is the stronger ordering signal. Removing gap tasks also hurts every benchmark.
Alex: That makes the organizing principle more credible, but not identical at every scale. Does the number of source books matter independently of training volume?
Sam: They test that too. Spreading a fixed token budget across more books improves performance in every tested subject. That supports source breadth as another useful dimension, beyond simply adding examples. But all these model experiments use the same backbone, so generalization across model families remains untested here.
Alex: We also need to trust the judgments of a successful proof. Are these machine-checked proofs?
Sam: No. Even the MiniF2F and ProofNet evaluations assess generated natural-language proofs using a language-model judge. For the textbook evaluation, repeated reviews must all find no substantive error. Human calibration still finds false acceptance, and estimates that roughly one in five rejected responses should have been accepted.
Alex: And you flagged a separate concern about the training records themselves.
Sam: The corpus audit reports that source groundedness and evidence alignment each pass for fewer than half the records. That does not mean every remaining answer is mathematically wrong. It does mean the pipeline’s source-faithfulness goals should not be mistaken for guaranteed source-faithfulness in the released records.
Alex: Given those reservations, who should read the full document, and where should they begin?
Sam: Researchers building mathematical training corpora, proof-generating models, or evidence-retrieval systems should read “Leveraging Textbook Structure.” Start with the matched-example curriculum comparisons. Then read the corpus quality audit and human calibration of the verifier. For everyone else, remember this: mathematical training data includes the connections between answers, not just the answers themselves.
Alex: Keep the connections, and check the evidence.