Sewon Min
9 min
Abstract
We study pretraining when compute is abundant but unique training data are finite, requiring repeated training on the same data. We first revisit infinite-compute pretraining (Kim et al. 2026) and show that its frontier depends critically on choices previously fixed or studied only in isolation. In particular, optimizing batch size together with epoch counts reveals a U-shaped relationship between validation loss and batch size, favoring smaller batches than typical pretraining, likely because small-batch gradient noise provides implicit regularization. We further find that training recipes, previously optimized under a single-pass assumption, can be tuned, e.g., repacking documents into new sequences every epoch, weight tying, and positive embedding weight decay. We then relax the infinite-compute assumption. In particular, smaller-than-typical batch sizes can require prohibitively long sequential training, motivating to treat GPUs and time as separate resources. Tighter deadlines favor larger batches, and latency-oriented parallelism can trade additional GPUs for shorter training. Nevertheless, even under a fixed time budget, a smaller batch with fewer GPUs can achieve lower validation loss than a larger batch with more GPUs, challenging the common practice of increasing data parallelism to maximize GPU utilization under a deadline. Together, our results identify underexplored factors in repeated-data training and provide concrete guidance on how training recipes and resource allocation should change from single-pass pretraining.
Alex: That’s the central methodological move. Each batch size gets its own tuned learning rate, weight decay, and stopping point. Weight decay shrinks parameter values during training. The comparison asks what each batch can achieve, not how it performs under somebody else’s settings.
Sam: What models and data support that comparison?
Alex: The measured models range from about 153 million parameters to about one and a half billion. They are dense language models, with all model parameters participating rather than selected expert components. Training uses the filtered DCLM-Baseline text corpus, with documents separated between training and evaluation.
Sam: Give me one result that captures the batch-size effect.
Alex: For the largest model, a batch of 64 sequences outperformed a batch of 512. Both comparisons use their best tuned endpoints on the same fixed data pool.
Sam: How much better was the held-out prediction error?
Alex: Validation cross-entropy fell from about 3.12 to about 2.98. That’s a language-model prediction measure, not a task accuracy percentage. Across the model sizes, the original recipe’s best results occurred at an intermediate batch, not either extreme.
Sam: So the result isn’t simply “add more gradient noise.” What happened when batches became too small?
Alex: The smallest batch eventually achieved lower training error but worse validation error than the intermediate batch. Those extra updates kept fitting the training text without improving unseen-text predictions. At the large-batch end, training error also eventually fell below the intermediate batch’s, while validation remained worse.
Sam: That makes simple undertraining an insufficient explanation for the large-batch result. But does it establish that noise is the protective mechanism?
Alex: No. The author describes the pattern as consistent with weaker stochastic regularization at large batches. That’s a supported interpretation, not a demonstrated causal mechanism. The strongest direct result is the U-shaped relationship between batch size and best validation error.
Sam: Now separate the batch result from the recipe changes. What can change if the underlying text stays identical?
Alex: How documents are assembled into training sequences. One loader repeats the same packed sequences and batches. Another reshuffles existing sequences into fresh batches. REPACK goes further: it reshuffles documents and builds new fixed-length sequences each pass.
Sam: So the words aren’t new, but their combinations and surrounding contexts can be. Did that help beyond ordinary shuffling?
Alex: It improved held-out prediction error at both tested batch sizes. The benefit was clearer at the larger batch, and the best checkpoint came later. That supports treating presentation across repeated passes as a training decision, not just a data-loading detail.
Sam: The recipe also changes the model’s word representations. What’s the distinction between sharing them and regularizing them?
Alex: Weight tying shares one parameter matrix between input word representations and output predictions. On its own, tying brought little or no improvement. Applying positive weight decay to the embedding parameters helped consistently, and combining decay with tying produced the best results.
Sam: Does sharing parameters quietly change model size, making the comparison harder to interpret?
Alex: It reduces the number of trainable parameters. The paper keeps the original model names for consistency. So the better tied model actually has fewer parameters than its untied baseline; the improvement isn’t explained by giving it a larger model.
Sam: What did the combined recipe change in practical training terms?
Alex: For the largest model, the selected stopping point moved from 20 passes to 32. Smaller models also benefited from longer training. The recipe improved validation error across the tested model sizes, but it also changed which batch size was best for the smaller models.
Sam: That suggests we shouldn’t export a single magic batch size. How does the deadline enter the decision?
Alex: A small batch can need a long sequence of parameter updates. More GPUs don’t automatically remove that sequential burden. The paper therefore treats GPU availability and elapsed training time as separate resources.
Sam: With a short deadline, a larger batch can process more text before time runs out. Does the study find that tradeoff?
Alex: Yes. Short budgets tend to favor larger batches. With more time, progressively smaller batches become preferable. In the largest-model example, a smaller batch using fewer GPUs eventually beats a larger batch using more GPUs under the same deadline.
Sam: Are these resource comparisons actual wall-clock training races, or another kind of estimate?
Alex: The main frontiers use observed training outcomes, with deadlines expressed in optimizer-step equivalents. Separate latency-first projections model ways to spend extra GPUs on faster updates without enlarging the global batch. Those projected speedups are not end-to-end measurements.
Sam: How much confidence should we put in the findings beyond these experiments?
Alex: Within the tested setting, the batch effect is persuasive: it appears across model sizes and remains consistent across independently sampled data pools. Downstream benchmark trends broadly support it, although task accuracy is noisier than validation error.
Sam: And what’s the biggest boundary on that confidence?
Alex: Scale and recipe dependence. The experiments use relatively small dense models, one corpus, and one optimizer and learning-rate schedule. The much larger model example is explicitly an extreme extrapolation, not a validated prediction. Mixture-of-experts models remain an especially important untested extension.
Sam: Who should read the full document, and where should they start?
Alex: Anyone planning repeated-data pretraining or allocating hardware under a deadline. Start with the controlled recipe ablations in Section Four, especially repacking versus shuffling and embedding decay versus tying. Then read Section Five’s joint GPU-and-time tradeoffs, keeping the measured results separate from the projections.
Sam: And what should everyone else remember?
Alex: When training data repeat, keeping every GPU busy is not the same as training the best model.
Sam: Keep the data limit in view.