Mathurin Videau, Badr Youbi-Idrissi, David Lopez-Paz, Kartik Ahuja
5 min
Abstract
Neural scaling laws are foundational for language model development, yet standard formulations systematically under- and overestimate loss at data-scarce and overtraining extremes. This failure originates in the underlying assumption that model size and training data impact the loss independently. To address this, we introduce the Skaling law, a generalized functional form that couples model capacity and data through a single interaction exponent. This simple extension reduces the Mean Absolute Percentage Error (MAPE) by 1.5-3x across both interpolation and extrapolation regimes. When paired with a sparse grid strategy restricted to low-compute regimes, the Skaling law achieves accurate full-grid extrapolation using approximately 10x less compute than uniform sweeps. By enabling reliable performance prediction from small-scale experiments, the Skaling law provides a more robust and resource-efficient framework for allocating compute budgets in next-generation model training.
Alex: So by fixing the math, they've made the whole process of planning a training run significantly more efficient.
Sam: Precisely. Instead of throwing more hardware at the problem, they used mathematical insight to get the same answer with a fraction of the resources. It's a meaningful step toward more sustainable research.
Alex: How did they actually test whether that interaction between size and data is real, and not just an assumption they built into their own formula?
Sam: They used a technique called "Moving least squares." Imagine a scattered cloud of data points on a graph. Rather than forcing a single rigid line through them, this tool creates a flexible curve that bends to follow the data's actual shape. That lets them calculate the rate of change of performance at any point on the surface.
Alex: And that rate of change tells them whether the variables are genuinely interacting?
Sam: Right. They looked for what's called a "mixed derivative." The question is: if you change both model size and data volume at the same time, does the result differ from what you'd get by changing them one at a time and adding the effects together? The study found the answer was clearly yes — the interaction is real and measurable.
Alex: So the math confirms the synergy.
Sam: Exactly. Scaling both together reduces prediction error more than scaling them separately would suggest. And by capturing that relationship, the Skaling law can predict performance reliably even for models considerably larger than anything the researchers directly tested.
Alex: That last point seems important. Predicting the behaviour of a model you haven't built yet is the whole point of having a scaling law in the first place.
Sam: It is. To verify that the predictions hold up outside the training data, the researchers built a cross-validation framework. They deliberately withheld some data points and then checked whether the formula could guess them correctly. The real test wasn't filling in gaps within familiar territory — it was predicting performance for models at scales they had never directly measured.
Alex: And the L-shape strategy fits into that testing process as well?
Sam: It does. By training only on the edges of the grid, they preserve the interior data points as a genuine blind test. That's what makes the validation rigorous rather than circular. They're not checking the formula against data it was already fitted to.
Alex: It sounds like a careful, systematic way to build confidence in a prediction before committing enormous resources to it.
Sam: That's exactly the goal. The broader point is that by accounting for how model size and data volume interact — rather than assuming they don't — researchers can make meaningfully better decisions about how to allocate computing resources, with significantly less waste in the process.
Alex: Thanks for walking us through that, Sam. And thanks to everyone listening to ResearchPod.