Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, Dario Amodei
8 min
Abstract
We study empirical scaling laws for language model performance on the cross-entropy loss. The loss scales as a power-law with model size, dataset size, and the amount of compute used for training, with some trends spanning more than seven orders of magnitude. Other architectural details such as network width or depth have minimal effects within a wide range. Simple equations govern the dependence of overfitting on model/dataset size and the dependence of training speed on model size. These relationships allow us to determine the optimal allocation of a fixed compute budget. Larger models are significantly more sample-efficient, such that optimally compute-efficient training involves training very large models on a relatively modest amount of data and stopping significantly before convergence.
Alex: But why not just crank data forever? Doesn't more always help?
Sam: More helps, but diminishing returns kick in. The paper's loss formula captures that: as data grows, gains slow, while model size keeps delivering.
Alex: Whoa, that's the stained-glass moment. No more data droughts—engineer your way out. But does it hold at trillion-param extremes?
Sam: The trends are smooth over six orders of magnitude, up to 1.5 billion params here, but they note limits when data can't keep pace. Still, it predicts massive efficiency gains ahead.
Alex: Massive gains ahead, love it. But what's the mechanical reason that three-quarters split crushes the others?
Sam: They start with how loss depends on model size and data: it's roughly the worse of two limits—a model-capacity ceiling that drops smoothly as size grows, or a data-coverage floor that drops as data grows. Overfitting kicks in when model gets too big relative to data. To minimize total loss for fixed compute—which multiplies model size, data processed, and steps—you balance so those two limits are equal. No waste on one side.
Alex: Equal limits? The model term improves faster per compute unit because its scaling is sharper—loss drops quicker with size than with extra data or steps?
Sam: Exactly. Plot fixed compute curves, and the dip is deepest there: model taking 73% of budget, data 27% via larger batches, steps just 3%. Train huge models briefly on modest data till test loss peaks—no overfitting if you stop there.
Alex: Steps only 3%—that's shocking. Isn't more training always better?
Sam: More steps help up to a point, but diminishing returns hit fast. They adjust for real runs using a critical batch size that depends only on the current loss level—roughly doubles every 13% loss drop, tracking gradient noise. Small batches add beneficial noise early, but past critical size, it wastes compute. Big models leverage that noise tolerance best, learning patterns from fewer examples without variance overwhelming signal.
Alex: Say someone's budget forces a model a bit off that perfect split, like smaller or bigger than optimal. How bad is the hit?
Sam: Not too bad. Models from 0.6 times to 2.2 times optimal size reach the same performance with just 20% more total compute. It's a wide sweet spot, so you have flexibility.
Alex: Nice, forgiving. And those steps being only 3%—does that mean giants train way faster?
Sam: Yes, larger-than-optimal models need fewer steps to hit peak loss, while smaller ones drag on longer. With enough parallel hardware, big models finish quicker overall.
Alex: But you hinted at limits—where does this magic pie break?
Sam: Theory predicts a clash far out. Compute-optimal data grows slowly, but to dodge overfitting, you need data scaling faster with model size. They intersect around 10^12 parameters and tokens—trillions—maybe marking where scaling laws fail or Transformers hit language's max predictability, around 1.7 nats per token.
Alex: Nats—like uncertainty measure? So perhaps that's human-level compression of English?
Sam: Exactly, a rough entropy estimate for natural text. The paper conjectures we could reach it with existing internet data, training trillion-param models in days at critical batch sizes. No petabytes needed; just scale smart till the wall. Very sensitive, though—exponents could shift the spot by an order of magnitude either way.
Alex: Days for trillion-params rivaling linguists? Guides the grind away from data hell. But is this just for language, or universal?
Sam: They conjecture it holds for other generative tasks using maximum likelihood loss, like images or audio, since the relations emerge from optimization dynamics, not language specifics. Bigger models prove far more sample-efficient overall. But they stress no solid theory yet; scalings remain mysterious without deeper statistical mechanics.
Alex: Mysterious under the hood, but works like clockwork on top. Researchers could ditch data scrambles and just size up confidently? It's like the ideal gas law tying volume, pressure, and temperature universally.
Sam: Precisely. Loss ties to model size, data, and steps in power-law ways that let you forecast the best setup for any compute budget. They even derive exactly when to early-stop to dodge overfitting. Fits weaken at tiny data, untested hyperparameters might shift things, and extreme contexts could confound estimates. Still, over six orders of magnitude, no breaks, guiding pushes to human-level text compression.
Alex: From data hell to engineering heaven. Wrapping my head around this makes me rethink those headlines about AI needing endless data oceans. It's more like smart budgeting turns the grind predictable and human-scale. Imagine skipping petabyte hunts; teams could prototype breakthroughs faster.
Sam: Spot on. It shifts focus from scraping every byte to elegant setups that just work. The wide sweet spot gives flexibility for hardware quirks. Trends hold strong up to billions of params, but extremes around trillions could shift if data lags—no guarantees.
Alex: Fair caution; keeps the hype in check. Personally, I'm stoked but skeptical—excited for quicker AI leaps, yet curious if we'll hit those walls sooner. Either way, it feels like clockwork progress. Thanks for breaking it down, Sam—this flips how I see AI's future.
Sam: My pleasure. Train huge, brief, balanced—big models sip data where small ones guzzle. Thanks for listening to ResearchPod.