scaling laws

A scaling law is an empirical power-law relationship describing how a language model's loss falls as model size, training data, or compute increases — smoothly and predictably enough that researchers can forecast the performance of a model they haven't trained yet, purely from the trend measured on smaller ones. Kaplan and colleagues' 2020 OpenAI paper "Scaling Laws for Neural Language Models" established this predictability, but it also drew the wrong conclusion about how to spend a fixed training budget, emphasizing parameter count over data; Hoffmann and colleagues' 2022 DeepMind paper introducing Chinchilla corrected that conclusion, finding that most large models up to that point — including GPT-3 — had been systematically undertrained relative to their size, and that model size and training data should scale together rather than favoring parameters alone.

IsoFLOP analysis is the specific method Chinchilla used to find the correction, and it works by holding compute fixed rather than holding model size fixed. Train many combinations of model size and data quantity that all consume the same total compute budget, plot the resulting loss for each combination, and the combination that reaches the lowest loss at that fixed compute level reveals the compute-optimal model-size-to-data ratio for that budget — repeat across several different compute budgets, and the resulting sequence of optimal points traces out the scaling relationship itself, visible as the dashed frontier connecting the lowest point on each IsoFLOP curve.

The concrete number Chinchilla's analysis produced was roughly 20 training tokens for every model parameter, and that ratio reset the field's intuition about where training budgets should actually go. A 70-billion-parameter model, by this ratio, wants roughly 1.4 trillion training tokens to be compute-optimal — GPT-3's 175 billion parameters trained on only about 300 billion tokens, by comparison, was a model with far more capacity than its training data could actually put to good use, which is exactly the undertraining Chinchilla's paper identified and which DeepMind's own 70-billion-parameter Chinchilla model, trained on the correct 1.4 trillion tokens, outperformed.

Frontier labs have since deliberately moved away from the Chinchilla-optimal point in one specific direction, and the reason is inference cost rather than training cost. Meta's Llama 3 405B model was trained on roughly 15 trillion tokens — far beyond what Chinchilla's ratio would call optimal for a model that size — because a model gets served to users far more times than it gets trained, so a smaller, more heavily-trained model that reaches the same quality as a larger, Chinchilla-optimal one is cheaper to run in production even though it cost more to train in the first place; this "overtraining" strategy trades a higher, one-time training cost for a lower, repeated inference cost, which dominates total cost of ownership at the scale frontier labs actually deploy at.

Scaling Laws — Power Laws of Neural Networks loss decreases as a power law in parameters, data, and compute: L(x) ∝ x^(-α) Three Knobs That Reduce Loss (Kaplan et al. 2020 / Chinchilla 2022) N — Parameters L(N) ∝ N^(-0.076) 10× params → ~17% lower loss 1B → 10B → 100B → 1T D — Data (tokens) L(D) ∝ D^(-0.095) 10× data → ~20% lower loss 100B → 1T → 10T tokens C — Compute (FLOPs) L(C) ∝ C^(-0.050) C ≈ 6·N·D (approx) 1e21 → 1e23 → 1e25 FLOP Chinchilla Optimal Allocation Kaplan (2020): "make model bigger, data secondary" Chinchilla (2022): "scale N and D equally" Optimal: D ≈ 20 × N tokens 70B model needs ~1.4T training tokens GPT-3 (175B, 300B tokens) was undertrained IsoFLOP Curves (log-log) Loss ↓ Parameters → 1e23 1e24 1e25 optimal frontier Frontier Models on the Scaling Curve GPT-4 ~1.8T params (MoE) ~13T tokens, ~2e25 FLOP Llama-3 405B 405B dense 15T tokens (over-trained) DeepSeek-V3 671B MoE (37B active) 14.8T tokens Chinchilla 70B dense 1.4T tokens (optimal) Gemini Ultra ~1T+ MoE multi-modal, ~3e25 FLOP Post-Chinchilla trend: over-train smaller models (more tokens than optimal) to reduce inference cost at fixed quality. Scaling laws are the physics of deep learning: predictable, smooth, and not yet saturating at frontier scale.

"Emergent" capabilities that appear to jump suddenly at a certain scale are sometimes a measurement artifact rather than a genuine discontinuity in the underlying model. A model's raw probability of getting a hard multi-step problem exactly right can improve smoothly and continuously with scale while still crossing a discrete pass/fail benchmark threshold abruptly — the underlying ability was never actually discontinuous, but a metric that only reports "solved" or "not solved" makes a smooth improvement look like a sudden jump, which is a genuine and ongoing methodological debate in how scaling-law researchers interpret capability benchmarks rather than a settled fact about how capabilities actually emerge.

Scaling laws turned training a frontier model from a guess into a forecast: given a compute budget, the Kaplan-to-Chinchilla progression tells a team roughly where to place their model size and token count to minimize loss for that budget, and the post-Chinchilla overtraining trend shows that even a well-established law gets deliberately overridden once a different cost — inference, not training — starts to dominate the actual economics of shipping a model.

Go deeper with CFSGPT

Get AI-powered deep-dives, save terms, and run advanced simulations — free account.

Create Free Account