Scaling laws for neural language models
Scaling laws are the empirical rules that predict how well a Large language model will perform before anyone finishes training it — a kind of astrology that actually works, because it's built on data instead of stars. In their most famous form, they describe how the Cross-Entropy test loss of a Transformer falls off in a smooth, predictable Power Law as three quantities grow: the number of model parameters (N), the size of the training dataset (D), and the total computational budget (C).
Origins
The field crystallized in 2020, when a team at OpenAI led by Jared Kaplan trained hundreds of models spanning enormous ranges of scale and found something startling: Kaplan and collaborators trained over 200 transformer language models spanning seven orders of magnitude in compute, four orders of magnitude in non-embedding parameter count, and three orders of magnitude in tokens, reporting clean power-law fits relating cross-entropy test loss to each factor. Formally, the loss L scales with a power law behavior as a function of these three scale factors: L(C) ∝ C^γ, L(N) ∝ N^α, L(D) ∝ D^β, where compute is measured in FLOPs, N is parameter count, and D is training tokens. The finding echoed earlier observations by Hestness and colleagues that deep learning error behaves predictably across scales, but Kaplan's team formalized it specifically for GPT-style architectures, turning "make it bigger" from a hunch into a slope on a graph. Before this work, model design was closer to alchemy; afterward, the discovery that performance scaled predictably with these factors changed large language model development from an art into more of a science, giving researchers mathematical tools to make informed decisions about how to invest computational resources.
Chinchilla and the compute-optimal correction
The original recipe suggested pouring most of one's compute budget into parameters rather than data — a lesson OpenAI followed when building GPT-3. But in 2022, DeepMind's Hoffmann et al. revisited the sweep and found the ratio was off: the "compute-optimal" criterion (the Chinchilla laws) proved that model parameters and training tokens must scale equiproportionally to mitigate diminishing returns under fixed compute budgets, rather than at the earlier 3-to-1 skew favoring parameters. The resulting model, Chinchilla, out-performed larger but under-trained peers by simply reading more tokens.
Caveats and frontiers
The constants in these equations are not universal — they must be refit for each tokenizer, architecture, and data mixture. Cross-entropy loss is also not the same as usefulness: a smooth power-law curve can mask sudden jumps in downstream ability, the phenomenon researchers call emergent capability, and newer work extends the framework to test-time computation, Mixture of experts routing, and data quality curation rather than raw quantity alone.