Chinchilla
Chinchilla is a 70-billion-parameter Large language model trained by DeepMind and introduced in the 2022 paper "Training Compute-Optimal Large Language Models" by Hoffmann and colleagues. Rather than chasing sheer parameter count, the project set out to answer a deceptively simple question: for a fixed amount of computing power, what is the best possible combination of model size and training data? The answer reshaped how the entire field trains language models.
Prior to Chinchilla, the prevailing wisdom—embodied in giants like GPT-3 and DeepMind's own earlier model Gopher—was that bigger models were better, and training data was treated almost as an afterthought. Hoffmann et al. revisited scaling laws with extended experiments and found that many recent LLMs were significantly under-trained for their size. By training over 400 models of varying sizes and token counts and fitting curves to the results, the researchers derived what became known as the "Chinchilla scaling laws": for a given compute budget, there is an optimal pair of model size and tokens, with a roughly linear relationship between them—doubling model size should go along with doubling training data.
To test this, DeepMind trained Chinchilla itself: a 70-billion parameter model trained for 1.3 trillion tokens. Crucially, Chinchilla uniformly and significantly outperformed Gopher (280B), GPT-3 (175B), Jurassic-1 (178B), and Megatron-Turing NLG (530B) on a large range of downstream evaluation tasks, despite using the same training compute as Gopher and having a fraction of its parameters.
The practical upshot was a quiet revolution: rather than building ever-larger Neural Network architectures and starving them of text, labs began scaling up datasets in tandem with model size, producing smaller, cheaper-to-run models that were nonetheless more capable. This principle now informs the design of countless successors, influencing everything from open-source Transformer-based models to commercial systems fine-tuned with Reinforcement learning from human feedback techniques pioneered alongside InstructGPT.
Like its cousins BERT, T5, and GPT, Chinchilla builds on the Self-attention mechanism described in Attention Is All You Need, itself a descendant of earlier Sequence to sequence learning and Neural machine translation work using Long short-term memory networks. But where those papers advanced architecture, Chinchilla advanced methodology—a reminder that in machine learning, as in ecology or economics, growth without balance leads to waste, and that true optimization often means asking not "how big" but "how proportioned."