BERT
BERT (Bidirectional Encoder Representations from Transformers) is a landmark Machine learning model for natural language processing, unveiled by researchers at Google in late 2018. Bidirectional encoder representations from transformers (BERT) is a language model introduced in October 2018 by researchers at Google. It did not merely nudge the field forward — it reset expectations for what a machine could understand about human language, and its name has since become a byword for the era when Self-attention ate the world.
Architecture and Origins
BERT is built entirely from the encoder half of the Transformer architecture first described in Attention Is All You Need. It uses the encoder-only transformer architecture. Where earlier approaches to Word embedding like Word2Vec assigned each word a single, fixed vector regardless of context, context-free models such as word2vec generate a single word embedding representation for each word in the vocabulary, whereas BERT takes into account the context for each occurrence of a given word, providing a contextualized embedding that differs according to the sentence. This was a direct consequence of BERT's core innovation: true bidirectionality. Earlier language models, including those built on Long short-term memory networks used in Sequence to sequence learning and Neural machine translation, read text in one direction. BERT instead reads the entire sentence at once, letting every word attend to every other word via Attention.
Training: Games of Hide-and-Seek
BERT learns through two clever self-supervised puzzles rather than labeled data. During pre-training, BERT is trained on large-scale unlabeled corpus over two self-supervised tasks: masked language modeling and next sentence prediction. In masked language modeling, BERT predicts randomly masked input tokens. In next sentence prediction, BERT predicts whether two input sentences are consecutive. This training regime devoured a staggering diet of text: the BERT-Base model has 12 layers, 768 hidden units, 12 self-attention heads, and 110 million parameters, pre-trained on the BooksCorpus and English Wikipedia — roughly 3.3 billion words.
Impact and Limitations
Once pre-trained, BERT proved remarkably adaptable. The pre-trained BERT model can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial task-specific architecture modifications. Its real-world reach extended beyond research papers: on October 25, 2019, Google announced that they had started applying BERT models to English-language search queries on Google Search within the US.
Yet BERT's bidirectional gift is also its constraint. Due to its encoder-only architecture lacking a decoder, BERT can't be prompted and can't generate text, while bidirectional models in general do not work effectively without the right side, thus being difficult to prompt. This is why BERT excels at understanding — classification, extraction, similarity — rather than generation, distinguishing its lineage from decoder-based cousins that write prose. Countless descendants and rivals, from RoBERTa to ELMo, trace their family tree back to this single 2018 paper, cementing BERT's place as one of the great convergence points in the History of physics-like arc of modern computing — a moment where architecture, data, and compute aligned.