Transformer
The Transformer is a machine learning architecture introduced in 2017 by researchers at Google Brain in the paper Attention Is All You Need. It discarded the recurrent loops that had dominated language modeling for years and replaced them with a single, elegant mechanism: Attention. In doing so, it triggered one of the fastest convergences of ideas in the history of computer science, unifying translation, vision, biology, and chess-playing programs under one architectural roof.
The Problem It Solved
Before the Transformer, the dominant approach to Neural machine translation and other text-generation tasks was Sequence to sequence learning built on Long short-term memory networks. These models read a sentence one word at a time, carrying a hidden memory forward — reliable, but slow to train and prone to "forgetting" distant words, much like a whisper fading down a long hallway. The Transformer's insight was that a model could instead look at every word simultaneously and learn which words matter to which, discarding sequential order in favor of parallel computation.
How It Works
At its core, the Transformer uses self-attention: for every word (or token), the model computes a weighted relationship to every other word in the sentence, allowing it to decide, say, that in "the trophy didn't fit in the suitcase because it was too big," "it" refers to the trophy, not the suitcase. This is done through learned queries, keys, and values — a retrieval-like scheme reminiscent of a networked lookup table. Multiple attention "heads" run in parallel, each attending to different relational patterns, before their outputs are recombined.
Because attention has no inherent sense of order, the architecture injects a positional encoding — a set of sinusoidal signals layered atop word representations, giving each token a kind of address within a Coordinate system of meaning. Words themselves enter the model as word embeddings, continuous vector representations popularized by tools like Word2Vec, which cluster semantically similar words together much as clustering algorithms group related data points.
The full architecture stacks an encoder (which reads input) and a decoder (which generates output), each built from repeated blocks of attention and feed-forward layers, stitched together with residual connections and normalization to keep training stable — a kind of internal responsive stability that lets signals flow through very deep networks without vanishing.
Legacy
The Transformer's parallel-friendly design made it possible to train on unprecedented amounts of text, giving rise to large-scale foundation models like BERT and GPT, and eventually spreading far beyond language into image generation, protein folding, and agent-based decision systems. Few architectural ideas have propagated so quickly across so many disciplines.