Self-attention
Self-attention is a mechanism by which every element of a sequence looks at every other element of the same sequence and decides, mathematically, how much it should care. Where ordinary Attention connects two different sequences — say, a sentence in French glancing sideways at its English translation — self-attention is a sequence turned inward, gossiping with itself, each word asking "which of my neighbors actually matter to me, and by how much?"
The Mechanism
In its canonical form, each position in a sequence is projected into three vectors: a Query, a Key, and a Value. The Query of one position is compared against the Keys of all positions (including itself), producing a score that indicates relevance. These scores are normalized — typically via softmax — into weights, which are then used to blend the Values into a new representation. The elegance lies in its simplicity: no recurrence, no convolution, just weighted averaging guided by learned relevance, computed in parallel across the whole sequence at once.
This parallelism was the mechanism's great gift to the field. Earlier sequence models, such as those built from Long short-term memory units, processed tokens one at a time, like a reader who cannot skip ahead. Self-attention let every token see every other token simultaneously, collapsing the distance between the first and last words of a sentence to a single computational step regardless of length.
Origins and the Transformer
Self-attention existed in scattered forms before its breakout moment, but it was crystallized and popularized by the paper Attention Is All You Need, which introduced the Transformer architecture built almost entirely from self-attention layers stacked with feed-forward networks. The paper's provocative title was also its thesis: recurrence and convolution, long assumed necessary for sequence modeling, could be discarded entirely. This reshaped Neural machine translation, displaced older Sequence to sequence learning designs, and rippled outward into nearly every corner of machine learning that touches sequences — text, code, even pixels arranged as patches.
Multi-Head Attention and Meaning
In practice, self-attention is rarely run once; it is run many times in parallel as "heads," each learning to attend to different kinds of relationships — one head might track grammatical agreement, another long-range thematic connections. This echoes older ideas from Word embedding and Word2Vec, where meaning emerged from context, but self-attention makes that context dynamic and position-aware rather than fixed at training time, often paired with a Coordinate system-like positional encoding so the model knows not just what is nearby but where.
Beyond Language
Though born in text, self-attention's logic — let every part of a system inform every other part, weighted by relevance — has proven strangely universal, echoing patterns found in Networked systems and Agent-based modeling, wherever local elements must somehow achieve global coherence.