Convolutions
In neural networks, a convolution is a mathematical operation that applies a small filter (or kernel) across an input to extract local patterns. Convolutions have been foundational to machine learning, especially in vision tasks, where they efficiently detect edges, textures, and features by sharing weights across spatial regions.
Before the Transformer architecture emerged in Attention Is All You Need, convolutions were a dominant paradigm for processing sequential and spatial data. They work by sliding a learned filter over input, computing dot products at each position—a process that, while powerful, introduces computational bottlenecks for long sequences.
The Transformer's breakthrough was architectural: it replaced both recurrent layers (like LSTMs) and convolutional layers with stacked Self-attention mechanisms. This elimination of convolutions allowed the model to capture long-range dependencies more directly and scale efficiently to massive datasets, fundamentally reshaping language modeling.
Today, convolutions remain essential in specialized domains—particularly vision—but the Attention paradigm has become dominant in sequence processing. The shift represents a fundamental rethinking of how neural networks should process information: from local, hierarchical feature extraction to global, context-aware reasoning.