Vision transformer
The Vision Transformer (commonly abbreviated ViT) is a Machine learning model architecture that treats an image not as a grid to be scanned by sliding filters, but as a sentence made of patches — a radical borrowing of ideas from language processing into the world of sight.
Origins
For years, convolutional networks ruled computer vision the way Long short-term memory once ruled language modeling: entrenched, effective, and seemingly irreplaceable. Then came the paper Attention Is All YouEin, properly titled Attention Is All You Need, whose Self-attention mechanism had already toppled recurrent models in Neural machine translation and Sequence to sequence learning. Researchers wondered: if attention could conquer text, could it conquer pixels too? The answer arrived in 2020 in a paper wryly titled An Image is Worth 16x16 Words. The Vision Transformer introduced by Dosovitskiy et al. brought a new approach to how attention-based models could be applied to image classification, at a time when transformers had already become dominant in natural language processing but their potential in computer vision was relatively unexplored.
How it works
A ViT slices an image into fixed-size squares — patches, often literally 16×16 pixels — and flattens each one into a vector, much as Word embedding turns a word into a vector before feeding it to a Transformer. These patch embeddings, augmented with positional information (since attention alone has no innate sense of Space and place or order), stream into a standard transformer encoder built from stacked Attention layers. There is no convolutional kernel sliding across the canvas, no built-in assumption that nearby pixels matter more than distant ones — the network must discover spatial relationships entirely from data, the same way BERT and GPT discover grammar from raw text.
Strengths, quirks, and appetite for data
Because ViT lacks the inductive biases baked into convolutional networks, it tends to be data-hungry: it shines only after pretraining on enormous image datasets, echoing how GPT-3 and T5 needed vast corpora before their abilities emerged. Once past that threshold, however, ViT models often out-perform convolutional rivals and transfer beautifully to new tasks. Their attention maps also offer a strange, almost biological transparency — echoes of how eyes and Cells filter a scene — letting researchers visualize exactly which patches a model "looked at" when making a decision.
Legacy and reach
ViT touched off a wave of hybrid and pure-attention vision architectures, influencing everything from object detection to generative 3D graphics pipelines and multimodal systems that pair pictures with prose. Its success is often cited as evidence of a broader Convergence in AI: that a single architectural idea, Self-attention, can serve as a near-universal engine for sequences of any kind — words, pixels, or patches of light.