GPT
GPT (Generative Pre-trained Transformer) names a lineage of machine learning models that learn to predict the next piece of text and, in doing so, absorb a startling amount of the world's knowledge, style, and reasoning. Built atop the Transformer architecture introduced in the paper Attention Is All You Need, GPT models represent one of the most consequential technical lineages of the early 21st century — a chain of scaled-up experiments that turned "predict the next word" into something resembling conversation, code, and creativity.
Origins
The core idea was to combine two previously separate tricks: unsupervised pre-training on enormous text corpora, and the Self-attention mechanism that lets a model weigh every word against every other word in a passage regardless of distance. OpenAI's key idea was to formalize this as generative pre‑training: first train a large decoder‑only transformer on a broad internet‑scale corpus to model text, then adapt that same model to downstream tasks with minimal additional training. This built on earlier breakthroughs in Word embedding techniques like Word2Vec, and on sequence modeling approaches such as Long short-term memory networks and Sequence to sequence learning, which GPT's parallel, attention-based design decisively outperformed.
Growth by scale
GPT-2, released in 2019, was the first GPT model that truly grabbed global attention, scaling the original GPT-1 architecture from 117 million parameters to 1.5 billion, showing how far simple scaling of a transformer language model could go. GPT-3 followed in 2020, becoming the first model OpenAI commercialized broadly, through the OpenAI API private beta in June 2020. The public explosion came with ChatGPT: the chatbot ChatGPT, released in late 2022 (using GPT-3.5), was followed by many competitor chatbots using their own generative pre-trained transformers to generate text, such as Gemini, DeepSeek and Claude. Later versions extended beyond pure text — GPTs are primarily used to generate text, but can be trained to generate other kinds of data, including images and audio, edging toward genuinely multimodal systems.
How it works, roughly
A GPT model is a stack of decoder blocks, each applying Attention to build contextual representations of a sequence, then predicting probabilities over the next token. Compare its cousin, BERT, which reads context bidirectionally for understanding tasks rather than generation. GPT's lineage also owes a debt to Neural machine translation research, where attention mechanisms first proved their worth translating between languages.
Cultural impact
GPT systems have triggered debate about labor, creativity, misinformation, and Cognitive bias in how people trust machine-generated text. They now function as a kind of Responsive system embedded in countless products, from search to coding assistants, reshaping the User Interface of everyday computing.