GPT-3
GPT-3 (Generative Pre-trained Transformer 3) is a large language model released by OpenAI in 2020, remembered as the moment the wider world first felt the uncanny fluency of machine-generated text. It is a decoder-only transformer model of deep neural network, which supersedes recurrence and convolution-based architectures with a technique known as attention. That mechanism — Self-attention, built on the ideas from Attention Is All You Need and the broader Transformer architecture — lets the model weigh which earlier words matter most when predicting the next one, an evolution of ideas first explored in Neural machine translation, Sequence to sequence learning, and older Long short-term memory networks.
Scale and Training
GPT-3's defining feature was sheer size. OpenAI researchers developed GPT-3, a language model with 175 billion parameters, significantly larger than its predecessor GPT-2, which had 1.5 billion parameters. The accompanying research paper, released in May 2020, was aptly titled describing the model as a "few-shot learner" — meaning it could perform new tasks from just a handful of examples embedded in its prompt, without retraining. This emergent flexibility surprised many, since earlier NLP systems like BERT typically required fine-tuning for each task. GPT-3 instead absorbed patterns of grammar, reasoning, and even rudimentary coding logic simply by predicting text across an enormous, filtered swath of the internet.
Underneath, its numeric representations of language descend from the lineage of Word2Vec and Word embedding, which taught earlier models to place similar words near one another in a coordinate-like vector space — GPT-3 extends this into deeply contextual embeddings shaped by Attention across long stretches of text.
Capabilities and Cultural Impact
GPT-3 could draft essays, write poetry, summarize documents, answer trivia, translate between languages, and even generate working snippets of computer code. It powered early user interfaces for chatbots, writing assistants, and creative tools, and became a foundation for products across the emerging AI industry. Its outputs sometimes displayed a kind of latent cognitive bias, echoing patterns present in its training data — a reminder that these models are mirrors of human text, not independent minds.
The model's popularity also sparked public wonder and unease in equal measure: wonder at machine-generated prose that felt startlingly human, unease about misinformation, academic dishonesty, and the concentration of power in a few well-resourced labs. GPT-3 is widely seen as a direct ancestor of ChatGPT and the broader wave of generative AI systems that followed, cementing GPT-family models as central figures in the modern history of artificial intelligence — a history some compare, in its rapid pace of discovery, to landmark leaps in History of physics.