Retrieval-augmented generation
Retrieval-augmented generation (RAG) is a technique for grounding large language models in facts they were never trained on — and, crucially, facts that might not have existed yet when the model was trained. Instead of asking a model to answer purely from the compressed knowledge baked into its weights, a RAG system first fetches relevant documents from an external store — a search index, a database, a folder of PDFs — and then feeds them into the model's context window alongside the question, so the generation step has real evidence to work from rather than a hazy statistical memory.
The idea was formalized in 2020 by researchers who paired a pretrained sequence-to-sequence generator with a neural retriever, showing that Retrieval-Augmented Generation (RAG) is a framework that combines pre-trained AI models with external data retrieval systems to improve the accuracy and relevance of generated responses. This married two lineages of progress: the Transformer architecture's Self-attention mechanism, popularized by the paper Attention Is All You Need, and decades of information-retrieval research on turning text into searchable embeddings — a tradition traceable back to Word2Vec's insight that meaning could live in a geometric space of vectors.
A typical RAG pipeline has two halves. A retriever encodes a user's query and compares it against a vector index of chunked documents, pulling back the most relevant passages — a task conceptually similar to clustering similar items together in a space of meaning. A generator, often a model descended from GPT, T5, or BERT-style Encoder-decoder designs, then reads the retrieved text and produces an answer, ideally citing or paraphrasing it faithfully rather than hallucinating.
Why bother? Because even titanic models trained under Chinchilla-style compute-optimal scaling laws freeze their knowledge at a training cutoff, like a snapshot of the sky from an aging Kepler Space Telescope — useful, but unable to see what's happened since. RAG lets a model stay current without full retraining, answer questions about obscure or private documents, and — ideally — reveal its sources, making it more of a Responsive system than an oracle. It pairs naturally with instruction-tuned models like InstructGPT and techniques such as Reinforcement learning from human feedback, which teach a model to actually use retrieved context rather than ignore it.
Challenges remain lively topics of research: retrievers can fetch irrelevant or contradictory passages, context windows strain under long documents (motivating efficiency tricks like FlashAttention), and multi-hop questions may require iterative, agent-like retrieval loops rather than a single fetch. Some architectures blend RAG with mixture-of-experts routing or lightweight low-rank adaptation fine-tuning to specialize retrieval and generation for a domain without retraining the whole system.