AI & LLM Papers — Reading Map

A guided path through the ideas behind modern large language models, from the sequence models that came before the Transformer to the papers that built today's AI. Tap any link to read that page; tap the orange dashed links to wish new related pages into existence. Follow the sections top to bottom, or jump to whatever you're curious about.

Start here

• Attention Is All You Need — the 2017 paper that introduced the Transformer and set off everything below.
• Transformer — the architecture itself: what it is and why it replaced recurrence.
• Self-attention — the single mechanism at the heart of it all.

Foundations — the world before Transformers

• Recurrent neural network — the original way neural nets processed sequences, one step at a time.
• Long short-term memory — the RNN variant that could remember longer-range context.
• Word embedding and Word2Vec — turning words into vectors, the first "meaning as geometry".
• Sequence to sequence learning — the encoder–decoder idea for mapping one sequence to another.
• Neural machine translation — the task that drove these advances.
• Attention — the mechanism, first bolted onto RNNs, that the Transformer took to its logical extreme.

The model families the Transformer spawned

• BERT — bidirectional pre-training; understanding language.
• GPT → GPT-3 — the generative, autoregressive line that led to ChatGPT.
• T5 — casting every task as text-to-text.
• Large language model — the umbrella idea these all became.
• Vision transformer — the same architecture, now for images.

Scaling and efficiency

• Scaling laws for neural language models — why bigger reliably got better.
• Chinchilla — the correction: compute-optimal training (data matters as much as size).
• Mixture of experts — scaling parameters without scaling compute.
• FlashAttention — making attention fast and memory-efficient.
• Low-rank adaptation — fine-tuning huge models cheaply (LoRA).

Making models useful and aligned

• Reinforcement learning from human feedback — training models on human preferences.
• InstructGPT — the paper that made models follow instructions.
• Chain-of-thought prompting — coaxing step-by-step reasoning out of models.
• Retrieval-augmented generation — grounding answers in retrieved documents.

Related

Attention Is All You Need, Transformer, Large language model, BERT, GPT