Reinforcement learning from human feedback
Reinforcement learning from human feedback (RLHF) is a training technique that teaches a Machine learning model to behave the way people actually want it to, rather than merely the way its training data predicts. It is the method most credited with turning raw, sometimes erratic Large language models into the helpful, conversational assistants many now take for granted — the invisible finishing school between a model that merely completes text and one that converses.
The core idea traces back to earlier work applying reinforcement learning to preferences over trajectories in game-playing and robotics agents, before being adapted to text. In language modeling, RLHF became famous through OpenAI's InstructGPT, a version of GPT-3 fine-tuned to follow user instructions using human feedback. Its training pipeline combined supervised fine-tuning on a small set of human-written demonstrations and PPO-based RLHF using a reward model trained from labeler rankings of model outputs.
The process unfolds in three stages. First, a pretrained GPT-style Transformer undergoes supervised fine-tuning on human-written demonstrations of ideal behavior. Second, human annotators compare pairs of model outputs and indicate which they prefer; these rankings train a separate reward model to predict human judgment as a numeric score. A pretrained large language model is first used to generate multiple candidate responses for each user prompt, and human annotators then compare these responses, often in pairs, indicating which one they prefer or assigning scalar scores. Third, the original model is treated as an agent whose "actions" are generated tokens, and it is optimized — typically with Proximal Policy Optimization (PPO) — to maximize the reward model's score, nudging its outputs ever closer to what humans actually favor.
Why bother, when Self-attention and Attention mechanisms already let models like BERT and T5 absorb vast quantities of text? Because next-token prediction alone rewards fluency, not helpfulness, honesty, or safety — a model can be eloquently wrong or politely useless. RLHF injects a human feedback loop directly into the objective function, turning subjective values like tact and truthfulness into a trainable signal.
RLHF is not without critics. Reward models can be gamed by the very policy they train ("reward hacking"), annotator biases seep into the system, and PPO's instability has spurred simpler alternatives like Direct Preference Optimization, which skips the explicit reward model entirely. Related efforts, such as Anthropic's Constitutional AI, attempt to reduce reliance on human labelers by having models critique themselves against written principles.
Despite its costs — armies of human raters, delicate optimization, and philosophical questions about whose preferences get encoded — RLHF remains the dominant paradigm for converging powerful Large language models toward genuinely useful, human-aligned behavior.