Mixture of experts
A mixture of experts (MoE) is a sparse architectural pattern in which a single giant model is quietly replaced by a committee of smaller specialists, only a handful of whom are ever asked to speak at once. Instead of every input trudging through every parameter of a dense network, a lightweight router (or "gating network") reads the input and decides which few "expert" subnetworks are best suited to handle it. The result is a model that can have an enormous total parameter count while keeping the actual computation per input surprisingly small — a trick of scale that has made MoE a favorite ingredient in the recipe for modern large language models.
The core idea
Picture a Transformer block where the usual feed-forward layer is swapped for dozens or hundreds of parallel feed-forward "experts." For each token, the gate computes a score for every expert and activates only the top few — often just one or two. This sparse activation means the model behaves like a vast network of specialists rather than one overworked generalist, echoing the way biological Cells differentiate into specialized roles while remaining part of a single organism. The technique traces its intellectual roots to early 1990s work on adaptive mixtures of local experts, long before it found its true calling in the era of GPT-scale training.
Why sparsity matters
Dense scaling — simply making a network bigger — multiplies both memory and compute cost together. MoE decouples them: total capacity can balloon into the hundreds of billions of parameters while the active compute per token stays modest, similar to how Self-attention let attention-based models discard the sequential bottleneck of Long short-term memory networks. This efficiency has made MoE attractive for pushing past the scaling curves suggested by projects like Chinchilla, since it offers a way to add knowledge capacity without a matching jump in inference cost.
Challenges and quirks
Routing is not free of trouble. Experts can become imbalanced, with the gate developing favorites and starving others of training signal — a load-balancing problem tackled with auxiliary losses that nudge tokens toward underused specialists. Training instability, communication overhead across devices, and the sheer engineering complexity of distributing experts across a model's many hosts remain active areas of research. Some architectures borrow ideas from clustering-like assignment schemes to decide which expert "owns" which kind of input, treating specialization almost like a convergence process toward stable niches.
Legacy and use
MoE layers now appear inside successors to models descended from the BERT and T5 lineage, and are frequently paired with fine-tuning tricks like Low-rank adaptation or alignment techniques such as Reinforcement learning from human feedback used in systems like InstructGPT. Whether guiding text, vision via Vision transformer designs, or multimodal reasoning, the mixture-of-experts idea keeps proving that bigger doesn't have to mean slower — only smarter about which parts of the brain get to wake up.