Long short-term memory
Long short-term memory (LSTM) is an architecture for recurrent neural networks designed to let sequences remember things across long stretches of time — a kind of engineered patience that ordinary networks sorely lacked. Introduced in 1997 by Sepp Hochreiter and Jürgen Schmidhuber, LSTM solved a problem that had quietly crippled sequence learning for years: when a network tries to learn from data spread far apart in a sequence, the signal used to correct its mistakes tends to shrink into nothing as it travels backward through time.
The vanishing gradient problem
Ordinary recurrent networks update a hidden state at every step, passing information forward like a message repeated down a long line of people — and like that game, the message degrades. During training, the error signal used to adjust weights must flow backward through every one of those steps, and with standard architectures it either vanishes to zero or explodes toward infinity long before it reaches the beginning of a long sequence. This made learning dependencies spanning more than a few dozen steps almost impossible, a serious limitation for tasks involving language, music, or any other unfolding process.
Gates and the cell state
LSTM's fix was architectural rather than merely mathematical: alongside the usual hidden state, it maintains a separate cell state — a kind of conveyor belt that information can ride along with minimal interference. Three learned "gates," small sigmoid-activated switches, regulate this belt. The forget gate decides what old information to discard, the input gate decides what new information to write in, and the output gate decides what to reveal to the rest of the network at that step. Because the cell state's updates are largely additive rather than repeatedly multiplied, gradients can flow across hundreds of steps with far less decay — a structural solution to what had been treated as an inescapable numerical fate.
Legacy and successors
LSTM became the workhorse behind a generation of applications: Speech Recognition, machine translation, handwriting generation, and countless machine learning models operating on Time Series and other sequential data. A simplified cousin, the Gated Recurrent Unit, later offered comparable performance with fewer parameters. Eventually, the field's attention shifted toward architectures described in Attention Is All You Need, which discarded recurrence entirely in favor of parallel attention mechanisms — but LSTM's core insight, that memory needs protected pathways rather than just deeper stacking, remains a foundational idea in how we think about learning over sequences. Like many good ideas in networked and cognitive science, its gates were a small mechanical trick with outsized consequences.