Recurrence

In neural network architecture, recurrence refers to the mechanism by which a network processes sequential data by feeding outputs back as inputs to the same computational layer. This creates a loop: each step depends on previous steps, allowing the network to maintain memory across time.

Recurrent neural networks (RNNs), including variants like Long short-term memory (LSTM), rely on recurrence to handle temporal dependencies in language, speech, and time-series data. The recurrent connection enables these models to theoretically access arbitrarily distant context—essential for understanding language where meaning often depends on earlier words.

However, recurrence has a critical limitation: it's inherently sequential. Each step must complete before the next begins, making these networks slow to train on long sequences and prone to gradient problems.

The Transformer architecture, introduced in Attention Is All You Need (2017), revolutionized natural language processing by eliminating recurrence entirely. Instead, self-attention mechanisms allow every position in a sequence to directly attend to every other position in parallel. This architectural shift enabled faster training, better scaling, and the emergence of modern large language models like GPT and BERT.

While recurrence is no longer dominant in cutting-edge systems, understanding it remains crucial for comprehending how neural machine translation and earlier sequence models functioned.

Related

Attention, Transformer, Long short-term memory, Self-attention, Neural machine translation