Order Matters
Most of the models we have built so far treat each input as an independent, fixed-size vector. Feed an image to a CNN and it returns a prediction — no knowledge of previous images is required. But a large and important class of data is fundamentally sequential: the meaning of a word depends on the words before it, the value of a sensor reading depends on prior readings, and a musical note makes sense only in the context of the notes that preceded it.
Sequential data includes natural language, time series (stock prices, EEG signals, weather), audio waveforms, video (frames ordered in time), DNA sequences, and event logs. The defining property is that position matters — shuffling the elements destroys meaning. Models that process sequences must somehow capture dependencies that span arbitrary distances across time or position.
Why Feedforward Networks Struggle with Sequences
A standard fully connected network takes a fixed-size input vector and produces a fixed-size output. To process a sequence of length T, the simplest approach is to concatenate all T inputs into one big vector and pass it through a feedforward network. This fails for two reasons. First, sequences vary in length — sentences are not all the same number of words, and time series have different durations. A fixed-size input cannot accommodate variable-length sequences without aggressive and lossy truncation or padding.
Second, and more fundamentally, a feedforward network that sees the entire concatenated input at once cannot share parameters across time steps. The weights applied to position 1 are completely separate from the weights applied to position 5. This means the network must relearn the same concept (e.g., “this word pattern signals negation”) at every position independently, wasting capacity and failing to generalize to positions not seen during training. Parameter sharing across time is the key insight behind recurrent networks.
A common workaround is to use a sliding window: take the last k time steps as input to a feedforward network. This handles variable length (just slide the window) and enforces a form of parameter sharing (the same network processes every window). But it hard-codes the context length k — any dependency longer than k steps is invisible. Language understanding often requires context from hundreds of positions back. Windows also discard information about the exact position of each step relative to the sequence start.
Recurrent Connections: Carrying a Hidden State
The solution is to give the network an internal hidden state ht that acts as a memory. At each time step t, the network reads the current input xt, combines it with the previous hidden state ht−1, and produces a new hidden state ht. This hidden state is a compressed summary of everything the network has seen from position 1 up to position t. Because the same weight matrices are used at every time step, the network shares parameters across all positions automatically.
The output at each step is computed from the hidden state: yt = Wy ht + by. Depending on the task, the network might produce an output at every step (sequence-to-sequence translation), only at the final step (text classification), or at selected steps (named entity recognition).
Visually, an RNN is a network with a loop: the hidden state feeds back into the network at the next time step. This loop is the defining structural feature. When we unroll the loop — draw out each time step as a separate copy of the network, connected in a chain — we see a very deep network: deep in the time dimension rather than in layers. Unrolling makes it clear how gradients must flow backward through all T steps to train the network on long dependencies.
Unfolding Through Time
Unfolding (also called unrolling) rewrites the recurrence as a computational graph that is acyclic. Each time step t gets its own copy of the hidden state ht, its own copy of the input xt, and produces its own output yt. The weight matrices Wh, Wx, and Wy are shared (tied) across all copies — this is what enforces parameter sharing.
Once unrolled, standard backpropagation applies. The total loss is the sum of per-step losses: L = Σt Lt. The gradient of L with respect to the shared weights accumulates contributions from every time step. This algorithm is called Backpropagation Through Time (BPTT).
In practice, full BPTT through very long sequences is computationally expensive and numerically unstable. Truncated BPTT limits gradient propagation to a fixed number of steps back (e.g., 20–100 steps), splitting a long sequence into chunks. Gradients do not cross chunk boundaries, which limits the ability to learn very long-range dependencies but makes training tractable.
The Vanishing Gradient Problem
The gradient flowing backward through k time steps involves a product of k Jacobian matrices, each related to the recurrent weight matrix Wh. When the spectral radius of Wh (its largest singular value) is less than 1, this product shrinks exponentially with k. When it is greater than 1, the product explodes exponentially. Both scenarios destroy learning of long-range dependencies.
Vanishing gradients mean that the gradient signal from a loss at time step T barely reaches time step T−50. The network effectively “forgets” the distant past — it cannot learn that the word at position 1 of a sentence determines the meaning of the phrase at position 50. This is not a numerical precision issue; it is a fundamental structural problem with the simple RNN architecture.
When the recurrent weight matrix has spectral radius greater than 1, gradients grow exponentially with sequence length, causing parameter updates to be catastrophically large — the “exploding gradient” problem. Unlike vanishing gradients, this is detectable (NaN losses, parameter values diverging to infinity) and has a simple fix: gradient clipping caps the gradient norm before the update step. If ‖∇L‖ exceeds a threshold θ, rescale it: ∇L ← θ · ∇L / ‖∇L‖. Gradient clipping is nearly universal in RNN training.
The vanishing gradient problem was diagnosed by Bengio et al. (1994) and Hochreiter (1991), and it motivated the development of Long Short-Term Memory (LSTM) networks, which solve it with a gating mechanism. Before LSTMs dominated, practitioners worked around it using careful weight initialization (identity or orthogonal matrices for Wh), gradient clipping, and very short sequence lengths. These remain useful tricks even with modern architectures.
What Simple RNNs Can Model
Despite the vanishing gradient limitation, simple RNNs can still learn short-range dependencies effectively. Many practical tasks do not require memory spanning hundreds of steps: sentiment classification of a short sentence, next-frame prediction in video, short-horizon time series forecasting. On these tasks, a well-tuned vanilla RNN remains competitive.
The hidden state of a trained RNN develops interpretable structure. In character-level language models, individual hidden units learn to track features like “currently inside a quote,” “indentation depth in code,” or “number of open parentheses.” This emergent memory — without explicit programming — reveals that the recurrent update rule, simple as it is, can extract surprisingly rich sequential structure from data.
The architecture described here — where the hidden state feeds back into the next hidden state — is called an Elman network (Elman, 1990). A related variant, the Jordan network, feeds the output yt back rather than ht. Elman networks are far more common in modern practice because the hidden state is a richer representation than the output, and because Elman networks are easier to train for tasks where the output space is small (e.g., binary classification at the final step).
Sequence-to-Sequence Architectures
Different tasks require different input–output topologies. A one-to-many RNN takes a single vector input (e.g., an image) and produces a sequence output (e.g., a caption) — used in image captioning. A many-to-one RNN reads a sequence and produces a single output (e.g., sentiment label) — the final hidden state encodes the whole sequence. A many-to-many RNN produces an output at every time step — used in language modeling, part-of-speech tagging, and real-time translation.
For tasks where the input and output are sequences of different lengths (translation, summarization), an encoder–decoder structure is standard. The encoder RNN reads the entire input sequence and compresses it into a fixed-size context vector (the final hidden state). The decoder RNN then generates the output sequence conditioned on this context vector. The bottleneck is that the entire input meaning must be compressed into a single vector — a limitation that the attention mechanism (introduced in Lesson 9.1) resolves by allowing the decoder to query all encoder hidden states directly.
- Sequential data (text, time series, audio, video) requires models that respect order and can capture long-range dependencies. Feedforward networks fail because they require fixed-size inputs and cannot share parameters across time steps.
- Recurrent neural networks maintain a hidden state ht updated at every time step: ht = tanh(Whht−1 + Wxxt + b). The same weight matrices are shared across all time steps, encoding the same patterns regardless of position.
- Unfolding the RNN turns the recurrence into an acyclic graph, enabling standard backpropagation. Backpropagation Through Time (BPTT) accumulates gradients from all time steps; truncated BPTT limits this to a fixed number of steps for efficiency.
- Vanishing gradients occur because gradient products over long sequences shrink exponentially when the spectral radius of Wh < 1. Exploding gradients (spectral radius > 1) are handled by gradient clipping. Vanishing gradients motivate LSTM and GRU architectures covered in the next lesson.
- Different tasks use different RNN topologies: one-to-many (image captioning), many-to-one (classification), many-to-many (tagging, LM). Sequence-to-sequence tasks use an encoder–decoder structure where the encoder compresses the input into a context vector for the decoder.