LSTM and GRU | ML-101 | Wireless 101

The Long-Memory Problem

The simple RNN introduced in the previous lesson suffers from a fundamental flaw: gradients vanish exponentially as they travel backward through time, making it nearly impossible to learn dependencies that span more than a few dozen steps. A language model reading a paragraph cannot connect a pronoun on line 10 to the noun it references on line 1. A time-series model cannot remember an event from 500 steps ago that determines the behavior of the current step.

Long Short-Term Memory (LSTM) networks, introduced by Hochreiter and Schmidhuber in 1997, solve this problem with a remarkably elegant architectural addition: a cell state that flows through the network with minimal modification at each step, protected by a set of learned gates. The gates decide what information to forget, what new information to store, and what portion of the memory to expose as output. This design creates “gradient highways” — paths through time where gradients can flow without exponential decay.

The Cell State: A Memory Highway

The key innovation in an LSTM is the separation of two types of state: the cell state c_t and the hidden state h_t. The cell state is the long-term memory. It runs along the top of the LSTM cell, modified only by a few multiplicative gates — not by the full nonlinear transformation that corrupts gradients in a simple RNN. The hidden state is the short-term output, computed from the cell state through a tanh squashing and an output gate.

Mathematically, the cell state is updated by an additive operation: c_t = f_t ⊙ c_t−1 + i_t ⊙ g_t. Here ⊙ denotes element-wise multiplication, f_t is the forget gate (a vector of values in [0,1] that scales down the previous cell state), i_t is the input gate (controls how much new information enters), and g_t is the candidate memory (the proposed new content). Because the update is additive rather than multiplicative-and-nonlinear, gradients can flow backward through c_t without shrinking — the derivative of c_t with respect to c_t−1 is just f_t, a learned value close to 1 for content that should be preserved.

The Three Gates

All three gates have the same structure: a sigmoid-activated linear combination of the current input x_t and the previous hidden state h_t−1. The sigmoid squashes outputs to (0, 1), where 0 means “block completely” and 1 means “pass fully.”

The forget gate decides what fraction of the previous cell state to retain. For a language model tracking grammatical number, the forget gate might reset the number register when a new subject clause begins. The input gate decides how much of the new candidate memory g_t to write into the cell state. The output gate controls what portion of the cell state to expose as the hidden state h_t, which feeds the next step and produces the prediction.

LSTM Gate Equations

\begin{aligned}f_t &= \sigma(W_f[h_{t-1},x_t]+b_f)\\ i_t &= \sigma(W_i[h_{t-1},x_t]+b_i)\\ o_t &= \sigma(W_o[h_{t-1},x_t]+b_o)\\ g_t &= \tanh(W_g[h_{t-1},x_t]+b_g)\end{aligned}

All gates are computed from the same inputs (x_t, h_t−1) with separate learned weight matrices. W_f, W_i, W_o are the forget, input, and output gate weights; W_g is the candidate weight. Each gate vector has the same dimensionality as the hidden state.

Cell State and Hidden State Update

\begin{aligned}c_t &= f_t \odot c_{t-1} + i_t \odot g_t\\ h_t &= o_t \odot \tanh(c_t)\end{aligned}

The cell state c_t is updated additively: old memory is scaled by the forget gate, and new candidate memory is scaled by the input gate. The hidden state h_t is the tanh-squashed cell state, further filtered by the output gate.

One LSTM cell has roughly four times the parameters of a simple RNN of the same hidden size (four separate weight matrices for f, i, o, and g). The computational overhead is real, but the ability to train on sequences with long-range dependencies justifies the cost in almost every practical application.

Why Additive Updates Fix Vanishing Gradients

In a simple RNN, the gradient flows through the tanh nonlinearity and the recurrent weight matrix W_h at every step — a multiplicative chain that shrinks or explodes exponentially. In an LSTM, the gradient of the loss with respect to c_t−1 is multiplied only by the forget gate f_t, a value the network learns to keep near 1 for important memories. This constant error carousel (as Hochreiter called it) allows gradients to flow across hundreds of time steps without decaying.

GRU: A Streamlined Alternative

In 2014, Cho et al. proposed the Gated Recurrent Unit (GRU), a simplified variant that merges the cell state and hidden state into a single state vector, and replaces the three LSTM gates with just two: a reset gate r_t and an update gate z_t.

The update gate z_t plays the combined role of the LSTM’s forget and input gates: it controls how much of the previous hidden state to carry forward versus how much new content to write. The reset gate r_t controls how much of the previous hidden state influences the candidate activation ˜h_t. When r_t is near zero, the GRU forgets the previous state entirely and behaves like a feedforward network for that step — useful for resetting context at sentence boundaries.

GRU Update Equations

\begin{aligned}r_t &= \sigma(W_r[h_{t-1},x_t])\\ z_t &= \sigma(W_z[h_{t-1},x_t])\\ \tilde{h}_t &= \tanh(W_h[r_t\odot h_{t-1},x_t])\\ h_t &= (1-z_t)\odot h_{t-1}+z_t\odot\tilde{h}_t\end{aligned}

The GRU hidden state h_t is a convex combination of the previous state h_t−1 and the candidate activation ˜h_t, weighted by the update gate z_t. When z_t = 1, the state is fully preserved; when z_t = 0, it is fully replaced.

On many benchmark tasks — language modeling, machine translation, speech recognition — GRU and LSTM perform comparably, with neither architecture consistently dominating. The GRU has fewer parameters and runs faster, making it attractive when compute is limited or sequence lengths are moderate. The LSTM’s explicit cell state can give it an edge on tasks requiring very long-range memory. In practice, both architectures are far superior to simple RNNs and the choice between them is often a hyperparameter to tune.

Bidirectional RNNs

A standard RNN processes sequences left to right: at time step t, it can only see x₁ through x_t. But many tasks benefit from knowing the full context in both directions. In the sentence “The bank by the river was steep,” knowing the word “river” to the right of “bank” disambiguates whether “bank” refers to a financial institution or a riverbank.

A bidirectional RNN (BiRNN) runs two separate RNNs over the same sequence: one in the forward direction (left to right) and one in the backward direction (right to left). At each position t, the hidden states of both passes are concatenated: h_t = [&overrightarrow;h_t ; &overleftarrow;h_t]. This doubled-width representation encodes both past and future context, dramatically improving performance on tasks like named entity recognition, machine translation, and BERT-style language understanding.

Bidirectional Constraints

Bidirectional RNNs require the entire input sequence to be available before processing begins — they cannot be used for online (real-time) inference where future tokens are unavailable. For autoregressive generation (language modeling, real-time speech recognition), only the forward-direction RNN is usable. Bidirectional models are standard for encoding tasks (feature extraction, classification, translation encoders) but not for decoding or generation.

Deep RNNs: Stacking Layers

Just as CNNs stack multiple convolutional layers to build hierarchical representations, RNNs can be stacked vertically. In a deep RNN, the hidden state of layer l at time step t, h_t^(l), is fed as input to layer l+1. Each layer operates at the same time resolution, processing the full sequence, so a 3-layer LSTM has three independent sets of recurrent weights, each learning different levels of temporal abstraction.

In practice, 2–4 layers is typical for NLP tasks; going deeper rarely helps for sequence models and can make training harder. Between stacked RNN layers, dropout is applied to the non-recurrent connections (the vertical paths from h_t^(l) to the next layer’s input) to regularize the model. Applying dropout to recurrent connections would disrupt the gradient flow that makes LSTMs effective — a specialized variant called variational dropout applies the same dropout mask at every time step, which is the recommended approach when recurrent dropout is needed.

By the mid-2010s, deep bidirectional LSTMs were the dominant architecture for sequence modeling, achieving state-of-the-art results on translation, speech recognition, and NLP benchmarks. They remained the standard until the Transformer architecture (covered in Module 9) overtook them in 2017 by replacing recurrence entirely with self-attention — a mechanism that can connect any two positions in a sequence in a single computation step, regardless of distance.

LSTM vs. GRU vs. Transformer

LSTMs and GRUs solve the vanishing gradient problem but are still inherently sequential — step t cannot be computed until step t−1 is done. This limits parallelism during training. Transformers break this bottleneck by computing all positions simultaneously with self-attention, enabling massive parallelism on modern GPUs. For most tasks where the full sequence is available (NLP, time series analysis), Transformers have largely superseded LSTMs. But for streaming data (online speech, IoT sensors) and tasks with strict latency constraints, LSTMs and GRUs remain practical choices because they process one step at a time with bounded compute.

Key Takeaways

LSTMs solve the vanishing gradient problem by introducing a cell state c_t that flows through the network with additive updates, protected by three learned gates (forget, input, output). The additive update creates gradient highways that allow learning across hundreds of steps.
The forget gate scales down old cell state content; the input gate controls how much new candidate memory to write; the output gate filters the cell state to produce the hidden state. All gates are sigmoid-activated linear combinations of x_t and h_t−1.
GRUs simplify the LSTM to two gates (reset and update), merging the cell and hidden state into one. They have fewer parameters, train faster, and perform comparably to LSTMs on most benchmarks. Choice between LSTM and GRU is typically a hyperparameter to tune.
Bidirectional RNNs run two passes — forward and backward — over the input and concatenate the hidden states, giving each position access to both past and future context. They require the full sequence upfront, so they cannot be used for real-time generation.
Deep RNNs stack 2–4 recurrent layers to build hierarchical temporal representations. Dropout between layers (not on recurrent connections) regularizes deep RNNs. The Transformer architecture later superseded deep bidirectional LSTMs for offline tasks by computing all positions in parallel.