Backpropagation | ML-101

The Credit Assignment Problem

You have a network with millions of weights. You feed it an input, compute a prediction, and measure the error. Now the hard question: which weights are responsible for the error, and by how much should each one change? This is the credit assignment problem — the central challenge of training deep networks.

For the output layer the answer is relatively straightforward: the error depends directly on the output weights. But a weight in the first hidden layer affects the output only indirectly, through every subsequent layer. Attributing blame across many layers, precisely and efficiently, is what backpropagation achieves.

Backpropagation is not a learning algorithm in itself — it is an efficient method for computing gradients. Combined with gradient descent (or its variants), it tells each weight exactly how to change to reduce the loss. The algorithm was known in the 1960s and 1970s but became famous after Rumelhart, Hinton, and Williams popularized it for neural networks in 1986. It remains the engine behind virtually every deep learning system today.

The Chain Rule: Backprop’s Foundation

Backpropagation is simply the chain rule of calculus applied systematically and efficiently to a nested composition of functions. A neural network is exactly such a composition: the loss is a function of the output, which is a function of the last layer, which is a function of the penultimate layer, and so on back to the input.

Chain Rule

\frac{\partial L}{\partial z} = \frac{\partial L}{\partial a} \cdot \frac{\partial a}{\partial z} = \frac{\partial L}{\partial a} \cdot f'(z)

If the loss L depends on z through intermediate variable a = f(z), then the gradient of L with respect to z equals the gradient of L with respect to a, multiplied by the derivative of a with respect to z. The chain rule applies recursively: each layer multiplies the incoming gradient by its own local derivative to pass it further back.

The key insight is that gradients can be computed by passing information backward through the same network graph used for the forward pass. Each node needs only two pieces of information: its own local derivative (computed during the forward pass and stored) and the gradient arriving from the layer above. Multiply them together and pass the result backward. This is both elegant and computationally cheap.

Two Passes: Forward and Backward

Training a neural network with backpropagation involves two sequential passes through the network for each batch of data.

The forward pass computes the prediction. Starting from the input, it applies each layer’s transformation in order, producing activations at every layer. Crucially, every intermediate value — every pre-activation z^[l] and every activation a^[l] — must be cached in memory. These stored values will be needed during the backward pass to compute local derivatives.

The backward pass computes the gradients. Starting from the loss at the output, it propagates the gradient backward layer by layer, all the way to the first hidden layer. At each step, the incoming gradient is used to compute the gradient with respect to that layer’s weights and biases, and then the gradient with respect to the layer’s inputs — which becomes the “incoming gradient” for the previous layer.

Memory Cost

The requirement to cache all intermediate activations during the forward pass is why deep networks are memory-intensive. For a batch of 256 images through a 50-layer network, the cached activations can consume gigabytes of GPU memory. Gradient checkpointing — recomputing some activations during the backward pass rather than storing all of them — trades computation for memory when needed.

The Backprop Equations

Let L be the loss, superscript [l] denote the l-th layer, and L denote the total number of layers. The backward pass computes an error signal δ^[l] for each layer, defined as the partial derivative of the loss with respect to the pre-activation z^[l]. These deltas are then used to compute weight gradients.

Output Layer Delta

\boldsymbol{\delta}^{[L]} = \nabla_{\mathbf{a}^{[L]}} L \odot f'\!\left(\mathbf{z}^{[L]}\right)

The error signal at the output layer is the element-wise product of the gradient of the loss with respect to the output activations and the derivative of the output activation function. For cross-entropy loss with softmax output, this simplifies beautifully to δ^[L] = â − y, where â is the predicted probability vector and y is the one-hot label vector.

Hidden Layer Delta

\boldsymbol{\delta}^{[l]} = \left(W^{[l+1]\top} \boldsymbol{\delta}^{[l+1]}\right) \odot f'\!\left(\mathbf{z}^{[l]}\right)

The error signal at a hidden layer is computed by projecting the next layer’s delta back through the weight matrix (transposed), then multiplying element-wise by the derivative of the activation function at the cached pre-activation. This is the core recurrence: each layer’s delta depends on the delta of the layer above it.

Weight & Bias Gradients

\frac{\partial L}{\partial W^{[l]}} = \boldsymbol{\delta}^{[l]} \mathbf{a}^{[l-1]\top} \qquad \frac{\partial L}{\partial \mathbf{b}^{[l]}} = \boldsymbol{\delta}^{[l]}

The gradient of the loss with respect to the weight matrix W^[l] is the outer product of the layer’s delta and the previous layer’s activations (transposed). The bias gradient is simply the delta itself. These gradients are then used by the optimizer — SGD, Adam, or another variant — to update the weights.

Vanishing and Exploding Gradients

The backward pass multiplies gradients together as it moves through layers. In a deep network, this product of many terms can become catastrophically small or catastrophically large.

Vanishing gradients occur when each layer multiplies the gradient by a number less than 1. The classic culprit is the sigmoid activation: its derivative is at most 0.25 (at z = 0) and falls toward zero for large |z|. After 20 layers of multiplication by 0.25, the gradient reaching the first layer is less than 10⁻¹². The early layers receive essentially no training signal and their weights barely change — the network fails to learn hierarchical representations.

Exploding gradients are the opposite: each layer multiplies the gradient by a number greater than 1, and the gradient grows exponentially. After 20 layers of multiplication by 2, the gradient is over a million. Weight updates become enormous, destabilizing the optimization and often producing NaN values.

Solutions in Practice

Vanishing gradients: Use ReLU activations (gradient is 1 for positive inputs), add skip connections (residual networks), use batch normalization to keep activations in well-behaved ranges, or use architectures like LSTMs for sequences that have learnable gates to control gradient flow.

Exploding gradients: Apply gradient clipping — if the gradient norm exceeds a threshold (commonly 1.0 or 5.0), scale the entire gradient vector down so its norm equals the threshold. This simple trick stabilizes training for recurrent networks and transformers. Careful weight initialization (Xavier/He) also prevents gradients from exploding at the start of training.

Weight Initialization

The starting values of the weights matter enormously. If all weights are initialized to zero, all neurons in a layer compute the same output and receive the same gradient — they are symmetric and will never differentiate. This is the symmetry-breaking problem: random initialization breaks symmetry, allowing different neurons to learn different features.

But random is not enough — the scale matters. If weights are too large, activations saturate; if too small, activations and gradients shrink to zero. Two principled initialization schemes are dominant in practice:

Xavier & He Initialization

W \sim \mathcal{N}\!\left(0,\, \frac{1}{n_{\text{in}}}\right) \text{ (Xavier)} \qquad W \sim \mathcal{N}\!\left(0,\, \frac{2}{n_{\text{in}}}\right) \text{ (He)}

Xavier (Glorot) initialization scales weights by 1/√n_in, keeping activation variance stable through the forward pass for sigmoid/tanh networks. He initialization scales by √(2/n_in), correcting for the fact that ReLU kills half its inputs on average. Using the wrong initialization with the wrong activation is a common source of training instability.

Putting It Together: The Training Loop

With the forward pass, the backward pass, and the weight update rule, the full training algorithm for a mini-batch is:

Mini-Batch Training Loop

1. Sample a mini-batch of B examples from the training set.

2. Forward pass: compute predictions and cache all intermediate activations and pre-activations.

3. Compute loss: evaluate the loss function on the batch predictions vs. true labels.

4. Backward pass: compute δ^[L], then propagate back to compute δ^[l] for every layer and ∇W^[l], ∇b^[l] for every weight matrix and bias.

5. Update weights: apply the optimizer (e.g., W ← W − η∇W). Repeat from step 1.

Each complete pass through the training set is an epoch. Modern networks typically require dozens to hundreds of epochs. The mini-batch size B is a hyperparameter — larger batches give more stable gradient estimates but require more memory; smaller batches introduce noise that can actually help escape local minima.

Key Takeaways

Backpropagation solves the credit assignment problem by computing the gradient of the loss with respect to every weight in the network, efficiently and exactly.
It is the chain rule of calculus applied recursively: each layer multiplies the incoming gradient by its local derivative and passes the result backward.
The forward pass computes and caches all intermediate activations; the backward pass uses these cached values to compute gradients layer by layer in reverse.
The output layer delta is ∇_aL ⊙ f′(z^[L]); for softmax + cross-entropy it simplifies to â − y.
Hidden layer deltas are computed by projecting the next layer’s delta through the transposed weight matrix, then multiplying by the local activation derivative.
Vanishing gradients (caused by saturating activations in deep networks) are addressed by ReLU, skip connections, and batch normalization. Exploding gradients are controlled by gradient clipping.
Weight initialization matters: Xavier for sigmoid/tanh, He for ReLU. Random initialization breaks symmetry; proper scaling keeps gradients well-behaved from the start.