If the artificial neuron is the brick of deep learning, backpropagation is the construction crew — the algorithm that actually figures out how to adjust every single weight in a network so it gets better at its task. Without backpropagation, deep learning as we know it simply would not exist; networks with millions or billions of parameters would have no efficient way to learn. This article explains backpropagation from the ground up: the intuition, the calculus, a worked example, and how it’s implemented in modern frameworks.
Table of Contents
- The Problem Backpropagation Solves
- Forward Pass Recap
- The Chain Rule: The Mathematical Core
- Backpropagation Step by Step
- A Worked Numerical Example
- Backpropagation Through a Full Network
- Computational Graphs and Autograd
- Code Example: Manual Backpropagation
- Code Example: Autograd in PyTorch
- Common Issues During Backpropagation
- Backpropagation vs Other Learning Rules
- Best Practices
- Summary and References
1. The Problem Backpropagation Solves
Training a neural network requires answering one core question repeatedly: how much did each individual weight contribute to the final error, and in which direction should it be nudged to reduce that error? With potentially millions of weights spread across dozens of layers, computing this by brute force (perturbing each weight one at a time and re-measuring the loss) would be catastrophically slow. Backpropagation solves this by computing all these contributions — the gradients — in a single efficient backward sweep through the network, using the chain rule of calculus.
2. Forward Pass Recap
Before backpropagation can happen, a forward pass computes the network’s prediction. For a network with layers $1$ through $L$:
$$ z^{(l)} = W^{(l)} a^{(l-1)} + b^{(l)}, \qquad a^{(l)} = \sigma(z^{(l)}) $$
with $a^{(0)} = x$ (the input) and $a^{(L)} = \hat{y}$ (the prediction). The loss is then computed by comparing $\hat{y}$ to the true label $y$:
$$ \mathcal{L} = \mathcal{L}(y, \hat{y}) $$
3. The Chain Rule: The Mathematical Core
Backpropagation is nothing more than a systematic application of the chain rule from calculus. If $\mathcal{L}$ depends on $z^{(l)}$ only through $a^{(l)}$, and $a^{(l)}$ depends on $z^{(l)}$, then:
$$ \frac{\partial \mathcal{L}}{\partial z^{(l)}} = \frac{\partial \mathcal{L}}{\partial a^{(l)}} \cdot \frac{\partial a^{(l)}}{\partial z^{(l)}} = \frac{\partial \mathcal{L}}{\partial a^{(l)}} \cdot \sigma'(z^{(l)}) $$
This local error term is often denoted $\delta^{(l)}$:
$$ \delta^{(l)} = \frac{\partial \mathcal{L}}{\partial z^{(l)}} $$
4. Backpropagation Step by Step
The algorithm proceeds backward from the output layer to the input layer:
Step 1 — Output layer error:
$$ \delta^{(L)} = \nabla_{a^{(L)}} \mathcal{L} \odot \sigma'(z^{(L)}) $$
where $\odot$ denotes element-wise multiplication.
Step 2 — Propagate error backward:
$$ \delta^{(l)} = \left( (W^{(l+1)})^T \delta^{(l+1)} \right) \odot \sigma'(z^{(l)}) $$
Step 3 — Compute gradients for weights and biases:
$$ \frac{\partial \mathcal{L}}{\partial W^{(l)}} = \delta^{(l)} (a^{(l-1)})^T, \qquad \frac{\partial \mathcal{L}}{\partial b^{(l)}} = \delta^{(l)} $$
Step 4 — Update parameters using gradient descent:
$$ W^{(l)} \leftarrow W^{(l)} – \eta \frac{\partial \mathcal{L}}{\partial W^{(l)}}, \qquad b^{(l)} \leftarrow b^{(l)} – \eta \frac{\partial \mathcal{L}}{\partial b^{(l)}} $$
This is repeated layer by layer, moving backward from the output to the input — hence the name “backpropagation.”
graph RL
L[Loss] -->|dL/da_L| A3[Layer 3 Output]
A3 -->|Chain Rule| A2[Layer 2 Output]
A2 -->|Chain Rule| A1[Layer 1 Output]
A1 -->|Chain Rule| X[Input Layer]
A3 -.->|Gradient Update| W3[Weights W3]
A2 -.->|Gradient Update| W2[Weights W2]
A1 -.->|Gradient Update| W1[Weights W1]5. A Worked Numerical Example
Consider a tiny network: one input $x = 2$, one hidden neuron with weight $w_1 = 0.5$, bias $b_1 = 0$, ReLU activation, followed by one output neuron with weight $w_2 = 0.8$, bias $b_2 = 0$, and squared error loss against target $y = 3$.
Forward pass:
$$ z_1 = w_1 x = 0.5 \times 2 = 1.0, \qquad a_1 = \text{ReLU}(1.0) = 1.0 $$
$$ \hat{y} = w_2 a_1 = 0.8 \times 1.0 = 0.8 $$
$$ \mathcal{L} = (y – \hat{y})^2 = (3 – 0.8)^2 = 4.84 $$
Backward pass:
$$ \frac{\partial \mathcal{L}}{\partial \hat{y}} = -2(y – \hat{y}) = -2(3 – 0.8) = -4.4 $$
$$ \frac{\partial \mathcal{L}}{\partial w_2} = \frac{\partial \mathcal{L}}{\partial \hat{y}} \cdot a_1 = -4.4 \times 1.0 = -4.4 $$
$$ \frac{\partial \mathcal{L}}{\partial a_1} = \frac{\partial \mathcal{L}}{\partial \hat{y}} \cdot w_2 = -4.4 \times 0.8 = -3.52 $$
$$ \frac{\partial \mathcal{L}}{\partial w_1} = \frac{\partial \mathcal{L}}{\partial a_1} \cdot \text{ReLU}'(z_1) \cdot x = -3.52 \times 1 \times 2 = -7.04 $$
With a learning rate $\eta = 0.01$, the weights update to $w_2 \approx 0.844$ and $w_1 \approx 0.5704$ — small steps toward reducing the loss, repeated over thousands of iterations across a real dataset.
6. Backpropagation Through a Full Network
In practice, networks have many neurons per layer, so scalars become vectors and matrices, but the principle is identical: propagate the error signal $\delta$ backward, multiplying by local derivatives (activation derivatives and transposed weight matrices) at each step, accumulating the gradient for every parameter along the way.
7. Computational Graphs and Autograd
Modern frameworks never require you to derive these formulas by hand. Instead, they build a computational graph during the forward pass, recording every operation performed. During the backward pass, they traverse this graph in reverse, applying the chain rule automatically — a system called automatic differentiation or autograd. This is what allows arbitrarily complex architectures (Transformers, GANs, custom research models) to be trained without anyone manually deriving gradients for each new design.
8. Code Example: Manual Backpropagation
import numpy as np
# Simple 2-layer network manual backprop
x = np.array([2.0])
y_true = np.array([3.0])
w1, b1 = 0.5, 0.0
w2, b2 = 0.8, 0.0
lr = 0.01
# Forward pass
z1 = w1 * x + b1
a1 = np.maximum(0, z1) # ReLU
y_pred = w2 * a1 + b2
loss = (y_true - y_pred) ** 2
# Backward pass
d_loss_d_ypred = -2 * (y_true - y_pred)
d_ypred_d_w2 = a1
d_loss_d_w2 = d_loss_d_ypred * d_ypred_d_w2
d_ypred_d_a1 = w2
d_a1_d_z1 = (z1 > 0).astype(float) # ReLU derivative
d_loss_d_w1 = d_loss_d_ypred * d_ypred_d_a1 * d_a1_d_z1 * x
# Gradient descent update
w2 -= lr * d_loss_d_w2[0]
w1 -= lr * d_loss_d_w1[0]
print(f"Updated w1: {w1:.4f}, Updated w2: {w2:.4f}, Loss: {loss[0]:.4f}")
9. Code Example: Autograd in PyTorch
import torch
x = torch.tensor([2.0])
y_true = torch.tensor([3.0])
w1 = torch.tensor([0.5], requires_grad=True)
w2 = torch.tensor([0.8], requires_grad=True)
# Forward pass
z1 = w1 * x
a1 = torch.relu(z1)
y_pred = w2 * a1
loss = (y_true - y_pred) ** 2
# Backward pass -- autograd computes all gradients automatically
loss.backward()
print("Gradient w.r.t w1:", w1.grad.item())
print("Gradient w.r.t w2:", w2.grad.item())
Notice how loss.backward() replaces the entire manual derivation from Section 8 — this is the power of automatic differentiation.
10. Common Issues During Backpropagation
- Vanishing gradients: repeated multiplication of small derivatives across many layers shrinks the gradient toward zero.
- Exploding gradients: the opposite problem, where gradients grow uncontrollably.
- Dead ReLU units: if a neuron’s output is always negative before activation, its gradient becomes permanently zero, and it stops learning.
- Incorrect gradient flow in custom operations: when implementing custom layers, a mistake in the backward formula silently corrupts training.
11. Backpropagation vs Other Learning Rules
| Method | How It Assigns Credit | Scalability |
|---|---|---|
| Backpropagation | Exact gradients via chain rule | Scales to billions of parameters |
| Hebbian learning | Local correlation-based updates | Limited, rarely used for deep supervised learning |
| Evolutionary strategies | Random perturbation + selection | Works without gradients, but sample-inefficient |
| Reinforcement learning (policy gradient) | Gradient estimate from sampled rewards | Used when the loss isn’t directly differentiable |
12. Best Practices
- Always verify gradients on a small custom layer using numerical gradient checking before trusting a new implementation.
- Use gradient clipping in recurrent architectures, which are especially prone to exploding gradients.
- Prefer built-in autograd systems over manual gradient derivations for anything beyond toy examples.
- Monitor gradient norms during training to catch vanishing/exploding issues early.
13. A Brief History of Backpropagation
The mathematical idea behind backpropagation — reverse-mode automatic differentiation — dates back further than its popularization in deep learning. It was independently discovered in various forms across the 1960s and 1970s in control theory and applied mathematics. It wasn’t until 1986, when David Rumelhart, Geoffrey Hinton, and Ronald Williams published “Learning representations by back-propagating errors,” that the technique became widely recognized as a practical way to train multi-layer neural networks, directly enabling the field to move past the limitations of the single-layer perceptron.
Despite its importance, backpropagation went through a long period of relative obscurity during the second AI winter, mainly because computers of the era lacked the processing power to train sufficiently deep networks on sufficiently large datasets for its advantages to become obvious. It wasn’t until the 2000s and 2010s, with the arrival of GPUs and big datasets, that backpropagation’s full potential was realized at scale.
14. Backpropagation in Matrix Form for a Full Layer
For clarity, the earlier sections used scalar notation for a single neuron. In practice, entire layers are processed simultaneously using matrix operations for computational efficiency. For a layer with weight matrix $W^{(l)} \in \mathbb{R}^{m \times n}$, the backward pass computes:
$$ \delta^{(l)} = \left((W^{(l+1)})^T \delta^{(l+1)}\right) \odot \sigma'(z^{(l)}) $$
$$ \nabla_{W^{(l)}} \mathcal{L} = \delta^{(l)} (a^{(l-1)})^T $$
$$ \nabla_{b^{(l)}} \mathcal{L} = \delta^{(l)} $$
These matrix formulations are what allow frameworks like PyTorch and TensorFlow to compute gradients for an entire batch of examples simultaneously on a GPU, rather than looping through neurons one at a time — a critical factor in making deep learning computationally practical.
15. Backpropagation Through Time (for Sequential Models)
For recurrent architectures processing sequences, backpropagation must be extended across time steps — a variant called Backpropagation Through Time (BPTT). The network is “unrolled” across all time steps in a sequence, and gradients are propagated backward through this unrolled structure:
$$ \frac{\partial \mathcal{L}}{\partial W} = \sum_{t=1}^{T} \frac{\partial \mathcal{L}_t}{\partial W} $$
This is precisely why recurrent networks are especially prone to vanishing and exploding gradients — the chain rule must be applied repeatedly across potentially hundreds of time steps, compounding the multiplicative effect discussed earlier in this series.
16. Frequently Asked Questions
Do I need to implement backpropagation manually in practice? Almost never for standard architectures. Frameworks like PyTorch, TensorFlow, and JAX implement automatic differentiation, which computes gradients for any composition of differentiable operations automatically. Manual implementation is mainly an educational exercise or occasionally necessary when writing custom low-level operations.
Why is it called “backpropagation” rather than just “gradient computation”? The name reflects the direction of computation: while the forward pass moves from input to output, the gradient computation moves in reverse — from the output layer backward to the input layer — reusing intermediate values computed along the way for efficiency.
Can backpropagation get stuck? Backpropagation itself is just a gradient computation method; it doesn’t “get stuck” by itself. However, the optimization process that uses these gradients (gradient descent) can get stuck in poor regions of the loss landscape, such as saddle points or sharp minima, which is why modern optimizers and learning rate schedules matter so much in practice.
17. Gradient Checking: Verifying Backpropagation Is Correct
When implementing a custom layer or operation, it’s good practice to verify that its backward pass is mathematically correct using numerical gradient checking. This compares the analytically computed gradient against a numerical approximation using finite differences:
$$ \frac{\partial \mathcal{L}}{\partial \theta} \approx \frac{\mathcal{L}(\theta + \epsilon) – \mathcal{L}(\theta – \epsilon)}{2\epsilon} $$
for a small $\epsilon$ (commonly $10^{-5}$ or smaller). If the analytical gradient computed by backpropagation closely matches this numerical approximation, it provides strong evidence that the backward pass implementation is correct. This technique is far too slow to use during actual training (since it requires two forward passes per parameter), but it’s an invaluable one-time debugging tool when writing custom architectures.
def numerical_gradient_check(loss_fn, param, epsilon=1e-5):
original_value = param.item()
param.data = torch.tensor([original_value + epsilon])
loss_plus = loss_fn()
param.data = torch.tensor([original_value - epsilon])
loss_minus = loss_fn()
param.data = torch.tensor([original_value]) # restore
numerical_grad = (loss_plus - loss_minus) / (2 * epsilon)
return numerical_grad
18. Modern Extensions and Alternatives Being Researched
While backpropagation remains the dominant training method, researchers continue to explore alternatives motivated by biological plausibility, memory efficiency, or parallelization concerns:
- Forward-forward algorithm: Proposed by Geoffrey Hinton as a potential alternative that avoids backpropagating gradients through the entire network, instead using two forward passes with different objectives for “positive” and “negative” data.
- Synthetic gradients: Approximates gradients using auxiliary small networks, allowing layers to update without waiting for a full backward pass to complete, potentially improving parallelization across layers.
- Equilibrium propagation: Draws on ideas from physics-based energy models to compute gradients through a more biologically plausible local update rule.
None of these alternatives have yet displaced standard backpropagation in mainstream practice, but they represent active areas of research aimed at addressing some of backpropagation’s practical and biological limitations.
19. Why Understanding Backpropagation Still Matters Today
Even though virtually no practitioner writes backpropagation by hand for standard architectures anymore, understanding the algorithm conceptually remains valuable for several practical reasons: it explains why vanishing and exploding gradients occur, why architectural choices like residual connections and normalization layers help stabilize training, why certain custom layers need careful backward-pass implementations, and why debugging a stalled training run often starts with inspecting gradient magnitudes rather than guessing at data or hyperparameter issues. In short, backpropagation is the mechanism that turns the abstract idea of “learning from data” into a concrete, efficient, and mathematically grounded procedure — and every more advanced training technique in deep learning builds directly on top of it.
19b. One-Sentence Recap
If you take away only one idea from this article, let it be this: backpropagation is simply the chain rule of calculus, applied systematically and efficiently from a network’s output back to its input, to compute exactly how much each individual weight contributed to the final error — nothing more mystical than that, but powerful enough to have made modern deep learning possible.
20. Summary
Backpropagation is the algorithm that makes training deep neural networks computationally feasible. By applying the chain rule systematically from the output layer backward to the input layer, it computes the exact gradient of the loss with respect to every parameter in a single efficient pass, which is then used by an optimizer like SGD or Adam to update weights. Modern frameworks automate this process entirely through computational graphs and automatic differentiation, but understanding the underlying calculus remains essential for debugging, designing custom architectures, and reasoning about training instability.
References
- Rumelhart, D., Hinton, G., & Williams, R. (1986). “Learning representations by back-propagating errors.” Nature.
- Goodfellow, I., Bengio, Y., & Courville, A. — Deep Learning, Chapter 6: https://www.deeplearningbook.org/
- PyTorch Autograd documentation: https://pytorch.org/docs/stable/autograd.html
- Stanford CS231n backpropagation notes: https://cs231n.github.io/optimization-2/