What Is the Vanishing Gradient Problem in Neural Networks?

For years, one puzzling issue held deep learning back: adding more layers to a network should make it more powerful, but in practice, very deep networks often performed worse than shallower ones, and their early layers barely learned anything at all. The culprit, in most cases, was the vanishing gradient problem — a mathematical quirk in how gradients flow backward through many layers during training.

This article explains exactly what the vanishing gradient problem is, why it happens, how to detect it, and the techniques the deep learning community developed to solve it (which are covered in more depth in the companion article on mitigation techniques).

Table of Contents

1. A Quick Refresher on Backpropagation
2. What Is the Vanishing Gradient Problem?
3. The Math Behind Vanishing Gradients
4. Why Activation Functions Are the Root Cause
5. How to Detect Vanishing Gradients
6. The Related Exploding Gradient Problem
7. Real-World Impact: RNNs and Deep CNNs
8. A Simple Demonstration in Code
9. Consequences for Model Performance
10. Comparison of Activation Functions
11. Best Practices
12. Summary

1. A Quick Refresher on Backpropagation

Neural networks learn by computing the gradient of the loss function with respect to every weight, then updating weights in the direction that reduces loss. This gradient is computed via the chain rule, propagating backward from the output layer to the input layer:

$$ \frac{\partial L}{\partial w_1} = \frac{\partial L}{\partial a_n} \cdot \frac{\partial a_n}{\partial a_{n-1}} \cdots \frac{\partial a_2}{\partial a_1} \cdot \frac{\partial a_1}{\partial w_1} $$

Where $a_1, a_2, \dots, a_n$ are activations at each layer, and $L$ is the loss. Notice this is a product of many terms — one for each layer between the output and the weight being updated.

2. What Is the Vanishing Gradient Problem?

The vanishing gradient problem occurs when these gradients become extremely small as they propagate backward through many layers, to the point where early layers receive essentially no meaningful update signal. Since each weight update is proportional to its gradient:

$$ w_{new} = w_{old} – \eta \cdot \frac{\partial L}{\partial w} $$

If $\frac{\partial L}{\partial w} \approx 0$, then $w_{new} \approx w_{old}$ — the weight barely changes, no matter how many training iterations pass. In deep networks, this effect tends to be worse for layers closer to the input, since their gradients depend on a longer chain of multiplications.

The practical result: the first few layers of a deep network can remain almost randomly initialized even after extensive training, while only the last few layers actually learn.

3. The Math Behind Vanishing Gradients

Consider a deep network using the sigmoid activation function:

$$ \sigma(z) = \frac{1}{1 + e^{-z}} $$

Its derivative is:

$$ \sigma'(z) = \sigma(z)(1 – \sigma(z)) $$

The maximum value of $\sigma'(z)$ occurs at $z=0$, where $\sigma'(0) = 0.25$. For any other input, the derivative is smaller — often much smaller, especially for large positive or negative $z$, where $\sigma'(z)$ approaches 0.

Now consider a network with $n$ layers, all using sigmoid activation. The gradient with respect to an early weight involves a product of $n$ derivative terms, each at most 0.25:

$$ \frac{\partial L}{\partial w_1} \propto \prod_{i=1}^{n} \sigma'(z_i) \leq (0.25)^n $$

For a network with just 10 layers, this upper bound is:

$$ (0.25)^{10} \approx 0.00000095 $$

That’s a gradient roughly a million times smaller than the loss signal at the output layer. With numbers this small, weight updates in early layers become negligible — the network effectively stops learning in those layers.

4. Why Activation Functions Are the Root Cause

The vanishing gradient problem is largely caused by saturating activation functions — functions whose derivative approaches zero for large positive or negative inputs.

flowchart TD
    A[Input Signal] --> B[Layer 1: Sigmoid<br/>Gradient x0.25 max]
    B --> C[Layer 2: Sigmoid<br/>Gradient x0.25 max]
    C --> D[Layer 3: Sigmoid<br/>Gradient x0.25 max]
    D --> E[...]
    E --> F[Layer N: Sigmoid]
    F --> G[Loss]
    G -.Backward Pass.-> F
    F -.Shrinking Gradient.-> D
    D -.Nearly Zero.-> C
    C -.Vanished.-> B

Both the sigmoid and tanh functions saturate — squashing large input ranges into a narrow output range (0 to 1, or -1 to 1), which flattens their slope near the extremes. Since backpropagation multiplies these small derivatives together across many layers, the compounded effect shrinks the gradient toward zero exponentially with depth.

5. How to Detect Vanishing Gradients

Common signs during training include:

• Loss plateaus very early and stops improving, even though the model has plenty of capacity.
• Weights in early layers barely change from their initial values across training.
• Gradient magnitudes, when logged/visualized per layer, are dramatically smaller in early layers compared to later ones.
• The network trains fine when shallow, but adding more layers makes performance worse rather than better.

Most deep learning frameworks let you log gradient norms per layer to diagnose this directly:

import torch

for name, param in model.named_parameters():
    if param.grad is not None:
        print(f"{name}: grad norm = {param.grad.norm().item():.8f}")

import torch

for name, param in model.named_parameters():
    if param.grad is not None:
        print(f"{name}: grad norm = {param.grad.norm().item():.8f}")

If gradient norms decrease by orders of magnitude from the last layer to the first, vanishing gradients are likely occurring.

5b. A Worked Numerical Walkthrough

It helps to see the compounding effect with real numbers rather than just an inequality. Suppose we have a 5-layer network using sigmoid activation, and at each layer the local derivative $\sigma'(z_i)$ happens to evaluate to a fairly typical value of 0.2 (well within sigmoid’s possible range, and often realistic once weights push activations away from zero).

The gradient reaching the first layer, relative to the gradient at the output, is approximately:

$$ \frac{\partial L}{\partial w_1} \propto \prod_{i=1}^{5} \sigma'(z_i) = (0.2)^5 = 0.00032 $$

So a gradient signal of magnitude 1.0 at the output layer arrives at the first layer already shrunk to roughly 0.00032 — a reduction of over 3,000x. Now extend this to a 20-layer network with the same per-layer derivative:

$$ (0.2)^{20} \approx 1.05 \times 10^{-14} $$

At this point the gradient is so close to zero that, in 32-bit floating point arithmetic, it can effectively round down to nothing. This is precisely why early deep learning practitioners found that adding layers beyond a certain depth stopped helping, and often actively hurt performance — the additional layers simply weren’t receiving any usable training signal.

6. The Related Exploding Gradient Problem

The opposite phenomenon, exploding gradients, occurs when gradients grow exponentially instead of shrinking — often in recurrent networks or networks with poorly initialized weights greater than 1 in effective magnitude. This leads to unstable training, wildly oscillating loss, or NaN values.

7. Real-World Impact: RNNs and Deep CNNs

Recurrent Neural Networks (RNNs) are especially vulnerable because they effectively “unroll” into a very deep network across time steps — a sequence of length 100 behaves like a 100-layer feed-forward network during backpropagation through time. This is a major reason plain RNNs struggle to learn long-range dependencies, and why LSTM and GRU architectures (which use gating mechanisms to preserve gradient flow) were developed.

Deep CNNs, before the introduction of residual connections (ResNet, 2015), also suffered from this problem once networks grew beyond roughly 20-30 layers — adding more layers actually hurt accuracy, a phenomenon known as the “degradation problem,” closely tied to vanishing gradients.

8. A Simple Demonstration in Code

The following example illustrates the vanishing gradient problem directly by comparing gradient magnitudes across layers in a deep sigmoid network versus a ReLU network:

import torch
import torch.nn as nn

def build_network(activation, depth=10):
    layers = []
    for _ in range(depth):
        layers.append(nn.Linear(50, 50))
        layers.append(activation())
    layers.append(nn.Linear(50, 1))
    return nn.Sequential(*layers)

x = torch.randn(1, 50)
target = torch.tensor([[1.0]])
loss_fn = nn.MSELoss()

for name, act in [("Sigmoid", nn.Sigmoid), ("ReLU", nn.ReLU)]:
    net = build_network(act)
    output = net(x)
    loss = loss_fn(output, target)
    loss.backward()
    first_layer_grad = net[0].weight.grad.abs().mean().item()
    print(f"{name} network - first layer avg gradient: {first_layer_grad:.10f}")

import torch
import torch.nn as nn

def build_network(activation, depth=10):
    layers = []
    for _ in range(depth):
        layers.append(nn.Linear(50, 50))
        layers.append(activation())
    layers.append(nn.Linear(50, 1))
    return nn.Sequential(*layers)

x = torch.randn(1, 50)
target = torch.tensor([[1.0]])
loss_fn = nn.MSELoss()

for name, act in [("Sigmoid", nn.Sigmoid), ("ReLU", nn.ReLU)]:
    net = build_network(act)
    output = net(x)
    loss = loss_fn(output, target)
    loss.backward()
    first_layer_grad = net[0].weight.grad.abs().mean().item()
    print(f"{name} network - first layer avg gradient: {first_layer_grad:.10f}")

Running this typically shows the sigmoid network’s first-layer gradient several orders of magnitude smaller than the ReLU network’s — a direct, hands-on demonstration of the problem.

8b. Historical Context: Why This Problem Held Back Deep Learning for Years

The vanishing gradient problem was first formally identified by Sepp Hochreiter in his 1991 diploma thesis, though it took years for the broader research community to fully appreciate its implications. Through much of the 1990s and early 2000s, neural networks with more than two or three hidden layers were widely considered impractical to train — not because deeper networks lacked theoretical representational power, but because nobody could get gradients to flow through them reliably.

This is a big part of why, during that period, other machine learning methods — support vector machines, random forests, and boosting methods — dominated many practical applications, while neural networks were seen as a promising but finicky academic curiosity. The turning point came through a combination of the innovations covered in the companion mitigation article: ReLU activations (which don’t saturate for positive inputs), better weight initialization schemes, and eventually architectural innovations like residual connections in 2015, which finally made networks with over 100 layers not just possible, but state-of-the-art. Understanding this history helps explain why “deep” learning only became practically dominant relatively recently, despite the core mathematical ideas being decades old.

9. Consequences for Model Performance

• Slow or stalled training — loss curves flatten out prematurely.
• Poor generalization — the network effectively becomes shallower than intended, since only the last few layers are actually learning.
• Wasted capacity — deep architectures don’t deliver the performance gains they should.
• Difficulty learning long-term dependencies — critical for sequence data like text, speech, and time series.

10. Comparison of Activation Functions

This comparison sets up why activation function choice is one of the primary tools for mitigating vanishing gradients — a topic explored in full detail in the companion article on mitigation techniques.

11. Best Practices

• Prefer ReLU or its variants (Leaky ReLU, ELU) over sigmoid/tanh in hidden layers of deep networks.
• Use proper weight initialization (He initialization for ReLU, Xavier/Glorot for tanh) to keep gradient magnitudes stable from the start.
• Add batch normalization to keep activations in a well-behaved range throughout training.
• For very deep networks, use residual/skip connections so gradients have a direct path backward.
• For sequence models, prefer LSTM or GRU cells over vanilla RNNs, or consider attention-based architectures (Transformers) which avoid the recurrence-based gradient path entirely.
• Monitor gradient norms during training, especially when experimenting with new, deeper architectures.

12. Summary

The vanishing gradient problem is a direct mathematical consequence of the chain rule combined with saturating activation functions: as gradients are multiplied across many layers, they shrink toward zero, leaving early layers of deep networks unable to learn. Understanding why this happens — the repeated multiplication of small derivatives — is the key to understanding why techniques like ReLU activations, careful weight initialization, batch normalization, and residual connections were developed, and why they’ve become standard components of virtually every modern deep learning architecture.

11b. A Practical Diagnostic Checklist

If you suspect vanishing gradients are affecting a model you’re training, working through these questions in order tends to be an efficient way to confirm the diagnosis before reaching for a fix:

1. Is the network unusually deep relative to what’s typical for the task (more than 10-15 layers for a plain feed-forward network, or long sequences for an RNN)?
2. What activation functions are in use? Sigmoid or tanh throughout a deep network is a strong warning sign.
3. Are early-layer weights barely changing across training, when logged or visualized epoch to epoch?
4. Does a shallower version of the same architecture train noticeably better than the deep version, despite having less theoretical capacity? This is a classic symptom.
5. Are gradient norms, when logged per layer, shrinking by orders of magnitude from the output layer back to the input layer?

If the answers point toward vanishing gradients, the companion article on mitigation techniques covers the specific fixes — switching activation functions, adjusting initialization, adding normalization layers, or introducing residual connections — in detail, along with a combined code example showing all of them applied together in a single deep network.

11c. Is This Still a Problem Today?

A fair question, given how thoroughly this article has covered the mechanics of the problem, is whether vanishing gradients are still something practitioners need to worry about in 2026. The honest answer is: much less than a decade ago, but not zero. Standard building blocks in virtually every modern architecture — ReLU-family activations, careful initialization, normalization layers, and residual connections — mean that vanishing gradients rarely derail training in well-designed, mainstream architectures like ResNets or Transformers. However, the problem can still resurface in less common situations: custom architectures that skip these safeguards, extremely deep networks pushed beyond what even residual connections comfortably handle, certain recurrent architectures applied to very long sequences, and research settings exploring genuinely novel network designs where the standard fixes haven’t yet been incorporated. Understanding the underlying mechanism, rather than just memorizing “use ReLU and residual connections,” is what allows you to recognize and diagnose the problem even in unfamiliar architectures where the usual safeguards might be missing or insufficient.

11d. Closing Perspective

The vanishing gradient problem is ultimately a story about how a purely mathematical property — the repeated multiplication of small numbers — can quietly determine whether an entire class of architectures is practically trainable at all. It’s a useful example of why deep learning progress often depends as much on identifying and fixing subtle numerical issues as it does on designing clever high-level architectures. The next time you stack another layer onto a network, or reach for ReLU instead of sigmoid without a second thought, it’s worth remembering that this default reflects years of hard-won understanding about exactly the phenomenon covered in this article.

References

• Hochreiter, S. (1991). Untersuchungen zu dynamischen neuronalen Netzen (foundational thesis identifying the vanishing gradient problem).
• Glorot, X., & Bengio, Y. (2010). Understanding the difficulty of training deep feedforward neural networks. https://proceedings.mlr.press/v9/glorot10a.html
• He, K., et al. (2015). Deep Residual Learning for Image Recognition. https://arxiv.org/abs/1512.03385
• Hochreiter, S., & Schmidhuber, J. (1997). Long Short-Term Memory. https://www.bioinf.jku.at/publications/older/2604.pdf
• PyTorch Documentation — Autograd Mechanics. https://pytorch.org/docs/stable/notes/autograd.html