Before ReLU took over the deep learning world, tanh was one of my go-to activation functions, especially for recurrent neural networks. I remember being confused about why tanh was preferred over sigmoid in certain layers until I actually plotted both functions side by side and looked at their outputs and gradients. Once I saw the zero-centered output of tanh compared to sigmoid’s always-positive output, the reasoning clicked immediately. Let me walk you through the same realization.
What Problem Does Tanh Solve?
Activation functions exist to introduce non-linearity into neural networks — without them, stacking multiple layers would be mathematically equivalent to a single linear layer, no matter how deep the network is. Tanh is one of the classic non-linear activation functions, and it’s especially useful when I want my activations to be zero-centered, which helps with the dynamics of gradient descent in certain architectures like RNNs and LSTMs.
The Mathematical Definition
The hyperbolic tangent function is defined as:
$$\tanh(x) = \frac{e^{x} – e^{-x}}{e^{x} + e^{-x}}$$
It can also be written in terms of the sigmoid function:
$$\tanh(x) = 2\sigma(2x) – 1$$
where $\sigma$ is the sigmoid function. This relationship is worth remembering — it shows that tanh is really just a rescaled and shifted version of sigmoid.
Key Properties
- Range: $\tanh(x) \in (-1, 1)$
- Zero-centered: $\tanh(0) = 0$
- Odd function: $\tanh(-x) = -\tanh(x)$
- Saturates at both ends: as $x \to \infty$, $\tanh(x) \to 1$; as $x \to -\infty$, $\tanh(x) \to -1$
The Derivative of Tanh
The derivative of tanh, which I need for backpropagation, has a wonderfully simple closed form:
$$\tanh'(x) = 1 – \tanh^2(x)$$
This is convenient computationally because if I’ve already computed $\tanh(x)$ in the forward pass, I can compute the derivative during the backward pass without needing to recompute exponentials.
A Worked Numerical Example
Let’s compute $\tanh(1)$:
$$\tanh(1) = \frac{e^1 – e^{-1}}{e^1 + e^{-1}} = \frac{2.718 – 0.368}{2.718 + 0.368} = \frac{2.350}{3.086} \approx 0.762$$
The derivative at this point:
$$\tanh'(1) = 1 – (0.762)^2 = 1 – 0.581 = 0.419$$
Compare this to $\tanh(3) \approx 0.995$, where the derivative is $1 – 0.990 = 0.010$ — much smaller, illustrating the saturation problem I’ll discuss below.
Tanh vs Sigmoid: Why Zero-Centering Matters
| Property | Sigmoid | Tanh |
|---|---|---|
| Range | $(0, 1)$ | $(-1, 1)$ |
| Zero-centered | No | Yes |
| Formula | $\frac{1}{1+e^{-x}}$ | $\frac{e^x – e^{-x}}{e^x + e^{-x}}$ |
| Derivative | $\sigma(x)(1-\sigma(x))$ | $1 – \tanh^2(x)$ |
| Max gradient | 0.25 | 1.0 |
Sigmoid’s output is always positive, which means that during backpropagation, the gradients for weights feeding into a sigmoid-activated neuron all tend to have the same sign. This causes zig-zagging during gradient descent, which can slow convergence. Tanh’s zero-centered output avoids this issue, which is one of the main reasons it was historically preferred over sigmoid in hidden layers.
Visualizing Where Tanh Fits in a Network
flowchart TD
A[Input x] --> B[Linear Transformation: Wx + b]
B --> C["Tanh Activation: tanh(z)"]
C --> D[Output in Range -1 to 1]
D --> E[Passed to Next Layer or Loss Function]
D --> F["Gradient: 1 - tanh^2(z)"]
F --> G[Backpropagation]
Where Tanh Is Commonly Used
- Recurrent Neural Networks (RNNs) and LSTMs: The candidate cell state and hidden state computations in vanilla RNNs and LSTMs traditionally use tanh, partly because its bounded, zero-centered output helps regulate the magnitude of information flowing through the recurrent connections over many time steps.
- Generative Adversarial Networks (GANs): The generator’s output layer often uses tanh when images are normalized to the range $[-1, 1]$.
- Older feedforward architectures: Before ReLU became standard, tanh was a common hidden-layer activation in fully connected networks.
- Reinforcement learning policy networks: Continuous action outputs are sometimes squashed with tanh to bound them within a valid action range.
Code Example
PyTorch
import torch
import torch.nn as nn
x = torch.tensor([-2.0, -1.0, 0.0, 1.0, 2.0])
tanh_output = torch.tanh(x)
print(f"Tanh output: {tanh_output}")
# As a layer in a model
model = nn.Sequential(
nn.Linear(10, 32),
nn.Tanh(),
nn.Linear(32, 1)
)
# Inside an LSTM cell (built-in, uses tanh internally)
lstm = nn.LSTM(input_size=10, hidden_size=20, batch_first=True)
TensorFlow / Keras
import tensorflow as tf
x = tf.constant([-2.0, -1.0, 0.0, 1.0, 2.0])
tanh_output = tf.math.tanh(x)
print(f"Tanh output: {tanh_output.numpy()}")
model = tf.keras.Sequential([
tf.keras.layers.Dense(32, activation='tanh', input_shape=(10,)),
tf.keras.layers.Dense(1)
])
# GAN generator example with tanh output for images normalized to [-1, 1]
generator_output = tf.keras.layers.Dense(784, activation='tanh')
Advantages of Tanh
- Zero-centered output helps gradient descent converge more efficiently compared to sigmoid, avoiding the zig-zagging issue caused by consistently positive activations.
- Stronger gradients near zero compared to sigmoid (maximum derivative of 1.0 vs sigmoid’s 0.25), which can speed up learning in the early stages of training.
- Bounded output $(-1, 1)$ is useful whenever I need normalized or symmetric outputs, such as generator outputs in GANs or certain RL action spaces.
- Smooth and differentiable everywhere, making it compatible with gradient-based optimization.
Disadvantages and Limitations
- Vanishing gradient problem: For large positive or negative inputs, tanh saturates, and its gradient approaches zero. In deep networks, this can cause gradients to vanish as they’re propagated backward through many layers, slowing or stalling learning.
- Computationally more expensive than ReLU, since it requires computing exponentials rather than a simple max operation.
- Largely replaced by ReLU and its variants in modern deep feedforward and convolutional architectures, due to the vanishing gradient issue.
Tanh vs Other Activation Functions
| Activation | Range | Zero-Centered | Vanishing Gradient Risk | Common Use |
|---|---|---|---|---|
| Tanh | $(-1, 1)$ | Yes | Yes (at extremes) | RNNs, LSTMs, GAN generators |
| Sigmoid | $(0, 1)$ | No | Yes | Binary classification output, gates in LSTMs |
| ReLU | $[0, \infty)$ | No | No (but “dying ReLU” possible) | CNNs, feedforward networks |
| Leaky ReLU | $(-\infty, \infty)$ | No | Reduced | Deep networks avoiding dying ReLU |
| Softmax | $(0,1)$, sums to 1 | No | N/A | Multi-class output layer |
Real-World Use Cases
- Speech and audio modeling — LSTMs using tanh gates for time-series audio processing.
- Language modeling — earlier RNN/LSTM-based language models relied heavily on tanh activations in their recurrent cells.
- Image generation — GAN generators commonly use tanh in the output layer when images are scaled to $[-1, 1]$.
- Financial time-series prediction — RNNs and LSTMs applied to stock or economic data often use tanh in their internal gating mechanisms.
- Robotics and control systems — bounding continuous control outputs to a symmetric range using tanh.
Best Practices
- Use tanh in RNN/LSTM cells where the bounded, zero-centered property helps stabilize long sequences of recurrent computation.
- Normalize inputs before passing through tanh layers to avoid pushing too many activations into the saturated regions.
- Prefer ReLU-family activations for deep feedforward and convolutional networks where vanishing gradients are a bigger concern than zero-centering.
- Pair tanh output layers with data scaled to $[-1, 1]$ in generative models, rather than the raw $[0, 255]$ pixel range.
- Watch for saturation during training by monitoring activation histograms — if most activations cluster near $\pm 1$, gradients will be very small and learning will slow down.
Summary
The tanh activation function squashes inputs into the range $(-1, 1)$ and is zero-centered, which historically made it a better choice than sigmoid for hidden layers. It remains a staple inside recurrent architectures like LSTMs and GRUs, and in generator networks for GANs, thanks to its bounded, symmetric output. However, like sigmoid, it suffers from vanishing gradients at saturation, which is why ReLU and its variants have largely replaced it in deep feedforward and convolutional networks. Knowing where tanh still shines — recurrent cells, bounded outputs, symmetric normalization — helps you use it deliberately rather than by default.
References
- LeCun, Y., Bottou, L., Orr, G.B., & Müller, K.R. (1998). “Efficient BackProp.” Neural Networks: Tricks of the Trade.
- Hochreiter, S., & Schmidhuber, J. (1997). “Long Short-Term Memory.” Neural Computation.
- Goodfellow, I., Bengio, Y., & Courville, A. (2016). “Deep Learning.” MIT Press.
- PyTorch Documentation: https://pytorch.org/docs/stable/generated/torch.nn.Tanh.html
- TensorFlow Documentation: https://www.tensorflow.org/api_docs/python/tf/math/tanh
