Popular Activation Functions Used in Neural Networks

For a long time, I thought of activation functions as an afterthought — a small detail you plug in after the “real” work of designing layers and connections. It took building a few networks that mysteriously refused to train before I realized how central activation function choice actually is to whether a network learns at all. In this article, I want to give you the tour I wish I’d had early on: what activation functions are, why they matter, and a practical comparison of the most widely used ones today.

Why Do Neural Networks Need Activation Functions?

Without a non-linear activation function, stacking multiple layers in a neural network would be pointless. Here’s why: if every layer just applies a linear transformation $z = Wx + b$, then stacking two layers gives:

$$z_2 = W_2(W_1 x + b_1) + b_2 = (W_2 W_1)x + (W_2 b_1 + b_2)$$

This is still just a linear function of $x$ — no matter how many linear layers I stack, the result collapses into a single equivalent linear transformation. Activation functions introduce the non-linearity that allows deep networks to approximate arbitrarily complex functions, which is formalized by the Universal Approximation Theorem.

A Map of Common Activation Functions

flowchart TD
    A[Activation Functions] --> B[Saturating / S-shaped]
    A --> C[Rectifier Family]
    A --> D[Output-Layer Specific]
    B --> B1[Sigmoid]
    B --> B2[Tanh]
    C --> C1[ReLU]
    C --> C2[Leaky ReLU / PReLU]
    C --> C3[ELU]
    C --> C4[GELU / Swish]
    D --> D1[Softmax]
    D --> D2["Linear (identity)"]

Sigmoid

$$\sigma(x) = \frac{1}{1+e^{-x}}$$

Squashes inputs into $(0, 1)$, making it ideal for binary classification outputs and gating mechanisms in LSTMs. Its derivative, $\sigma(x)(1-\sigma(x))$, saturates for large positive or negative inputs, causing vanishing gradients when used in deep hidden layers.

Tanh

$$\tanh(x) = \frac{e^x – e^{-x}}{e^x + e^{-x}}$$

Similar to sigmoid but zero-centered, with a range of $(-1, 1)$. This zero-centering historically made it preferable to sigmoid for hidden layers, though it still suffers from vanishing gradients at saturation.

ReLU (Rectified Linear Unit)

$$\text{ReLU}(x) = \max(0, x)$$

The default choice for hidden layers in most modern CNNs and feedforward networks. Its gradient is exactly 1 for all positive inputs, avoiding saturation on that side, but it can suffer from the “dying ReLU” problem where neurons become permanently inactive.

Leaky ReLU and PReLU

$$\text{LeakyReLU}(x) = \begin{cases} x & x > 0 \ \alpha x & x \leq 0 \end{cases}$$

Allows a small gradient for negative inputs (fixed $\alpha$ for Leaky ReLU, learnable for Parametric ReLU), addressing the dying neuron problem.

ELU (Exponential Linear Unit)

$$\text{ELU}(x) = \begin{cases} x & x > 0 \ \alpha(e^x – 1) & x \leq 0 \end{cases}$$

Smooths the negative region with an exponential curve, pushing mean activations closer to zero and often improving convergence speed compared to ReLU.

GELU (Gaussian Error Linear Unit)

$$\text{GELU}(x) = x \cdot \Phi(x)$$

where $\Phi(x)$ is the standard normal CDF. GELU is smooth everywhere (unlike ReLU’s sharp corner at zero) and has become the standard choice in Transformer-based architectures like BERT and GPT.

Swish / SiLU

$$\text{Swish}(x) = x \cdot \sigma(x)$$

Discovered partly through automated architecture search, Swish is smooth, non-monotonic, and has been shown to outperform ReLU in some deep architectures, particularly in image classification models like EfficientNet.

Softmax

$$\text{softmax}(z)i = \frac{e^{z_i}}{\sum{j=1}^{C}e^{z_j}}$$

Not typically used in hidden layers — instead, it’s the standard output-layer activation for multi-class, single-label classification, converting logits into a valid probability distribution.

Comparison Table

Code Example: Comparing Activations in PyTorch

import torch
import torch.nn as nn
import matplotlib.pyplot as plt

x = torch.linspace(-5, 5, 200)

activations = {
    'Sigmoid': torch.sigmoid(x),
    'Tanh': torch.tanh(x),
    'ReLU': torch.relu(x),
    'Leaky ReLU': nn.functional.leaky_relu(x, 0.1),
    'ELU': nn.functional.elu(x),
    'GELU': nn.functional.gelu(x),
    'Swish (SiLU)': nn.functional.silu(x)
}

for name, output in activations.items():
    print(f"{name} at x=2.0: {output[torch.argmin(torch.abs(x-2.0))].item():.4f}")

import torch
import torch.nn as nn
import matplotlib.pyplot as plt

x = torch.linspace(-5, 5, 200)

activations = {
    'Sigmoid': torch.sigmoid(x),
    'Tanh': torch.tanh(x),
    'ReLU': torch.relu(x),
    'Leaky ReLU': nn.functional.leaky_relu(x, 0.1),
    'ELU': nn.functional.elu(x),
    'GELU': nn.functional.gelu(x),
    'Swish (SiLU)': nn.functional.silu(x)
}

for name, output in activations.items():
    print(f"{name} at x=2.0: {output[torch.argmin(torch.abs(x-2.0))].item():.4f}")

Code Example: Building a Network with Different Activations

import torch.nn as nn

model = nn.Sequential(
    nn.Linear(784, 256),
    nn.ReLU(),
    nn.Linear(256, 128),
    nn.GELU(),
    nn.Linear(128, 10)
    # No softmax here if using CrossEntropyLoss — it applies softmax internally
)

import torch.nn as nn

model = nn.Sequential(
    nn.Linear(784, 256),
    nn.ReLU(),
    nn.Linear(256, 128),
    nn.GELU(),
    nn.Linear(128, 10)
    # No softmax here if using CrossEntropyLoss — it applies softmax internally
)

How I Choose an Activation Function

I generally follow this reasoning process when starting a new architecture:

1. Hidden layers in CNNs/feedforward networks → start with ReLU; switch to Leaky ReLU or ELU if I notice dead neurons.
2. Transformer-based architectures → use GELU, following the convention set by BERT and GPT.
3. Recurrent architectures (LSTMs, GRUs) → use the built-in tanh (hidden state) and sigmoid (gates) combination, since these are baked into most framework implementations.
4. Binary classification output → sigmoid.
5. Multi-class, single-label output → softmax.
6. Multi-label output → independent sigmoids per label.
7. Regression output → linear (identity) activation, i.e., no activation function at all on the final layer.

Advantages and Disadvantages Summary

Real-World Use Cases

1. Computer vision — ReLU and its variants dominate CNN architectures like ResNet, VGG, and EfficientNet (which specifically popularized Swish).
2. Natural language processing — GELU is the standard in Transformer models like BERT, GPT, and RoBERTa.
3. Speech and time-series modeling — LSTMs and GRUs rely on tanh and sigmoid internally for their gating mechanisms.
4. Generative modeling — tanh is common in GAN generator outputs; sigmoid appears in GAN discriminator outputs for binary real/fake classification.
5. Tabular data and recommendation systems — deep feedforward networks typically use ReLU or its variants in hidden layers.

Best Practices

• Match initialization to activation: use He/Kaiming initialization with ReLU-family activations, and Xavier/Glorot initialization with sigmoid/tanh.
• Watch for dead neurons when using plain ReLU; monitor activation statistics during training.
• Default to GELU for Transformer architectures and ReLU for CNNs, since these reflect strong empirical consensus in current research.
• Never apply an activation function to regression outputs unless you specifically want to bound the output range.
• Use batch normalization or layer normalization alongside ReLU-family activations to help stabilize training, since these activations are unbounded on the positive side.
• Benchmark activation choices empirically on your specific dataset and architecture — while general guidelines are useful, the best choice can still vary by task.

Summary

Activation functions are what give neural networks their expressive power, transforming simple stacked linear transformations into universal function approximators. Sigmoid and tanh were the historical default but suffer from vanishing gradients in deep networks; ReLU and its many variants (Leaky ReLU, PReLU, ELU, GELU, Swish) have become the modern standard for hidden layers, each balancing computational simplicity against smoothness and gradient behavior differently. Softmax remains the standard for multi-class output layers. Understanding the trade-offs between these functions — saturation, zero-centering, computational cost, and gradient behavior — is essential for designing networks that train efficiently and generalize well.

References

• Nair, V., & Hinton, G.E. (2010). “Rectified Linear Units Improve Restricted Boltzmann Machines.” ICML.
• Hendrycks, D., & Gimpel, K. (2016). “Gaussian Error Linear Units (GELUs).” arXiv:1606.08415.
• Ramachandran, P., Zoph, B., & Le, Q.V. (2017). “Searching for Activation Functions” (Swish). arXiv:1710.05941.
• Goodfellow, I., Bengio, Y., & Courville, A. (2016). “Deep Learning.” MIT Press.
• PyTorch Documentation: https://pytorch.org/docs/stable/nn.html#non-linear-activations-weighted-sum-nonlinearity
• TensorFlow Documentation: https://www.tensorflow.org/api_docs/python/tf/keras/activations