Offensive Security Researcher & Quantum Cryptography Analyst

What Is the Purpose of the Sigmoid Activation Function

Abdul Wahab Junaid
What is the purpose of the sigmoid activation function

Sigmoid was the very first activation function I learned, back when I was building a simple logistic regression model that I didn’t even realize was a “one-layer neural network.” Its S-shaped curve is one of the most recognizable images in all of machine learning, and understanding exactly why it looks that way — and why it eventually fell out of favor for hidden layers — taught me a lot about how gradients actually behave inside deep networks.

What Problem Does Sigmoid Solve?

The sigmoid function squashes any real-valued input into the range $(0, 1)$. This makes it perfect whenever I need to interpret an output as a probability — for instance, “what is the probability this email is spam?” It was also historically used as a general-purpose non-linearity in hidden layers, though that use case has mostly been replaced by ReLU-family functions today.

The Mathematical Definition

The sigmoid function (also called the logistic function) is defined as:

$$\sigma(x) = \frac{1}{1 + e^{-x}}$$

Key Properties

The Derivative of Sigmoid

One of the reasons sigmoid was so popular historically is its elegant derivative:

$$\sigma'(x) = \sigma(x)(1 – \sigma(x))$$

This means that once I’ve computed $\sigma(x)$ in the forward pass, computing the gradient during backpropagation is just a simple multiplication — no need to recompute exponentials.

A Worked Numerical Example

Let’s compute $\sigma(2)$:

$$\sigma(2) = \frac{1}{1+e^{-2}} = \frac{1}{1+0.135} = \frac{1}{1.135} \approx 0.881$$

The derivative at this point:

$$\sigma'(2) = 0.881 \times (1 – 0.881) = 0.881 \times 0.119 \approx 0.105$$

Now compare this to $\sigma(6) \approx 0.9975$, where the derivative is $0.9975 \times 0.0025 \approx 0.0025$ — dramatically smaller. This illustrates the vanishing gradient problem I’ll cover shortly.

Why Sigmoid Was Historically Popular

  1. Probabilistic interpretation: Its output naturally maps to a probability, which is exactly what’s needed for binary classification output layers.
  2. Smooth and differentiable everywhere, making it compatible with gradient-based optimization.
  3. Biological inspiration: Early neural network research was inspired by the firing behavior of biological neurons, and sigmoid’s smooth “on/off” curve resembled this behavior more than a hard step function would.

Visualizing Where Sigmoid Fits

flowchart TD
    A[Input Features] --> B[Neural Network Layers]
    B --> C[Output Logit z]
    C --> D["Sigmoid: 1 / (1 + e^-z)"]
    D --> E[Probability Between 0 and 1]
    E --> F[Binary Cross-Entropy Loss]
    F --> G["Gradient: y-hat minus y"]
    G --> H[Backpropagation]

Code Examples

PyTorch

import torch
import torch.nn as nn

x = torch.tensor([-2.0, -1.0, 0.0, 1.0, 2.0])
sigmoid_output = torch.sigmoid(x)
print(f"Sigmoid output: {sigmoid_output}")

model = nn.Sequential(
    nn.Linear(10, 32),
    nn.ReLU(),
    nn.Linear(32, 1),
    nn.Sigmoid()  # for binary classification, output a probability
)

# For gates inside LSTM cells, sigmoid is used internally and automatically
lstm_cell = nn.LSTMCell(input_size=10, hidden_size=20)

TensorFlow / Keras

import tensorflow as tf

x = tf.constant([-2.0, -1.0, 0.0, 1.0, 2.0])
sigmoid_output = tf.math.sigmoid(x)
print(f"Sigmoid output: {sigmoid_output.numpy()}")

model = tf.keras.Sequential([
    tf.keras.layers.Dense(32, activation='relu', input_shape=(10,)),
    tf.keras.layers.Dense(1, activation='sigmoid')
])
model.compile(optimizer='adam', loss='binary_crossentropy')

Where Sigmoid Is Still Used Today

Even though sigmoid has largely been replaced by ReLU-family activations in hidden layers, it remains important in a few specific places:

  1. Binary classification output layers: Converting a final logit into a probability of the positive class.
  2. Gating mechanisms in LSTMs and GRUs: The forget gate, input gate, and output gate in an LSTM all use sigmoid to produce values between 0 and 1, representing “how much information to let through.”
  3. Multi-label classification: Independent sigmoid outputs per label when multiple labels can be true simultaneously for a single example.
  4. Attention mechanisms in certain architectures: Some gating variants use sigmoid to scale feature importance between 0 and 1.

Advantages of Sigmoid

Disadvantages and Limitations

Sigmoid vs Other Activation Functions

ActivationRangeZero-CenteredVanishing Gradient RiskCommon Use
Sigmoid$(0, 1)$NoYesBinary classification output, LSTM gates
Tanh$(-1, 1)$YesYesRNN/LSTM hidden states
ReLU$[0, \infty)$NoNo (dying ReLU possible)Hidden layers in CNNs/feedforward nets
Softmax$(0,1)$, sums to 1NoN/AMulti-class output layer

Real-World Use Cases

  1. Medical diagnosis models — predicting the probability a patient has a particular condition.
  2. Spam and fraud detection — binary classification tasks outputting a probability of the positive class.
  3. LSTM-based time-series forecasting — sigmoid gates regulate information flow through memory cells.
  4. Multi-label image tagging — independent sigmoid outputs per possible tag.
  5. Recommendation systems — predicting the probability a user clicks or engages with an item.

Best Practices

Summary

The sigmoid activation function squashes any real number into the range $(0, 1)$, giving it a natural probabilistic interpretation that makes it ideal for binary classification outputs and gating mechanisms in LSTMs and GRUs. Its smooth, simple derivative made it historically popular as a general hidden-layer activation, but the vanishing gradient problem and its non-zero-centered output have led modern architectures to prefer ReLU-family activations for hidden layers, reserving sigmoid for the specific tasks where its bounded, probabilistic output is genuinely needed.

References

Exit mobile version