Sigmoid was the very first activation function I learned, back when I was building a simple logistic regression model that I didn’t even realize was a “one-layer neural network.” Its S-shaped curve is one of the most recognizable images in all of machine learning, and understanding exactly why it looks that way — and why it eventually fell out of favor for hidden layers — taught me a lot about how gradients actually behave inside deep networks.
What Problem Does Sigmoid Solve?
The sigmoid function squashes any real-valued input into the range $(0, 1)$. This makes it perfect whenever I need to interpret an output as a probability — for instance, “what is the probability this email is spam?” It was also historically used as a general-purpose non-linearity in hidden layers, though that use case has mostly been replaced by ReLU-family functions today.
The Mathematical Definition
The sigmoid function (also called the logistic function) is defined as:
$$\sigma(x) = \frac{1}{1 + e^{-x}}$$
Key Properties
- Range: $\sigma(x) \in (0, 1)$
- $\sigma(0) = 0.5$
- Monotonically increasing
- Asymptotic: approaches 0 as $x \to -\infty$ and approaches 1 as $x \to \infty$
The Derivative of Sigmoid
One of the reasons sigmoid was so popular historically is its elegant derivative:
$$\sigma'(x) = \sigma(x)(1 – \sigma(x))$$
This means that once I’ve computed $\sigma(x)$ in the forward pass, computing the gradient during backpropagation is just a simple multiplication — no need to recompute exponentials.
A Worked Numerical Example
Let’s compute $\sigma(2)$:
$$\sigma(2) = \frac{1}{1+e^{-2}} = \frac{1}{1+0.135} = \frac{1}{1.135} \approx 0.881$$
The derivative at this point:
$$\sigma'(2) = 0.881 \times (1 – 0.881) = 0.881 \times 0.119 \approx 0.105$$
Now compare this to $\sigma(6) \approx 0.9975$, where the derivative is $0.9975 \times 0.0025 \approx 0.0025$ — dramatically smaller. This illustrates the vanishing gradient problem I’ll cover shortly.
Why Sigmoid Was Historically Popular
- Probabilistic interpretation: Its output naturally maps to a probability, which is exactly what’s needed for binary classification output layers.
- Smooth and differentiable everywhere, making it compatible with gradient-based optimization.
- Biological inspiration: Early neural network research was inspired by the firing behavior of biological neurons, and sigmoid’s smooth “on/off” curve resembled this behavior more than a hard step function would.
Visualizing Where Sigmoid Fits
flowchart TD
A[Input Features] --> B[Neural Network Layers]
B --> C[Output Logit z]
C --> D["Sigmoid: 1 / (1 + e^-z)"]
D --> E[Probability Between 0 and 1]
E --> F[Binary Cross-Entropy Loss]
F --> G["Gradient: y-hat minus y"]
G --> H[Backpropagation]
Code Examples
PyTorch
import torch
import torch.nn as nn
x = torch.tensor([-2.0, -1.0, 0.0, 1.0, 2.0])
sigmoid_output = torch.sigmoid(x)
print(f"Sigmoid output: {sigmoid_output}")
model = nn.Sequential(
nn.Linear(10, 32),
nn.ReLU(),
nn.Linear(32, 1),
nn.Sigmoid() # for binary classification, output a probability
)
# For gates inside LSTM cells, sigmoid is used internally and automatically
lstm_cell = nn.LSTMCell(input_size=10, hidden_size=20)
TensorFlow / Keras
import tensorflow as tf
x = tf.constant([-2.0, -1.0, 0.0, 1.0, 2.0])
sigmoid_output = tf.math.sigmoid(x)
print(f"Sigmoid output: {sigmoid_output.numpy()}")
model = tf.keras.Sequential([
tf.keras.layers.Dense(32, activation='relu', input_shape=(10,)),
tf.keras.layers.Dense(1, activation='sigmoid')
])
model.compile(optimizer='adam', loss='binary_crossentropy')
Where Sigmoid Is Still Used Today
Even though sigmoid has largely been replaced by ReLU-family activations in hidden layers, it remains important in a few specific places:
- Binary classification output layers: Converting a final logit into a probability of the positive class.
- Gating mechanisms in LSTMs and GRUs: The forget gate, input gate, and output gate in an LSTM all use sigmoid to produce values between 0 and 1, representing “how much information to let through.”
- Multi-label classification: Independent sigmoid outputs per label when multiple labels can be true simultaneously for a single example.
- Attention mechanisms in certain architectures: Some gating variants use sigmoid to scale feature importance between 0 and 1.
Advantages of Sigmoid
- Clear probabilistic interpretation, perfect for binary classification and gating mechanisms.
- Smooth, differentiable, and bounded, which is useful when I need an output constrained between 0 and 1.
- Simple, well-understood derivative that reuses the forward pass computation.
Disadvantages and Limitations
- Vanishing gradient problem: As shown in my numerical example, sigmoid’s derivative approaches zero for large positive or negative inputs. In deep networks, gradients passing through many sigmoid layers can shrink to nearly nothing, stalling learning in earlier layers.
- Not zero-centered: Sigmoid’s output is always positive, which can cause the “zig-zagging” gradient descent dynamics discussed in the tanh comparison, since all gradients for weights feeding into the neuron will tend to share the same sign.
- Computationally more expensive than ReLU due to the exponential computation.
- Can cause “saturated” neurons: Neurons whose pre-activation values are consistently very large or very small in magnitude effectively stop learning, since their gradient is near zero.
Sigmoid vs Other Activation Functions
| Activation | Range | Zero-Centered | Vanishing Gradient Risk | Common Use |
|---|---|---|---|---|
| Sigmoid | $(0, 1)$ | No | Yes | Binary classification output, LSTM gates |
| Tanh | $(-1, 1)$ | Yes | Yes | RNN/LSTM hidden states |
| ReLU | $[0, \infty)$ | No | No (dying ReLU possible) | Hidden layers in CNNs/feedforward nets |
| Softmax | $(0,1)$, sums to 1 | No | N/A | Multi-class output layer |
Real-World Use Cases
- Medical diagnosis models — predicting the probability a patient has a particular condition.
- Spam and fraud detection — binary classification tasks outputting a probability of the positive class.
- LSTM-based time-series forecasting — sigmoid gates regulate information flow through memory cells.
- Multi-label image tagging — independent sigmoid outputs per possible tag.
- Recommendation systems — predicting the probability a user clicks or engages with an item.
Best Practices
- Use sigmoid only at output layers for binary or multi-label classification, rather than in deep hidden layers, to avoid vanishing gradients.
- Pair sigmoid with binary cross-entropy, ideally using a combined, numerically stable implementation like
BCEWithLogitsLossin PyTorch. - Initialize weights carefully (e.g., Xavier/Glorot initialization) when using sigmoid, since poor initialization can push activations into the saturated region early in training.
- Monitor activation distributions during training — if most sigmoid outputs cluster near 0 or 1, gradients will be tiny and learning will slow.
- Prefer ReLU-family activations for hidden layers in deep networks, reserving sigmoid for output layers or gating mechanisms.
Summary
The sigmoid activation function squashes any real number into the range $(0, 1)$, giving it a natural probabilistic interpretation that makes it ideal for binary classification outputs and gating mechanisms in LSTMs and GRUs. Its smooth, simple derivative made it historically popular as a general hidden-layer activation, but the vanishing gradient problem and its non-zero-centered output have led modern architectures to prefer ReLU-family activations for hidden layers, reserving sigmoid for the specific tasks where its bounded, probabilistic output is genuinely needed.
References
- Bishop, C.M. (2006). “Pattern Recognition and Machine Learning.” Springer.
- Hochreiter, S., & Schmidhuber, J. (1997). “Long Short-Term Memory.” Neural Computation.
- Goodfellow, I., Bengio, Y., & Courville, A. (2016). “Deep Learning.” MIT Press.
- PyTorch Documentation: https://pytorch.org/docs/stable/generated/torch.nn.Sigmoid.html
- TensorFlow Documentation: https://www.tensorflow.org/api_docs/python/tf/math/sigmoid
