Offensive Security Researcher & Quantum Cryptography Analyst

The Adam Optimizer: How It Works and Why It’s the Default Choice

Abdul Wahab Junaid
What is the Adam optimizer and how does it work

If you’ve trained even a handful of neural networks, you’ve almost certainly typed optimizer = Adam(...) without thinking twice. Adam has become the default optimizer across nearly every deep learning framework and tutorial. But what’s actually happening inside it? In this article, I’ll unpack Adam piece by piece — where it came from, the math that drives it, why it works so well in practice, and where it falls short.

Where Adam Comes From

Adam stands for Adaptive Moment Estimation, introduced by Diederik Kingma and Jimmy Ba in 2014. It was designed to combine the best properties of two earlier optimizers:

  1. Momentum — which smooths the optimization trajectory by accumulating a moving average of past gradients (the “first moment”).
  2. RMSProp — which adapts the learning rate for each parameter individually based on the magnitude of recent squared gradients (the “second moment”).

Adam essentially says: let’s track both of these simultaneously, and use them together to compute a smarter, per-parameter update.

The Two Moving Averages

Adam maintains two exponentially decaying moving averages for every parameter:

First moment estimate (mean of gradients):

$$m_t = \beta_1 m_{t-1} + (1 – \beta_1) g_t$$

Second moment estimate (mean of squared gradients):

$$v_t = \beta_2 v_{t-1} + (1 – \beta_2) g_t^2$$

where $g_t = \nabla_\theta J(\theta_t)$ is the gradient at step $t$, and $\beta_1$, $\beta_2$ are decay rates, typically set to $0.9$ and $0.999$ respectively.

Bias Correction

Since $m_t$ and $v_t$ are initialized at zero, they’re biased toward zero in the early steps of training (especially when $\beta_1$ and $\beta_2$ are close to 1). Adam corrects for this bias explicitly:

$$\hat{m}_t = \frac{m_t}{1 – \beta_1^t}$$

$$\hat{v}_t = \frac{v_t}{1 – \beta_2^t}$$

As $t$ grows large, $\beta_1^t$ and $\beta_2^t$ shrink toward zero, so this correction naturally fades and $\hat{m}_t \to m_t$, $\hat{v}_t \to v_t$.

The Final Update Rule

$$\theta_{t+1} = \theta_t – \eta \cdot \frac{\hat{m}_t}{\sqrt{\hat{v}_t} + \epsilon}$$

where $\eta$ is the base learning rate and $\epsilon$ is a small constant (commonly $10^{-8}$) added purely for numerical stability, to avoid dividing by zero when $\hat{v}_t$ is tiny.

Let’s unpack why this update is so effective:

This means Adam effectively gives each parameter its own adaptive learning rate, scaled inversely to the recent magnitude of its gradients.

Default Hyperparameters

HyperparameterTypical valueRole
$\eta$ (learning rate)0.001Base step size
$\beta_1$0.9Decay rate for first moment (momentum)
$\beta_2$0.999Decay rate for second moment (variance)
$\epsilon$$10^{-8}$Numerical stability constant

These defaults, proposed in the original paper, work remarkably well out of the box across a huge range of problems — a big reason for Adam’s popularity.

Adam Step by Step

flowchart TD
    A["Compute gradients"] --> B["Update first moment estimate"]
    B --> C["Update second moment estimate"]
    C --> D["Apply bias correction"]
    D --> E["Compute parameter update"]
    E --> F["Update model parameters"]
    F --> G{"Training converged?"}
    G -- No --> A
    G -- Yes --> H["Training complete"]

Implementing Adam From Scratch

import numpy as np

def adam(grad_fn, theta_init, lr=0.001, beta1=0.9, beta2=0.999, eps=1e-8, steps=1000):
    theta = theta_init.copy()
    m = np.zeros_like(theta)
    v = np.zeros_like(theta)

    for t in range(1, steps + 1):
        g = grad_fn(theta)
        m = beta1 * m + (1 - beta1) * g
        v = beta2 * v + (1 - beta2) * (g ** 2)

        m_hat = m / (1 - beta1 ** t)
        v_hat = v / (1 - beta2 ** t)

        theta = theta - lr * m_hat / (np.sqrt(v_hat) + eps)

    return theta

In PyTorch, using Adam takes a single line:

import torch

model = torch.nn.Linear(10, 1)
optimizer = torch.optim.Adam(model.parameters(), lr=0.001, betas=(0.9, 0.999), eps=1e-8)

And in TensorFlow/Keras:

import tensorflow as tf

model.compile(
    optimizer=tf.keras.optimizers.Adam(learning_rate=0.001),
    loss='categorical_crossentropy'
)

Advantages of Adam

  1. Per-parameter adaptive learning rates — no need to hand-tune a single global learning rate as carefully as with plain SGD.
  2. Fast convergence — in most cases, Adam reaches a low training loss faster than SGD, especially in early training.
  3. Works well with sparse gradients — a common scenario in NLP tasks with large embedding layers, where only a subset of parameters receive nonzero gradients at each step.
  4. Robust default hyperparameters — the original paper’s suggested defaults work well across a very wide range of problems with little tuning.
  5. Combines the strengths of momentum and adaptive scaling — you get smoothing and per-parameter step-size adaptation simultaneously.

Disadvantages and Limitations

  1. Can generalize worse than SGD — several empirical studies have found that models trained with Adam sometimes achieve lower test accuracy than equivalent models trained with well-tuned SGD + momentum, particularly in computer vision.
  2. Memory overhead — Adam stores two additional state vectors ($m_t$ and $v_t$) per parameter, tripling the memory footprint of the optimizer state compared to plain SGD (which stores none) — relevant for very large models.
  3. Convergence issues in some cases — the original Adam algorithm was shown to fail to converge on certain simple convex problems; this led to the AMSGrad variant, which uses the maximum of past $v_t$ values instead of the exponential moving average to fix the issue.
  4. Sensitive to $\epsilon$ in edge cases — in some settings (particularly mixed-precision training), the default $\epsilon$ can cause numerical issues, requiring adjustment.
  5. Weight decay coupling issue — naive L2 regularization added directly to the gradient interacts oddly with Adam’s adaptive scaling; this motivated the AdamW variant (see below).

Adam vs. AdamW

A subtle but important issue: when you add L2 regularization to a loss function and then optimize with Adam, the regularization term gets divided by $\sqrt{\hat{v}_t} + \epsilon$ just like the primary gradient. This isn’t what you actually want, since weight decay should shrink weights uniformly, not adaptively.

AdamW fixes this by decoupling weight decay from the gradient-based update entirely, applying it directly to the parameters:

$$\theta_{t+1} = \theta_t – \eta \cdot \frac{\hat{m}_t}{\sqrt{\hat{v}_t} + \epsilon} – \eta \lambda \theta_t$$

where $\lambda$ is the weight decay coefficient. AdamW has become the standard choice for training transformers and large language models.

optimizer = torch.optim.AdamW(model.parameters(), lr=0.001, weight_decay=0.01)

Why the Second Moment Matters: An Intuitive Walkthrough

It’s worth slowing down on exactly why dividing by $\sqrt{\hat{v}_t}$ is so useful. Imagine two parameters in a network: parameter $A$ consistently receives large gradients (say, magnitude around 1.0) on every batch, while parameter $B$ receives small, sparse gradients (say, magnitude around 0.01, and only nonzero occasionally). Under vanilla SGD with a single global learning rate, parameter $A$ would move in huge steps while parameter $B$ would barely move at all, even though $B$ might genuinely need larger relative updates to learn effectively.

Adam’s second moment estimate $v_t$ tracks the recent squared-gradient magnitude for each parameter separately. Dividing by $\sqrt{\hat{v}_t}$ effectively normalizes each parameter’s update to a comparable scale: parameter $A$’s large gradients get divided by a large $\sqrt{\hat{v}_t}$, shrinking its effective step; parameter $B$’s small gradients get divided by a small $\sqrt{\hat{v}_t}$, growing its effective step. The net effect is that every parameter tends to move by a roughly similar relative amount per step, rather than being dominated by whichever parameters happen to have the largest raw gradients. This is especially valuable in architectures like embeddings, where most rows of a large embedding table receive gradient updates only rarely.

Adam’s Sensitivity to Its Three Hyperparameters

HyperparameterEffect of increasing itEffect of decreasing it
$\eta$ (learning rate)Faster but riskier convergence, more prone to instabilitySlower but more stable convergence
$\beta_1$Smoother, more momentum-like trajectory, slower to react to gradient direction changesLess smoothing, more reactive to recent gradients, closer to plain adaptive SGD
$\beta_2$Longer memory of past squared gradients, slower adaptation to changing gradient scaleFaster adaptation, but noisier scaling estimates

In practice, $\beta_1$ and $\beta_2$ are rarely tuned away from their defaults (0.9 and 0.999) except in specialized settings — the learning rate $\eta$ remains the primary hyperparameter worth tuning carefully for most tasks.

A Common Misconception: Adam Doesn’t Eliminate the Need for a Learning Rate Schedule

Because Adam already adapts per-parameter step sizes, newcomers sometimes assume it removes the need for a global learning rate schedule entirely. This isn’t quite right — Adam adapts the relative scaling between parameters, but the overall global learning rate $\eta$ still benefits enormously from scheduling (particularly warmup for transformer models, and decay over the course of long training runs). Think of Adam’s adaptive scaling and a learning rate schedule as solving two different, complementary problems: one balances updates across parameters, the other controls the overall step size over time.

Adam vs. Other Optimizers: Comparison Table

OptimizerAdaptive LRMomentumMemory overheadCommon use case
SGDNoOptionalNoneCV models, when generalization matters most
RMSPropYesNo1x extra stateRNNs, older architectures
AdamYesYes2x extra stateDefault for most tasks
AdamWYesYes2x extra stateTransformers, LLMs
AdaGradYesNo1x extra stateSparse data, NLP with rare features

Real-World Use Cases

Best Practices

Frequently Asked Questions

Why does Adam sometimes generalize worse than SGD with momentum, despite converging faster? This is one of the more actively studied puzzles in deep learning optimization. One widely cited explanation is that Adam’s adaptive per-parameter scaling can steer the optimizer toward sharper minima in the loss landscape — regions that fit the training data very well but are more sensitive to small perturbations, which tends to correlate with worse generalization. SGD’s less aggressive, uniform step sizes appear to more often find flatter minima, which tend to generalize better. This isn’t a universal rule, but it’s common enough in computer vision benchmarks that many top-performing CNN training recipes still favor well-tuned SGD with momentum over Adam for final model quality.

Do I need to tune $\beta_1$ and $\beta_2$? Rarely. The default values (0.9 and 0.999) were chosen carefully in the original paper and tend to work robustly across an enormous range of tasks. The learning rate $\eta$ remains, by far, the most important hyperparameter to tune when using Adam. Some specialized settings — such as training with very large batch sizes, or certain reinforcement learning setups with highly non-stationary objectives — occasionally benefit from adjusting $\beta_2$ downward to make the second-moment estimate more reactive, but this is the exception rather than the rule.

Is AdamW strictly better than Adam? For any setting where you’re using weight decay (which is most practical training setups), yes — AdamW’s decoupled weight decay is a strictly more principled implementation of the same underlying idea, and it has become the standard choice in large language model and transformer training precisely for this reason. If you’re not using any weight decay at all, Adam and AdamW behave identically.

Can Adam fail to converge even on simple problems? Yes — this was actually demonstrated formally in the paper that introduced AMSGrad, which constructed simple convex counterexamples where standard Adam fails to converge to the optimal solution due to how quickly its exponential moving average of squared gradients can “forget” large gradients encountered early in training. In practice, this failure mode is rare on typical deep learning loss landscapes, but it’s worth knowing about if you ever observe genuinely strange non-convergent behavior on a well-behaved, simple problem.

Practical Checklist Before Training With Adam

A Quick Mental Model to Keep

Adam is best understood as “momentum plus adaptive per-parameter step sizing, with a correction for early-training bias.” If you’re already comfortable with classical momentum, the leap to understanding Adam is mostly about grasping how the second moment estimate $v_t$ normalizes each parameter’s step individually — everything else is bookkeeping to make sure that normalization behaves well from the very first training step onward.

Summary

Adam works by tracking two exponentially decaying moving averages of the gradient — one estimating its direction (momentum) and one estimating its magnitude (adaptive scaling) — and combining them into a per-parameter adaptive update rule with bias correction for early training steps. This combination gives Adam fast, robust convergence with minimal hyperparameter tuning, which is exactly why it has become the default optimizer across so much of deep learning. That said, it isn’t a universal best choice: for some vision tasks, well-tuned SGD with momentum can still generalize better, and for tasks using weight decay, AdamW is the more principled choice. Understanding the mechanics behind Adam — not just calling it — helps you know when to reach for it and when to reach for something else.

References and Further Reading

Exit mobile version