Offensive Security Researcher & Quantum Cryptography Analyst

L1 vs. L2 Regularization: What’s the Difference?

Abdul Wahab Junaid
What is the difference between L1 and L2 regularization?

Regularization is one of those topics that seems simple on the surface — “just add a penalty term to the loss” — but the specific shape of that penalty has profound and quite different consequences depending on whether you choose L1 or L2. In this article, I’ll walk through both from first principles, show you the math and the geometry behind why they behave so differently, and help you decide which one fits your situation.

The Basic Idea of Regularization

When training a model, we minimize a loss function $J(\theta)$ that measures prediction error on the training data. Left unconstrained, a sufficiently flexible model (like a large neural network) can drive this training loss arbitrarily close to zero by fitting noise as well as signal — a recipe for overfitting.

Regularization adds a penalty term to the loss that discourages overly complex solutions, typically by discouraging large parameter values:

$$J_{regularized}(\theta) = J(\theta) + \lambda R(\theta)$$

where $R(\theta)$ is the regularization term and $\lambda$ is a hyperparameter controlling how strongly it’s enforced. L1 and L2 regularization differ in exactly how $R(\theta)$ is defined.

L2 Regularization (Ridge / Weight Decay)

L2 regularization adds a penalty proportional to the sum of squared weights:

$$R_{L2}(\theta) = \sum_{i} \theta_i^2 = |\theta|_2^2$$

$$J_{L2}(\theta) = J(\theta) + \lambda \sum_i \theta_i^2$$

Taking the gradient of this penalty with respect to a single weight $\theta_i$ gives:

$$\frac{\partial R_{L2}}{\partial \theta_i} = 2\theta_i$$

So the gradient update becomes:

$$\theta_i \leftarrow \theta_i – \eta \left( \frac{\partial J}{\partial \theta_i} + 2\lambda \theta_i \right) = \theta_i(1 – 2\eta\lambda) – \eta \frac{\partial J}{\partial \theta_i}$$

Notice the term $\theta_i (1 – 2\eta\lambda)$: on every update, the weight is multiplicatively shrunk toward zero by a small factor, in addition to the usual gradient-based update. This is exactly why L2 regularization is often called weight decay — every weight decays a little on every step, proportional to its own current magnitude.

L1 Regularization (Lasso)

L1 regularization adds a penalty proportional to the sum of absolute values of the weights:

$$R_{L1}(\theta) = \sum_i |\theta_i| = |\theta|_1$$

$$J_{L1}(\theta) = J(\theta) + \lambda \sum_i |\theta_i|$$

The gradient of the absolute value function is the sign of the weight (constant magnitude, regardless of how large the weight currently is):

$$\frac{\partial R_{L1}}{\partial \theta_i} = \text{sign}(\theta_i)$$

So the update becomes:

$$\theta_i \leftarrow \theta_i – \eta \left( \frac{\partial J}{\partial \theta_i} + \lambda , \text{sign}(\theta_i) \right)$$

Unlike L2’s proportional shrinkage, L1 subtracts a constant amount from each weight on every step (in the direction that pushes it toward zero), regardless of the weight’s current magnitude. This constant, non-proportional pressure is what causes L1’s most distinctive behavior: it can push small weights all the way to exactly zero, effectively removing them from the model entirely.

The Key Difference: Sparsity

This is the single most important distinction between the two:

Geometric Intuition

A classic way to visualize the difference is to think of the regularization term as a constraint region and the loss function’s contours as ellipses centered on the unregularized optimum.

When you find the point where the loss contours first touch the constraint region, the sharp corners of the L1 diamond sit exactly on the coordinate axes. This geometric property makes it much more likely that the optimal solution touches the constraint region precisely at a corner — where one or more coordinates are exactly zero. The smooth, round L2 circle has no such corners, so the optimal touching point almost never lands exactly on an axis; instead, it shrinks all coordinates toward small but nonzero values.

Side-by-Side Comparison

PropertyL1 RegularizationL2 Regularization
Penalty term$\sum_i \lvert \theta_i \rvert$$\sum_i \theta_i^2$
Gradient of penaltyConstant ($\text{sign}(\theta_i)$)Proportional to $\theta_i$
Effect on weightsDrives many weights to exactly zeroShrinks all weights smoothly, rarely to zero
Resulting modelSparse (fewer active features)Dense (all features retained, but small)
Feature selectionYes, automaticNo
Robustness to outliers in weightsMore robust to a few large weightsPenalizes large weights very heavily (quadratic)
Solution uniquenessCan have multiple equally optimal sparse solutionsTypically a unique, smooth solution
Common nameLassoRidge / weight decay
Differentiability at zeroNot differentiable at $\theta_i = 0$Fully differentiable everywhere

The Bayesian Interpretation: Priors on Weights

There’s an elegant probabilistic way to understand both penalties, which also explains why they produce such different behavior. Regularization can be viewed as placing a prior distribution on the model’s weights and finding the maximum a posteriori (MAP) estimate rather than the pure maximum likelihood estimate.

This Bayesian framing is more than just a mathematical curiosity — it clarifies why the two penalties behave so differently at a deeper level than the optimization mechanics alone.

What Happens as $\lambda$ Increases: A Walkthrough

It helps to trace through what happens to a simple two-weight model as you gradually increase the regularization strength $\lambda$ for each penalty type:

$\lambda$Effect under L2Effect under L1
0 (no regularization)Weights fit training data exactlyWeights fit training data exactly
SmallAll weights shrink slightly, none reach zeroLeast useful weights start approaching zero
ModerateAll weights shrink further, still nonzeroSeveral weights become exactly zero; remaining weights adjust to compensate
LargeWeights shrink dramatically toward (but not exactly) zeroMost weights become exactly zero; only the most predictive few remain
Very largeAll weights nearly zero, severe underfittingAll weights exactly zero, model predicts a constant

Correlated Features: Where the Two Penalties Diverge Most

One of the most practically important differences between L1 and L2 shows up when your input features are highly correlated with each other. Suppose two features are nearly identical (highly correlated) and both are genuinely predictive of the target.

This is precisely the motivation behind Elastic Net, discussed next — it combines L2’s stability under correlated features with L1’s sparsity-inducing behavior.

Elastic Net: Combining Both

Since L1 and L2 each have distinct strengths, Elastic Net regularization combines them in a single weighted penalty:

$$J_{Elastic}(\theta) = J(\theta) + \lambda_1 \sum_i |\theta_i| + \lambda_2 \sum_i \theta_i^2$$

This gives you some of L1’s sparsity-inducing behavior along with L2’s smoother, more stable shrinkage — often useful when you have many correlated features, a scenario where pure L1 can behave somewhat erratically (arbitrarily picking one feature among a correlated group and zeroing out the rest).

Visualizing the Two Penalty Shapes

flowchart LR
    subgraph L1["L1 Penalty (Diamond Constraint)"]
        A1[Sharp corners on axes] --> A2[Solutions often land exactly on axes] --> A3[Produces sparse weights]
    end
    subgraph L2["L2 Penalty (Circular Constraint)"]
        B1[Smooth, round boundary] --> B2[Solutions rarely land on axes] --> B3[Produces small but nonzero weights]
    end

Implementing L1 and L2 in Code

Manually, in a training loop (NumPy-style pseudocode):

def compute_gradient_with_l2(grad_data, theta, lam):
    return grad_data + 2 * lam * theta

def compute_gradient_with_l1(grad_data, theta, lam):
    return grad_data + lam * np.sign(theta)

In PyTorch, L2 regularization is built directly into most optimizers via the weight_decay argument:

import torch

model = torch.nn.Linear(20, 1)
optimizer = torch.optim.SGD(model.parameters(), lr=0.01, weight_decay=1e-4)  # L2

L1 regularization isn’t built in the same way and is typically added manually to the loss:

l1_lambda = 1e-4
l1_penalty = sum(p.abs().sum() for p in model.parameters())
loss = criterion(output, target) + l1_lambda * l1_penalty
loss.backward()

In Keras, both are available directly on layers:

from tensorflow.keras.layers import Dense
from tensorflow.keras.regularizers import l1, l2, l1_l2

Dense(64, activation='relu', kernel_regularizer=l2(0.001))       # L2
Dense(64, activation='relu', kernel_regularizer=l1(0.001))       # L1
Dense(64, activation='relu', kernel_regularizer=l1_l2(l1=0.001, l2=0.001))  # Elastic Net

Advantages and Disadvantages

L1 Regularization

Advantages:

Disadvantages:

L2 Regularization

Advantages:

Disadvantages:

Real-World Use Cases

Best Practices

Frequently Asked Questions

Can I use L1 and L2 regularization together? Yes — this is exactly what Elastic Net does, combining both penalties with separate strength hyperparameters. It’s particularly useful when you want some sparsity but also want to guard against the instability L1 alone can exhibit with correlated features.

Does L1 always produce a sparser model than L2, no matter the setting? In the vast majority of practical cases, yes — this is essentially the defining characteristic of the L1 penalty, rooted in the geometry and gradient behavior described earlier. There are edge cases (e.g., extremely small regularization strength, or unusual loss landscapes) where the difference in sparsity may be negligible, but as a general rule, L1’s sparsity-inducing property is highly reliable.

Why is L2 called “weight decay” in deep learning but not usually called that in classical statistics? The term “weight decay” specifically describes the effect of L2 regularization on the gradient descent update rule — the multiplicative shrinkage applied to weights on every step. In classical statistics and simpler linear models, L2 regularization is more often described in terms of its role in the loss function itself (as “ridge regression”) rather than its effect on an iterative optimization procedure, since many classical methods solve for the regularized solution in closed form rather than through gradient-based iteration.

How do I choose between L1, L2, and Elastic Net for a specific problem? Consider what you value most: if interpretability and automatic feature selection matter most, lean toward L1. If you have many correlated features and want a stable solution, lean toward L2 or Elastic Net. If you’re working with a standard deep neural network without a strong need for sparsity, L2 (weight decay) is almost always the simpler and more standard default.

Summary

L1 and L2 regularization both discourage overly large weights, but they do so in fundamentally different geometric ways: L2 applies a smooth, proportional shrinkage that rarely zeroes out weights entirely, while L1 applies constant pressure that frequently drives weights to exactly zero, producing sparse, more interpretable models. Neither is universally “better” — the right choice depends on whether you want automatic feature selection (L1), smoother and more stable optimization (L2), or a hybrid of both (Elastic Net). Understanding the underlying math and geometry, rather than just treating them as interchangeable “add a penalty” tricks, will help you choose the right one for your specific modeling problem.

References and Further Reading

Exit mobile version