Offensive Security Researcher & Quantum Cryptography Analyst

The Role of L1 and L2 Regularization in Neural Networks

Abdul Wahab Junaid
What is the role of L1 and L2 regularization in neural networks

Neural networks, especially large ones, are almost absurdly good at memorizing whatever you feed them. Give a big enough network enough epochs on a modestly-sized dataset, and it will often drive training loss to near zero — including fitting the noise, mislabeled examples, and quirks specific to that exact dataset. L1 and L2 regularization exist specifically to push back against this tendency. In this article, I want to focus specifically on the role these penalties play inside neural network training — not just the abstract math, but how they actually function within the architecture, the optimization process, and the resulting model behavior.

Why Neural Networks Need Regularization at All

A modern neural network can easily have millions or billions of parameters. The number of parameters often vastly exceeds the number of training examples, meaning the model has more than enough capacity to fit the training data exactly, including noise, rather than learning the true underlying signal. This is the essence of overfitting: excellent training performance paired with poor performance on new, unseen data.

Regularization techniques constrain the effective capacity of the model without necessarily changing its architecture, by discouraging solutions where individual weights become unnecessarily large or numerous.

Where L1 and L2 Fit Into the Loss Function

In a neural network, the total loss being minimized during training becomes:

$$J_{total}(\theta) = J_{data}(\theta) + \lambda_1 \sum_i |\theta_i| + \lambda_2 \sum_i \theta_i^2$$

where $J_{data}(\theta)$ is the task loss (e.g., cross-entropy for classification, mean squared error for regression), and the two summation terms are the L1 and L2 penalties respectively, each with its own strength hyperparameter. Most practical setups use only one of the two (commonly L2), though both can be combined as an Elastic Net-style penalty if desired.

Critically, this penalty is typically applied to the network’s weight matrices (the parameters connecting layers), not usually to bias terms, since biases don’t contribute to model complexity in the same way weights do — regularizing them yields little benefit and can slightly hurt performance.

The Role of L2 Regularization (Weight Decay) in Neural Networks

L2 regularization is by far the more commonly used of the two in deep learning, almost always under the name weight decay. Its role in a neural network is to:

  1. Prevent any single weight from growing excessively large, which would make the model’s output overly sensitive to small changes in specific input features.
  2. Encourage the network to distribute its “reliance” across many weights, rather than concentrating predictive power in just a few, which tends to make representations more robust and generalizable.
  3. Smooth the loss landscape, since the added quadratic term is convex and well-behaved, making optimization more numerically stable.
  4. Act as an implicit constraint on model complexity, effectively reducing the “effective” number of parameters the model can meaningfully use, even though the literal parameter count doesn’t change.

In modern training loops, weight decay is often applied directly by the optimizer rather than by adding an explicit term to the loss function. Recall the L2 gradient update:

$$\theta_i \leftarrow \theta_i(1 – \eta\lambda) – \eta\frac{\partial J_{data}}{\partial \theta_i}$$

This shows the weight decay effect directly: every single weight is multiplied by a factor slightly less than 1 on every step, in addition to the usual gradient-based adjustment, continuously pulling weights toward zero unless the data gradient consistently pushes back.

The Role of L1 Regularization in Neural Networks

L1 regularization is used less frequently in deep learning than L2, but it has a specific and valuable role: encouraging sparsity in the network’s weights. Its role includes:

  1. Automatic feature/connection pruning — weights that don’t meaningfully contribute to reducing the task loss get pushed all the way to exactly zero, effectively removing those connections from the network.
  2. Model compression — a sparse weight matrix (with many exact zeros) can be stored and computed more efficiently using sparse matrix representations, which is valuable for deploying models on resource-constrained devices.
  3. Interpretability — in smaller networks or the first layer of a network processing structured/tabular data, L1-induced sparsity can highlight which specific input features the model actually relies on.

L1 is less commonly applied throughout an entire deep network’s hidden layers (since sparsity in the middle of a network can sometimes disrupt gradient flow and hurt training dynamics), but it’s occasionally used on the first layer for feature selection, or on the entire network when compression/sparsity is a specific design goal.

Visualizing Where Regularization Acts

flowchart TD
    A[Input layer] --> B[Hidden layer weights: W1]
    B --> C[Hidden layer weights: W2]
    C --> D[Output layer weights: W3]
    D --> E[Task loss: cross-entropy / MSE]
    B -.L2 penalty on W1.-> F[Total loss]
    C -.L2 penalty on W2.-> F
    D -.L2 penalty on W3.-> F
    E --> F[Total regularized loss]

Weight Decay in Practice: A Subtlety With Adaptive Optimizers

An important, often overlooked detail: when you use weight decay with adaptive optimizers like Adam, naively adding the L2 penalty to the gradient interacts oddly with Adam’s per-parameter adaptive scaling. The regularization term gets divided by the same $\sqrt{\hat{v}_t}$ term as the data gradient, which distorts its intended uniform shrinking effect. This led to the development of AdamW, which decouples weight decay from the gradient-based update entirely:

$$\theta_{t+1} = \theta_t – \eta \cdot \frac{\hat{m}_t}{\sqrt{\hat{v}_t} + \epsilon} – \eta\lambda\theta_t$$

If you’re training with Adam and want proper L2-style regularization, AdamW is almost always the better choice over adding an L2 term manually to the loss.

import torch

model = torch.nn.Sequential(
    torch.nn.Linear(784, 256),
    torch.nn.ReLU(),
    torch.nn.Linear(256, 10)
)

# Proper decoupled weight decay
optimizer = torch.optim.AdamW(model.parameters(), lr=1e-3, weight_decay=1e-2)

Applying L1 Regularization to a Neural Network

Since most frameworks don’t build L1 into the optimizer directly, it’s typically added explicitly to the loss:

import torch

model = torch.nn.Sequential(
    torch.nn.Linear(784, 256),
    torch.nn.ReLU(),
    torch.nn.Linear(256, 10)
)
optimizer = torch.optim.SGD(model.parameters(), lr=0.01)
criterion = torch.nn.CrossEntropyLoss()

l1_lambda = 1e-5

for x_batch, y_batch in dataloader:
    optimizer.zero_grad()
    output = model(x_batch)
    data_loss = criterion(output, y_batch)

    l1_penalty = sum(p.abs().sum() for p in model.parameters() if p.dim() > 1)  # skip biases
    total_loss = data_loss + l1_lambda * l1_penalty

    total_loss.backward()
    optimizer.step()

In Keras, both are available per-layer:

from tensorflow.keras.layers import Dense
from tensorflow.keras.regularizers import l1, l2, l1_l2

model = tf.keras.Sequential([
    Dense(256, activation='relu', kernel_regularizer=l2(1e-4), input_shape=(784,)),
    Dense(10, activation='softmax', kernel_regularizer=l1(1e-5))
])

Choosing Regularization Strength ($\lambda$) in Neural Networks

$\lambda$ valueEffect on network
Too small (e.g., 1e-6 or smaller)Negligible regularization, overfitting risk remains
Reasonable range (1e-5 to 1e-2, task-dependent)Meaningful reduction in overfitting without hurting capacity too much
Too largeUnderfitting; weights are pushed too aggressively toward zero, and the model can’t fit even the training data well

The right value depends heavily on model size, dataset size, and the base learning rate — it’s best tuned via validation performance, often through a grid or random search over a log-scaled range of $\lambda$ values.

L1 vs. L2 in Neural Network Contexts: Practical Comparison

AspectL1 in neural networksL2 in neural networks
Primary roleSparsity, pruning, feature selectionGeneral overfitting control (weight decay)
Typical usage frequencyLess common, used selectivelyVery common, near-default in most training setups
Optimizer supportUsually manual (added to loss)Often built into the optimizer (weight_decay argument)
Effect on inference costCan reduce cost via sparsityNo direct effect on inference cost
Interaction with AdamRequires manual careRequires AdamW for correct decoupling
Common layers applied toFirst layer, or entire network for compressionNearly all layers uniformly

Advantages in the Neural Network Context

L2 (weight decay):

L1:

Disadvantages in the Neural Network Context

L2 (weight decay):

L1:

How Regularization Strength Interacts With Network Depth and Width

An often underappreciated aspect of applying L1/L2 regularization in neural networks specifically (as opposed to simpler linear models) is that the same $\lambda$ value can have very different practical effects depending on the size of the network. A wide layer with thousands of units distributes the total penalty across many more weights than a narrow layer, so the effective “pressure” per individual weight is diluted. In very deep networks, the compounding effect of weight decay applied uniformly to every layer can also interact with vanishing/exploding gradient dynamics — a poorly chosen $\lambda$ in early layers can occasionally accelerate vanishing gradients if weights are pushed too aggressively toward zero before the network has learned useful early-layer features. In practice, this means:

Regularization and the Bias-Variance Trade-off in Practice

It’s useful to connect L1/L2 regularization strength directly to the classic bias-variance trade-off when tuning a neural network:

Regularization strengthBiasVarianceTypical symptom
Too lowLowHighLarge train/validation gap, clear overfitting
Well-tunedBalancedBalancedTrain and validation metrics track closely and both are strong
Too highHighLowBoth train and validation performance are poor (underfitting)

A practical tuning workflow: start with no regularization (or a very small $\lambda$) and confirm the model can overfit the training data at all — this validates that your model has sufficient capacity. Then gradually increase $\lambda$ while monitoring the gap between training and validation performance, stopping once the gap narrows to an acceptable level without validation performance itself starting to degrade.

Weight Decay’s Special Relationship With Batch Normalization

An interesting and somewhat subtle interaction: when a convolutional or linear layer is immediately followed by batch normalization, the scale of that layer’s weights becomes largely irrelevant to the network’s actual output, because batch normalization re-normalizes activations regardless of their incoming scale. This means weight decay applied to such a layer doesn’t change the function the network computes in the same direct way it would without normalization — instead, its main effect becomes influencing the effective learning rate of that layer indirectly, through its impact on gradient magnitudes relative to weight norm. This is a genuinely subtle area of ongoing research, and it’s part of why some architectures (particularly ones leaning heavily on normalization layers) sometimes use smaller weight decay values than older architectures without normalization.

Real-World Use Cases

Best Practices

Summary

Inside a neural network, L1 and L2 regularization serve related but distinct purposes: L2 (weight decay) provides broad, smooth control over weight magnitudes and is the default choice for general overfitting prevention across virtually the entire network, while L1 induces sparsity, making it especially useful for feature selection, interpretability, and model compression. Both work by adding a penalty term to the training objective that discourages unnecessarily large or numerous weights, and both are cheap to implement and broadly compatible with standard training pipelines. In practice, most modern neural network training defaults to L2-style weight decay (ideally via AdamW when using adaptive optimizers), reaching for L1 or Elastic Net-style combinations only when sparsity is a specific, deliberate goal.

References and Further Reading

Exit mobile version