The Role of L1 and L2 Regularization in Neural Networks

Neural networks, especially large ones, are almost absurdly good at memorizing whatever you feed them. Give a big enough network enough epochs on a modestly-sized dataset, and it will often drive training loss to near zero — including fitting the noise, mislabeled examples, and quirks specific to that exact dataset. L1 and L2 regularization exist specifically to push back against this tendency. In this article, I want to focus specifically on the role these penalties play inside neural network training — not just the abstract math, but how they actually function within the architecture, the optimization process, and the resulting model behavior.

Why Neural Networks Need Regularization at All

A modern neural network can easily have millions or billions of parameters. The number of parameters often vastly exceeds the number of training examples, meaning the model has more than enough capacity to fit the training data exactly, including noise, rather than learning the true underlying signal. This is the essence of overfitting: excellent training performance paired with poor performance on new, unseen data.

Regularization techniques constrain the effective capacity of the model without necessarily changing its architecture, by discouraging solutions where individual weights become unnecessarily large or numerous.

Where L1 and L2 Fit Into the Loss Function

In a neural network, the total loss being minimized during training becomes:

$$J_{total}(\theta) = J_{data}(\theta) + \lambda_1 \sum_i |\theta_i| + \lambda_2 \sum_i \theta_i^2$$

where $J_{data}(\theta)$ is the task loss (e.g., cross-entropy for classification, mean squared error for regression), and the two summation terms are the L1 and L2 penalties respectively, each with its own strength hyperparameter. Most practical setups use only one of the two (commonly L2), though both can be combined as an Elastic Net-style penalty if desired.

Critically, this penalty is typically applied to the network’s weight matrices (the parameters connecting layers), not usually to bias terms, since biases don’t contribute to model complexity in the same way weights do — regularizing them yields little benefit and can slightly hurt performance.

The Role of L2 Regularization (Weight Decay) in Neural Networks

L2 regularization is by far the more commonly used of the two in deep learning, almost always under the name weight decay. Its role in a neural network is to:

1. Prevent any single weight from growing excessively large, which would make the model’s output overly sensitive to small changes in specific input features.
2. Encourage the network to distribute its “reliance” across many weights, rather than concentrating predictive power in just a few, which tends to make representations more robust and generalizable.
3. Smooth the loss landscape, since the added quadratic term is convex and well-behaved, making optimization more numerically stable.
4. Act as an implicit constraint on model complexity, effectively reducing the “effective” number of parameters the model can meaningfully use, even though the literal parameter count doesn’t change.

In modern training loops, weight decay is often applied directly by the optimizer rather than by adding an explicit term to the loss function. Recall the L2 gradient update:

$$\theta_i \leftarrow \theta_i(1 – \eta\lambda) – \eta\frac{\partial J_{data}}{\partial \theta_i}$$

This shows the weight decay effect directly: every single weight is multiplied by a factor slightly less than 1 on every step, in addition to the usual gradient-based adjustment, continuously pulling weights toward zero unless the data gradient consistently pushes back.

The Role of L1 Regularization in Neural Networks

L1 regularization is used less frequently in deep learning than L2, but it has a specific and valuable role: encouraging sparsity in the network’s weights. Its role includes:

1. Automatic feature/connection pruning — weights that don’t meaningfully contribute to reducing the task loss get pushed all the way to exactly zero, effectively removing those connections from the network.
2. Model compression — a sparse weight matrix (with many exact zeros) can be stored and computed more efficiently using sparse matrix representations, which is valuable for deploying models on resource-constrained devices.
3. Interpretability — in smaller networks or the first layer of a network processing structured/tabular data, L1-induced sparsity can highlight which specific input features the model actually relies on.

L1 is less commonly applied throughout an entire deep network’s hidden layers (since sparsity in the middle of a network can sometimes disrupt gradient flow and hurt training dynamics), but it’s occasionally used on the first layer for feature selection, or on the entire network when compression/sparsity is a specific design goal.

Visualizing Where Regularization Acts

flowchart TD
    A[Input layer] --> B[Hidden layer weights: W1]
    B --> C[Hidden layer weights: W2]
    C --> D[Output layer weights: W3]
    D --> E[Task loss: cross-entropy / MSE]
    B -.L2 penalty on W1.-> F[Total loss]
    C -.L2 penalty on W2.-> F
    D -.L2 penalty on W3.-> F
    E --> F[Total regularized loss]

Weight Decay in Practice: A Subtlety With Adaptive Optimizers

An important, often overlooked detail: when you use weight decay with adaptive optimizers like Adam, naively adding the L2 penalty to the gradient interacts oddly with Adam’s per-parameter adaptive scaling. The regularization term gets divided by the same $\sqrt{\hat{v}_t}$ term as the data gradient, which distorts its intended uniform shrinking effect. This led to the development of AdamW, which decouples weight decay from the gradient-based update entirely:

$$\theta_{t+1} = \theta_t – \eta \cdot \frac{\hat{m}_t}{\sqrt{\hat{v}_t} + \epsilon} – \eta\lambda\theta_t$$

If you’re training with Adam and want proper L2-style regularization, AdamW is almost always the better choice over adding an L2 term manually to the loss.

import torch

model = torch.nn.Sequential(
    torch.nn.Linear(784, 256),
    torch.nn.ReLU(),
    torch.nn.Linear(256, 10)
)

# Proper decoupled weight decay
optimizer = torch.optim.AdamW(model.parameters(), lr=1e-3, weight_decay=1e-2)

import torch

model = torch.nn.Sequential(
    torch.nn.Linear(784, 256),
    torch.nn.ReLU(),
    torch.nn.Linear(256, 10)
)

# Proper decoupled weight decay
optimizer = torch.optim.AdamW(model.parameters(), lr=1e-3, weight_decay=1e-2)

Applying L1 Regularization to a Neural Network

Since most frameworks don’t build L1 into the optimizer directly, it’s typically added explicitly to the loss:

import torch

model = torch.nn.Sequential(
    torch.nn.Linear(784, 256),
    torch.nn.ReLU(),
    torch.nn.Linear(256, 10)
)
optimizer = torch.optim.SGD(model.parameters(), lr=0.01)
criterion = torch.nn.CrossEntropyLoss()

l1_lambda = 1e-5

for x_batch, y_batch in dataloader:
    optimizer.zero_grad()
    output = model(x_batch)
    data_loss = criterion(output, y_batch)

    l1_penalty = sum(p.abs().sum() for p in model.parameters() if p.dim() > 1)  # skip biases
    total_loss = data_loss + l1_lambda * l1_penalty

    total_loss.backward()
    optimizer.step()

import torch

model = torch.nn.Sequential(
    torch.nn.Linear(784, 256),
    torch.nn.ReLU(),
    torch.nn.Linear(256, 10)
)
optimizer = torch.optim.SGD(model.parameters(), lr=0.01)
criterion = torch.nn.CrossEntropyLoss()

l1_lambda = 1e-5

for x_batch, y_batch in dataloader:
    optimizer.zero_grad()
    output = model(x_batch)
    data_loss = criterion(output, y_batch)

    l1_penalty = sum(p.abs().sum() for p in model.parameters() if p.dim() > 1)  # skip biases
    total_loss = data_loss + l1_lambda * l1_penalty

    total_loss.backward()
    optimizer.step()

In Keras, both are available per-layer:

from tensorflow.keras.layers import Dense
from tensorflow.keras.regularizers import l1, l2, l1_l2

model = tf.keras.Sequential([
    Dense(256, activation='relu', kernel_regularizer=l2(1e-4), input_shape=(784,)),
    Dense(10, activation='softmax', kernel_regularizer=l1(1e-5))
])

from tensorflow.keras.layers import Dense
from tensorflow.keras.regularizers import l1, l2, l1_l2

model = tf.keras.Sequential([
    Dense(256, activation='relu', kernel_regularizer=l2(1e-4), input_shape=(784,)),
    Dense(10, activation='softmax', kernel_regularizer=l1(1e-5))
])

Choosing Regularization Strength ($\lambda$) in Neural Networks

The right value depends heavily on model size, dataset size, and the base learning rate — it’s best tuned via validation performance, often through a grid or random search over a log-scaled range of $\lambda$ values.

L1 vs. L2 in Neural Network Contexts: Practical Comparison

Advantages in the Neural Network Context

L2 (weight decay):

• Simple to apply broadly across an entire network with a single hyperparameter.
• Numerically stable and compatible with virtually every optimizer.
• Well-supported natively in essentially every deep learning framework.

L1:

• Enables genuine model compression through induced sparsity.
• Can improve interpretability, particularly for the input layer of models processing structured data.
• Useful in resource-constrained deployment scenarios (mobile, edge devices).

Disadvantages in the Neural Network Context

L2 (weight decay):

• Doesn’t reduce the actual number of active parameters, so it provides no direct benefit for model compression or inference speed.
• Too strong a value can meaningfully hurt model capacity and underfit.

L1:

• Non-differentiability at zero can slightly complicate optimization (though in practice, subgradient-based approaches work fine).
• Sparsity in the middle of very deep networks can sometimes disrupt gradient flow and training stability if applied too aggressively.
• Less standardized support across frameworks compared to L2/weight decay.

How Regularization Strength Interacts With Network Depth and Width

An often underappreciated aspect of applying L1/L2 regularization in neural networks specifically (as opposed to simpler linear models) is that the same $\lambda$ value can have very different practical effects depending on the size of the network. A wide layer with thousands of units distributes the total penalty across many more weights than a narrow layer, so the effective “pressure” per individual weight is diluted. In very deep networks, the compounding effect of weight decay applied uniformly to every layer can also interact with vanishing/exploding gradient dynamics — a poorly chosen $\lambda$ in early layers can occasionally accelerate vanishing gradients if weights are pushed too aggressively toward zero before the network has learned useful early-layer features. In practice, this means:

• Larger, wider networks often tolerate (and benefit from) somewhat stronger weight decay values than smaller networks, since they have more capacity to spare.
• Very deep networks sometimes benefit from layer-wise or group-wise regularization strength, applying lighter regularization to early layers (which need to learn general, reusable features) and stronger regularization to later layers (which are more prone to overfitting to task-specific patterns).
• Modern architectures with residual/skip connections and normalization layers (BatchNorm, LayerNorm) tend to be less sensitive to weight decay strength than older, plain feed-forward architectures, since normalization already constrains activation scales somewhat independently of weight magnitude.

Regularization and the Bias-Variance Trade-off in Practice

It’s useful to connect L1/L2 regularization strength directly to the classic bias-variance trade-off when tuning a neural network:

A practical tuning workflow: start with no regularization (or a very small $\lambda$) and confirm the model can overfit the training data at all — this validates that your model has sufficient capacity. Then gradually increase $\lambda$ while monitoring the gap between training and validation performance, stopping once the gap narrows to an acceptable level without validation performance itself starting to degrade.

Weight Decay’s Special Relationship With Batch Normalization

An interesting and somewhat subtle interaction: when a convolutional or linear layer is immediately followed by batch normalization, the scale of that layer’s weights becomes largely irrelevant to the network’s actual output, because batch normalization re-normalizes activations regardless of their incoming scale. This means weight decay applied to such a layer doesn’t change the function the network computes in the same direct way it would without normalization — instead, its main effect becomes influencing the effective learning rate of that layer indirectly, through its impact on gradient magnitudes relative to weight norm. This is a genuinely subtle area of ongoing research, and it’s part of why some architectures (particularly ones leaning heavily on normalization layers) sometimes use smaller weight decay values than older architectures without normalization.

Real-World Use Cases

• Standard CNN and transformer training: L2/weight decay is applied almost universally as a baseline regularizer, typically with $\lambda$ values in the range of 1e-4 to 1e-2.
• Mobile and edge deployment: L1 regularization (often followed by explicit pruning of near-zero weights) is used to shrink models for deployment on resource-constrained hardware.
• Tabular data / structured input models: L1 applied to the first layer can serve as an automatic feature selection mechanism, highlighting which input features the network actually uses.
• Combined regularization strategies: Many production training pipelines combine L2 weight decay with dropout and early stopping simultaneously, since these techniques attack overfitting from different angles and tend to be complementary rather than redundant.

Best Practices

• Default to L2/weight decay (via AdamW’s weight_decay argument, or SGD’s weight_decay argument) as your baseline regularization for most neural network training.
• Don’t apply weight decay to bias terms or normalization layer parameters (like BatchNorm scale/shift) — most frameworks exclude these by default, but double-check if implementing manually.
• Use L1 specifically when your goal is sparsity — for compression, feature selection, or deployment efficiency — rather than as a general-purpose overfitting fix.
• Tune $\lambda$ using validation performance, and expect the ideal value to shift when you change model size, dataset size, or learning rate.
• Combine L1/L2 regularization with dropout and early stopping for a well-rounded regularization strategy rather than relying on any single technique alone.
• If using Adam-family optimizers, use AdamW rather than manually adding an L2 term to the loss, to avoid the decoupling issue described above.

Summary

Inside a neural network, L1 and L2 regularization serve related but distinct purposes: L2 (weight decay) provides broad, smooth control over weight magnitudes and is the default choice for general overfitting prevention across virtually the entire network, while L1 induces sparsity, making it especially useful for feature selection, interpretability, and model compression. Both work by adding a penalty term to the training objective that discourages unnecessarily large or numerous weights, and both are cheap to implement and broadly compatible with standard training pipelines. In practice, most modern neural network training defaults to L2-style weight decay (ideally via AdamW when using adaptive optimizers), reaching for L1 or Elastic Net-style combinations only when sparsity is a specific, deliberate goal.

References and Further Reading

• Goodfellow, I., Bengio, Y., & Courville, A. (2016). “Deep Learning,” Chapter 7: Regularization for Deep Learning.
• Krogh, A., & Hertz, J. A. (1991). “A Simple Weight Decay Can Improve Generalization.” NeurIPS.
• Loshchilov, I., & Hutter, F. (2019). “Decoupled Weight Decay Regularization.” ICLR. https://arxiv.org/abs/1711.05101
• Han, S., Pool, J., Tran, J., & Dally, W. (2015). “Learning both Weights and Connections for Efficient Neural Networks.” NeurIPS. (Relevant to L1-induced pruning)
• PyTorch weight_decay documentation: https://pytorch.org/docs/stable/generated/torch.optim.AdamW.html
• Keras regularizers documentation: https://www.tensorflow.org/api_docs/python/tf/keras/regularizers