Mean squared error was the first loss function I ever encountered — long before I even knew what a neural network was, I was fitting a straight line through scattered data points in a high school statistics class using “least squares.” It turns out that’s exactly the same idea that powers regression loss functions in deep learning today. In this article, I’ll go through the definition, the math, the intuition, and the practical trade-offs of using MSE as a loss function.
The Problem MSE Solves
Whenever I’m predicting a continuous numerical value — house prices, temperature, stock returns, sensor readings — I need a way to measure how far off my predictions are from the actual values. Mean squared error does exactly that by squaring the difference between predicted and actual values, then averaging across all examples.
The Mathematical Definition
For a single prediction $\hat{y}_i$ and true value $y_i$, the squared error is:
$$e_i = (y_i – \hat{y}_i)^2$$
For a dataset of $N$ examples, the mean squared error is:
$$\text{MSE} = \frac{1}{N}\sum_{i=1}^{N}(y_i – \hat{y}_i)^2$$
If I’m working with a model that outputs a vector (multiple regression targets per example), I typically average over both examples and output dimensions:
$$\text{MSE} = \frac{1}{N \cdot D}\sum_{i=1}^{N}\sum_{j=1}^{D}(y_{i,j} – \hat{y}_{i,j})^2$$
where $D$ is the number of output dimensions.
Why Squaring the Error?
I used to wonder why we square the error instead of just taking the absolute value. There are a few good reasons:
- Differentiability: The squared error function is smooth and differentiable everywhere, including at zero, which makes gradient-based optimization straightforward. The absolute error function (used in Mean Absolute Error, or MAE) has a sharp corner at zero where the derivative is undefined.
- Penalizes large errors more: Squaring amplifies larger errors disproportionately, which pushes the model to avoid big mistakes even at the cost of slightly larger small errors.
- Connection to Gaussian likelihood: Minimizing MSE is mathematically equivalent to performing maximum likelihood estimation under the assumption that errors are normally distributed. This gives MSE a solid probabilistic foundation.
The Maximum Likelihood Connection
If I assume that my target $y$ is generated as $y = f(x) + \epsilon$, where $\epsilon \sim \mathcal{N}(0, \sigma^2)$, then the likelihood of observing $y_i$ given prediction $\hat{y}_i$ is:
$$p(y_i | \hat{y}_i) = \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left(-\frac{(y_i – \hat{y}_i)^2}{2\sigma^2}\right)$$
Taking the negative log-likelihood and dropping constants that don’t depend on $\hat{y}_i$, I’m left with something directly proportional to $(y_i – \hat{y}_i)^2$ — which is exactly the MSE loss. This is why MSE is the natural loss function when I believe my prediction errors are roughly Gaussian.
The Gradient of MSE
For a single example, the derivative of the squared error with respect to the prediction is:
$$\frac{\partial}{\partial \hat{y}_i}(y_i – \hat{y}_i)^2 = -2(y_i – \hat{y}_i)$$
This gradient is proportional to the size of the error — larger errors produce proportionally larger gradient updates, which is part of why MSE-trained models can be sensitive to outliers.
A Worked Numerical Example
Suppose I have four predictions:
| Example | True $y$ | Predicted $\hat{y}$ | Squared Error |
|---|---|---|---|
| 1 | 3.0 | 2.5 | 0.25 |
| 2 | 5.0 | 5.2 | 0.04 |
| 3 | 2.0 | 4.0 | 4.00 |
| 4 | 7.0 | 6.8 | 0.04 |
$$\text{MSE} = \frac{0.25 + 0.04 + 4.00 + 0.04}{4} = \frac{4.33}{4} \approx 1.08$$
Notice how example 3, with an error of only 2.0, contributes 4.0 to the sum — nearly the entire total — because the error gets squared. This is the outlier sensitivity I mentioned above, in action.
Visualizing the Pipeline
flowchart TD
A[Input Features] --> B[Neural Network Layers]
B --> C[Predicted Value y-hat]
D[True Value y] --> E["Compute Error: y minus y-hat"]
C --> E
E --> F[Square the Error]
F --> G[Average Across All Examples]
G --> H[MSE Loss]
H --> I[Backpropagation to Update Weights]
Code Examples
PyTorch
import torch
import torch.nn as nn
y_true = torch.tensor([3.0, 5.0, 2.0, 7.0])
y_pred = torch.tensor([2.5, 5.2, 4.0, 6.8], requires_grad=True)
mse_loss = nn.MSELoss()
loss = mse_loss(y_pred, y_true)
print(f"MSE Loss: {loss.item():.4f}")
# Simple regression model example
model = nn.Sequential(
nn.Linear(10, 32),
nn.ReLU(),
nn.Linear(32, 1)
)
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)
criterion = nn.MSELoss()
TensorFlow / Keras
import tensorflow as tf
y_true = tf.constant([3.0, 5.0, 2.0, 7.0])
y_pred = tf.constant([2.5, 5.2, 4.0, 6.8])
mse = tf.keras.losses.MeanSquaredError()
loss = mse(y_true, y_pred)
print(f"MSE Loss: {loss.numpy():.4f}")
model = tf.keras.Sequential([
tf.keras.layers.Dense(32, activation='relu', input_shape=(10,)),
tf.keras.layers.Dense(1)
])
model.compile(optimizer='adam', loss='mse', metrics=['mae'])
MSE vs Mean Absolute Error (MAE)
This comparison comes up constantly in practice, so it’s worth spelling out clearly:
| Aspect | MSE | MAE |
|---|---|---|
| Formula | $\frac{1}{N}\sum(y_i – \hat{y}_i)^2$ | $\frac{1}{N}\sum|y_i – \hat{y}_i|$ |
| Sensitivity to outliers | High (errors squared) | Lower (linear penalty) |
| Gradient behavior | Smooth, proportional to error | Constant magnitude, undefined at zero |
| Probabilistic interpretation | Assumes Gaussian noise | Assumes Laplacian noise |
| Common use | General regression | Regression with many outliers |
There’s also Huber Loss, which combines the best of both worlds — quadratic for small errors, linear for large errors:
$$\mathcal{L}_{\delta}(y, \hat{y}) = \begin{cases} \frac{1}{2}(y-\hat{y})^2 & \text{if } |y-\hat{y}| \leq \delta \ \delta \left(|y – \hat{y}| – \frac{1}{2}\delta\right) & \text{otherwise} \end{cases}$$
I tend to reach for Huber loss when I know my dataset has some outliers but I still want smooth gradients for the majority of “normal” errors.
Advantages of MSE
- Smooth and differentiable everywhere, making optimization straightforward with gradient descent and its variants.
- Strong theoretical grounding in maximum likelihood estimation under Gaussian noise assumptions.
- Simple to compute and interpret, especially when reported alongside its square root (RMSE), which has the same units as the target variable.
- Convex for linear models, guaranteeing a single global minimum in that setting.
Disadvantages and Limitations
- Highly sensitive to outliers: A single large error can dominate the total loss, as shown in my worked example above.
- Units are squared: Raw MSE values are in squared units of the target, making them less directly interpretable — this is why Root Mean Squared Error (RMSE) is often reported instead:
$$\text{RMSE} = \sqrt{\text{MSE}}$$
- Assumes homoscedastic Gaussian noise: If the real error distribution isn’t Gaussian, or the variance differs across examples, MSE may not be the ideal choice.
- Can lead to underfitting on skewed targets: If the target distribution is highly skewed, minimizing MSE tends to push predictions toward the mean rather than being well-calibrated across the whole range.
Real-World Use Cases
- House price prediction — regression on continuous property prices.
- Demand forecasting — predicting future sales, energy consumption, or inventory needs.
- Sensor calibration — predicting continuous physical measurements like temperature or pressure.
- Financial modeling — predicting stock returns, though often combined with more robust losses due to outlier sensitivity.
- Autoencoders — MSE is a very common reconstruction loss for continuous input data (e.g., images normalized to a continuous range).
- Time-series forecasting — predicting future values in weather, traffic, or economic data.
Best Practices
- Normalize or standardize your target variable before training, since MSE is scale-sensitive and unnormalized targets can lead to unstable gradients.
- Check for outliers in your data before committing to MSE — if outliers are present and not meaningful, consider MAE or Huber loss instead.
- Report RMSE alongside MSE for easier interpretation in the original units of your target variable.
- Combine with regularization (L1/L2) to prevent overfitting, especially in high-dimensional regression problems.
- Visualize residuals ($y – \hat{y}$) after training to check for patterns that suggest the model is systematically biased in certain regions.
Summary
Mean squared error is the default loss function for regression problems, rooted in the assumption that prediction errors follow a Gaussian distribution. Its smoothness makes it easy to optimize with gradient-based methods, but its sensitivity to outliers means it isn’t always the right choice — alternatives like MAE and Huber loss exist for exactly this reason. Understanding when MSE is appropriate, and when it isn’t, is one of the most practical skills I’ve picked up working with regression models.
References
- Bishop, C.M. (2006). “Pattern Recognition and Machine Learning.” Springer.
- Goodfellow, I., Bengio, Y., & Courville, A. (2016). “Deep Learning.” MIT Press.
- PyTorch Documentation: https://pytorch.org/docs/stable/generated/torch.nn.MSELoss.html
- TensorFlow Documentation: https://www.tensorflow.org/api_docs/python/tf/keras/losses/MeanSquaredError
- Huber, P.J. (1964). “Robust Estimation of a Location Parameter.” Annals of Mathematical Statistics.
