How to Set Up Nginx as a Load Balancer

The first time I set up Nginx as a load balancer, it was because a client’s single application server kept falling over during traffic spikes. Adding a second server didn’t help until I put Nginx in front of both, distributing requests so neither one got overwhelmed. That’s really the whole point of load balancing — turning a single point of failure into a distributed, resilient setup.

In this guide, I’ll walk through how Nginx load balancing actually works, the different algorithms available, a full working configuration, health checks, SSL termination, testing, troubleshooting, and the performance and security details I always check before calling a setup “production ready.”

What Load Balancing Actually Does

At its core, a load balancer sits between clients and your backend servers, deciding which backend handles each incoming request. Nginx does this using the upstream block, which defines a named pool of backend servers, and then routes requests to that pool from within a server block.

The benefits are straightforward:

• Higher availability — if one backend server goes down, Nginx stops sending it traffic and routes around it.
• Better performance under load — requests get spread across multiple machines instead of piling up on one.
• Easier scaling — adding capacity often just means adding another backend to the upstream block.
• Zero-downtime deployments — you can take one server out of rotation, update it, and bring it back without users noticing.

Prerequisites

• Nginx installed on the server that will act as the load balancer.
• At least two backend servers (or two backend processes/ports) capable of serving the same application.
• Network connectivity between the load balancer and each backend (same VPC, private network, or reachable IPs).
• Basic familiarity with editing Nginx config files.

Verify Nginx is installed and check the version:

nginx -v

Load Balancing Methods in Nginx

Nginx supports several algorithms out of the box, and picking the right one matters more than people expect.

Round Robin (default)

Requests are distributed sequentially across the backend list. No configuration needed — it’s the default behavior of an upstream block.

upstream backend {
    server 192.168.1.10:8080;
    server 192.168.1.11:8080;
}

Least Connections

Sends the next request to whichever backend currently has the fewest active connections. I use this when backend requests vary a lot in how long they take to process.

upstream backend {
    least_conn;
    server 192.168.1.10:8080;
    server 192.168.1.11:8080;
}

IP Hash

Routes a given client IP to the same backend consistently, based on a hash of the IP address. Useful when your application relies on server-side session state and you haven’t set up shared sessions (like Redis-backed sessions).

upstream backend {
    ip_hash;
    server 192.168.1.10:8080;
    server 192.168.1.11:8080;
}

Weighted Distribution

Useful when backend servers have different capacities. A server with weight=3 gets roughly three times the traffic of a default-weight server.

upstream backend {
    server 192.168.1.10:8080 weight=3;
    server 192.168.1.11:8080 weight=1;
}

Generic Hash

Distributes based on a custom key you define — a URI, a header, whatever makes sense for your app.

upstream backend {
    hash $request_uri consistent;
    server 192.168.1.10:8080;
    server 192.168.1.11:8080;
}

Step-by-Step Configuration

Step 1: Define the Upstream Block

I put this above the server block, usually near the top of the config file or in a separate conf.d/upstream.conf file that I include.

upstream app_backend {
    least_conn;
    server 192.168.1.10:8080 max_fails=3 fail_timeout=30s;
    server 192.168.1.11:8080 max_fails=3 fail_timeout=30s;
    server 192.168.1.12:8080 backup;
}

A few things worth explaining:

• max_fails=3 fail_timeout=30s — if a backend fails 3 times within 30 seconds, Nginx marks it unavailable for 30 seconds and stops routing to it.
• backup — this server only receives traffic if all the primary servers are down. Handy for a disaster-recovery instance.

Step 2: Reference the Upstream in the Server Block

server {
    listen 80;
    server_name app.example.com;

    location / {
        proxy_pass http://app_backend;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

The proxy_set_header lines matter more than people realize — without them, your backend applications will see every request as coming from the load balancer’s IP instead of the real client, which breaks logging, rate limiting, and geo-based logic on the backend.

Step 3: Test the Configuration

sudo nginx -t
sudo systemctl reload nginx

Step 4: Add Passive Health Checks

Nginx’s open-source version does passive health checks by default through max_fails and fail_timeout — it marks a server down after failures, and periodically retries it. This is usually enough for most setups.

For active health checks (proactively probing backends on a schedule rather than waiting for a failure), that’s a feature of Nginx Plus, but I can approximate similar behavior in open-source Nginx by combining it with an external tool or a simple script that checks backend health and updates the config, or using something like nginx_upstream_check_module if you’re willing to compile a custom build.

A Complete Example Configuration

Here’s a full setup I’d deploy for a small application cluster behind SSL:

upstream app_backend {
    least_conn;
    server 10.0.0.11:8080 max_fails=3 fail_timeout=30s;
    server 10.0.0.12:8080 max_fails=3 fail_timeout=30s;
    server 10.0.0.13:8080 backup;
    keepalive 32;
}

server {
    listen 80;
    server_name app.example.com;
    return 301 https://$host$request_uri;
}

server {
    listen 443 ssl http2;
    server_name app.example.com;

    ssl_certificate     /etc/letsencrypt/live/app.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/app.example.com/privkey.pem;

    location / {
        proxy_pass http://app_backend;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        proxy_connect_timeout 5s;
        proxy_send_timeout 60s;
        proxy_read_timeout 60s;
    }

    location /health {
        access_log off;
        return 200 "OK";
        add_header Content-Type text/plain;
    }
}

Notes on this config:

• keepalive 32; in the upstream block keeps connections to backends open for reuse, which reduces the overhead of repeatedly opening TCP connections — I combine this with proxy_http_version 1.1 and clearing the Connection header, since keepalive requires HTTP/1.1.
• The /health endpoint on the load balancer itself is separate from backend health — this is for external monitoring tools (like UptimeRobot or a Kubernetes liveness probe) to check that the load balancer itself is responding.

Testing the Load Balancer

I test in a few stages.

1. Confirm config syntax:

sudo nginx -t

2. Confirm traffic is actually being distributed. I add a temporary endpoint on each backend that returns its own hostname or port, then hit it repeatedly:

for i in {1..10}; do curl -s http://app.example.com/whoami; echo; done

With round robin or least connections, I expect to see responses alternating between backends.

3. Simulate a backend failure. Stop one backend service and confirm Nginx routes around it:

sudo systemctl stop myapp   # on one backend server
curl -I http://app.example.com/

The response should still come back successfully from the remaining backend(s). Check the Nginx error log to confirm it logged the failure:

sudo tail -f /var/log/nginx/error.log

You should see something like upstream server temporarily disabled or connection refused entries pointing to the downed backend.

4. Load test. For a real stress test, I use a tool like ab (Apache Bench) or wrk:

ab -n 1000 -c 50 http://app.example.com/

This fires 1,000 requests with 50 concurrent connections and gives you a breakdown of response times and failure rates.

Troubleshooting Common Issues

502 Bad Gateway errors. This almost always means Nginx can’t reach a backend — either it’s down, the port is wrong, or a firewall is blocking the connection. Check:

curl http://192.168.1.10:8080/

directly from the load balancer server to rule out a network issue.

All traffic going to one server. If you’re using ip_hash and testing from a single machine, this is expected — the whole point of ip_hash is that one client IP always maps to the same backend. Test from multiple IPs or switch to round robin/least_conn for validation purposes.

Sessions breaking (users getting logged out randomly). This happens with round robin or least_conn if your application stores session state locally on each backend instead of in a shared store. Either switch to ip_hash, or better, move sessions to a shared backend like Redis or a database so any server can handle any request.

Backend marked down and never recovering. Check fail_timeout — after that window, Nginx will retry the backend automatically. If it’s still not recovering, verify the backend actually came back up and is listening on the expected port.

Slow response times under load. Check proxy_read_timeout and proxy_connect_timeout — if they’re too short, Nginx will cut off slow-but-legitimate backend responses. Also check backend CPU/memory usage; the load balancer can only distribute load, it can’t create capacity that doesn’t exist.

Security Considerations

• Restrict direct access to backend servers. Backends should only accept connections from the load balancer’s IP, not the public internet. Configure this with firewall rules (ufw, iptables, or cloud security groups).
• Terminate SSL at the load balancer and use internal-only HTTP (or re-encrypted HTTPS) between Nginx and backends, depending on your compliance requirements.
• Rate limit at the load balancer level to protect all backends uniformly:

limit_req_zone $binary_remote_addr zone=one:10m rate=10r/s;

location / {
    limit_req zone=one burst=20 nodelay;
    proxy_pass http://app_backend;
}

• Hide backend identifying headers. Strip or override headers like Server and X-Powered-By from backend responses so attackers can’t fingerprint your stack easily.
• Validate X-Forwarded-For handling on the backend — since the backend now sees requests from the load balancer’s IP, make sure your application trusts and correctly parses the forwarded headers rather than logging every request as coming from the same internal IP.

Performance Tips

• Enable keepalive connections to upstream servers (as shown above) — this cuts down TCP handshake overhead significantly under high request volume.
• Tune worker processes and connections in the main Nginx config to match your server’s CPU cores:

worker_processes auto;
events {
    worker_connections 4096;
}

• Use least_conn instead of round robin for workloads with variable request processing times — it prevents a slow backend from accumulating a backlog while a faster one sits idle.
• Cache what you can at the load balancer layer for GET requests that don’t need to hit the backend every time (see proxy caching, which pairs well with load balancing).
• Monitor upstream response times using the $upstream_response_time variable in your log format, so you can spot a degrading backend before it fails outright:

log_format upstream_time '$remote_addr - $upstream_response_time - $request';
access_log /var/log/nginx/access.log upstream_time;

Real-World Use Cases

• A Node.js API cluster running multiple instances via PM2 on the same server, load balanced across ports 3000–3003 with least_conn to smooth out uneven request durations.
• A multi-server WordPress deployment using ip_hash because the client wasn’t ready to migrate sessions to Redis yet — a pragmatic short-term fix.
• Blue-green deployments, where I’d add the new “green” server as backup initially, verify it manually, then promote it to a primary weight and demote the old “blue” server — all without downtime.
• A microservices gateway, where different location blocks routed to entirely different upstream pools depending on the URL path (/api/users to one service, /api/orders to another).

Best Practices Summary

• Choose your load balancing algorithm based on your application’s actual behavior — not just because round robin is the default.
• Always forward real client IP and protocol headers to backends.
• Restrict backend access to the load balancer only.
• Set sensible timeouts — too short causes false failures, too long lets a broken backend hang requests.
• Test failover manually before trusting it in production.
• Monitor upstream response times, not just whether the load balancer itself is up.
• Keep SSL termination and backend routing logic cleanly separated for maintainability.

Once this is set up correctly, adding capacity to your application becomes almost boring — spin up another backend, add one line to the upstream block, reload Nginx, done. That boring reliability is exactly the point.

Sticky Sessions in More Detail

I mentioned ip_hash earlier as one way to keep a client on the same backend, but it’s worth digging into why this matters and what the alternatives look like, because it’s one of the questions I get asked most often when setting up a load balancer for the first time.

Many applications store session data — login state, shopping cart contents, form progress — in memory on whichever server first handled that user’s request. If the next request from the same user lands on a different backend under round robin or least_conn, that server has no idea who they are, and the user appears logged out or loses their cart. This is often the very first “bug” people notice after switching from a single server to a load-balanced setup, and it usually isn’t a bug in the application at all — it’s a mismatch between how the app manages state and how the load balancer distributes traffic.

ip_hash solves this cheaply at the load balancer level, but it has real limitations: users behind a shared corporate NAT or a mobile carrier’s NAT gateway can all appear to come from the same IP, unevenly loading one backend; and a user whose IP changes mid-session (switching from Wi-Fi to mobile data) loses their session entirely because they now hash to a different backend.

The more robust long-term fix is to remove the need for sticky sessions altogether by moving session storage out of each backend’s local memory and into a shared store like Redis. Once every backend can read the same session data regardless of which one handled the previous request, you can go back to least_conn or round robin without worrying about session affinity at all. I consider ip_hash a reasonable short-term fix, not a permanent architecture decision.

Combining Load Balancing With Health-Aware Failover

Beyond the passive max_fails/fail_timeout approach covered earlier, I sometimes pair Nginx with an external health-check script for tighter control, especially in environments without Nginx Plus’s active health check feature. A simple pattern: a cron job or systemd timer periodically curls each backend’s dedicated health endpoint, and if a backend fails a threshold number of checks, the script rewrites the upstream block (commenting out or removing the failing server) and triggers nginx -s reload.

#!/bin/bash
# simple external health check example
for server in 10.0.0.11:8080 10.0.0.12:8080; do
    if ! curl -sf --max-time 2 "http://$server/health" > /dev/null; then
        echo "$(date): $server failed health check" >> /var/log/backend-health.log
    fi
done

This isn’t as elegant as a proper active health-check system, but it’s a practical middle ground for teams running open-source Nginx who want more proactive detection than the built-in passive checks provide, without taking on the cost of Nginx Plus or a full service mesh.

Frequently Asked Questions

Can I load balance across servers in different data centers? Yes, as long as there’s reliable network connectivity between the load balancer and each backend, though latency to distant backends will affect response times unevenly. For true multi-region setups, DNS-based or anycast load balancing at a layer above Nginx is usually a better fit than trying to make a single Nginx instance balance across continents.

What’s the difference between load balancing and a reverse proxy? A reverse proxy forwards requests to a single backend; a load balancer is a reverse proxy that chooses between multiple backends. Every load balancer is a reverse proxy, but not every reverse proxy is a load balancer — Nginx can do either, and the upstream block is what turns basic proxying into load balancing.

Does Nginx itself become a single point of failure? Yes, unless you also add redundancy at that layer — commonly with a second Nginx instance and a floating/virtual IP managed by something like keepalived, or by putting a cloud load balancer in front of two or more Nginx instances. I always flag this to clients: load balancing your backends solves one single point of failure while potentially introducing a new one at the load balancer itself, unless that’s addressed too.

Should I load balance database connections the same way? No — this guide covers HTTP/application load balancing. Databases have their own replication and load-balancing patterns (read replicas, connection poolers like PgBouncer) that operate very differently from stateless HTTP request distribution.