How to Implement Load Balancing with Nginx Upstreams

I remember the exact moment a single application server I was running couldn’t keep up anymore — a marketing email went out, traffic spiked hard, and response times crawled to a halt while one poor server tried to handle everything alone. The fix wasn’t a bigger server; it was spreading the load across multiple servers, and Nginx’s upstream module made that shockingly simple to set up. In this article, I’ll cover everything you need to actually implement load balancing with Nginx, from the basic mechanics through the load balancing algorithms, health checks, and the details that matter once you move past a toy example into something running real traffic.

What an Upstream Actually Is

In Nginx, an upstream block defines a named group of backend servers that Nginx can distribute requests across. Once defined, you reference that group by name in a proxy_pass directive instead of pointing at a single server directly. Nginx then handles the distribution logic — which backend gets which request — based on the algorithm you choose.

upstream backend_servers {
    server 10.0.0.11:8080;
    server 10.0.0.12:8080;
    server 10.0.0.13:8080;
}

server {
    listen 80;
    server_name example.com;

    location / {
        proxy_pass http://backend_servers;
    }
}

That’s the whole basic pattern. Everything from here is refinement: choosing the right algorithm, weighting servers appropriately, handling failures gracefully, and keeping sessions consistent where needed.

Requirements

• At least two backend servers running identical (or compatible) application instances, reachable from your Nginx host over the network
• Nginx installed on a separate host acting as the load balancer, or co-located if you’re doing local testing
• Application servers configured to handle requests identically, since clients shouldn’t notice which backend actually served them
• If your application maintains server-side session state, a plan for either sharing that state (recommended) or handling session persistence at the load balancer level

Step 1: Confirm Backend Servers Are Independently Reachable

Before configuring Nginx, verify each backend responds correctly on its own:

curl -I http://10.0.0.11:8080
curl -I http://10.0.0.12:8080
curl -I http://10.0.0.13:8080

If any of these fail, fix that first — load balancing won’t help a backend that’s fundamentally broken.

Step 2: Choose a Load Balancing Algorithm

Nginx supports several distribution methods, each suited to different scenarios.

Round Robin (Default)

Requests are distributed sequentially across servers in order. No configuration needed beyond listing the servers:

upstream backend_servers {
    server 10.0.0.11:8080;
    server 10.0.0.12:8080;
    server 10.0.0.13:8080;
}

This works well when all your backend servers have roughly equal capacity and requests are roughly equal in cost to process.

Weighted Round Robin

If your servers have different capacities (say, one is a beefier instance than the others), assign weights:

upstream backend_servers {
    server 10.0.0.11:8080 weight=3;
    server 10.0.0.12:8080 weight=2;
    server 10.0.0.13:8080 weight=1;
}

Here, for every 6 requests, the first server gets roughly 3, the second gets 2, and the third gets 1, proportional to their weights.

Least Connections

Instead of blindly rotating, Nginx sends each new request to whichever backend currently has the fewest active connections. This is a better fit when request processing times vary significantly, since round robin can otherwise pile up slow requests on one server while others sit idle:

upstream backend_servers {
    least_conn;
    server 10.0.0.11:8080;
    server 10.0.0.12:8080;
    server 10.0.0.13:8080;
}

IP Hash

This method routes a given client IP consistently to the same backend server, which is useful for maintaining session affinity without needing shared session storage (though shared storage is still generally the more robust solution):

upstream backend_servers {
    ip_hash;
    server 10.0.0.11:8080;
    server 10.0.0.12:8080;
    server 10.0.0.13:8080;
}

Be aware that ip_hash can lead to uneven distribution if a disproportionate share of your traffic comes from behind a small number of NAT’d IPs (common with corporate networks or certain mobile carriers), since all of those users will hash to the same backend.

Generic Hash

For more control over what determines routing consistency, hash lets you key on any variable, such as a cookie or URL:

upstream backend_servers {
    hash $request_uri consistent;
    server 10.0.0.11:8080;
    server 10.0.0.12:8080;
    server 10.0.0.13:8080;
}

The consistent parameter enables consistent hashing, which minimizes redistribution disruption when servers are added or removed — useful for cache-backend scenarios where you want a given URI to reliably map to the same backend to maximize cache hit rates.

Step 3: Configure Health Checks (Passive, Built Into Open-Source Nginx)

The open-source version of Nginx supports passive health checks — it monitors for failed connections and temporarily removes unhealthy backends from rotation:

upstream backend_servers {
    server 10.0.0.11:8080 max_fails=3 fail_timeout=30s;
    server 10.0.0.12:8080 max_fails=3 fail_timeout=30s;
    server 10.0.0.13:8080 max_fails=3 fail_timeout=30s;
}

• max_fails=3 — after 3 consecutive failed attempts to reach this server, Nginx marks it as unavailable.
• fail_timeout=30s — the server stays marked as unavailable for 30 seconds, after which Nginx will try it again.

Note that active health checks (proactively polling a health endpoint on a schedule, independent of real traffic) are a feature of Nginx Plus, the commercial version. In open-source Nginx, you can approximate this using an external tool or script that pings each backend and manipulates the config or uses the zone/state shared memory mechanism with a third-party module, but out of the box, passive checks based on real request failures are what you get.

Step 4: Handle Backup Servers

You can designate a server as a backup, which only receives traffic when all primary servers are unavailable:

upstream backend_servers {
    server 10.0.0.11:8080;
    server 10.0.0.12:8080;
    server 10.0.0.14:8080 backup;
}

This is useful for a disaster-recovery instance that you don’t want absorbing regular traffic but want available as a fallback.

Step 5: Write the Full Server Block

Here’s a complete, production-ready configuration combining several of these pieces:

upstream backend_servers {
    least_conn;
    server 10.0.0.11:8080 weight=2 max_fails=3 fail_timeout=30s;
    server 10.0.0.12:8080 weight=1 max_fails=3 fail_timeout=30s;
    server 10.0.0.13:8080 weight=1 max_fails=3 fail_timeout=30s;
    server 10.0.0.14:8080 backup;

    keepalive 32;
}

server {
    listen 80;
    server_name example.com;

    location / {
        proxy_pass http://backend_servers;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        proxy_http_version 1.1;
        proxy_set_header Connection "";

        proxy_connect_timeout 5s;
        proxy_send_timeout 60s;
        proxy_read_timeout 60s;

        proxy_next_upstream error timeout http_502 http_503 http_504;
        proxy_next_upstream_tries 3;
    }
}

Key additions worth explaining:

• keepalive 32; — maintains a pool of 32 idle keepalive connections per worker process to backend servers, reducing the overhead of establishing new TCP connections for every request. Requires proxy_http_version 1.1; and clearing the Connection header, as shown, to actually take effect.
• proxy_next_upstream error timeout http_502 http_503 http_504; — if a request to one backend fails with any of these conditions, Nginx automatically retries the request against a different backend in the pool.
• proxy_next_upstream_tries 3; — caps the number of backends Nginx will try for a single request before giving up, preventing a cascade of retries against every server in a large pool if something is systemically broken.

Step 6: Test and Reload

sudo nginx -t
sudo systemctl reload nginx

Verifying Load Distribution

A simple way to confirm requests are actually being distributed: have each backend return an identifying header or footer.

On each backend (adjust for your framework), add something like:

X-Backend-Server: server-1

Then hit your load balancer repeatedly and observe the distribution:

for i in {1..10}; do curl -sI http://example.com | grep X-Backend-Server; done

With round robin and three equally weighted servers, you should see a roughly even spread across the identifying headers.

Testing Failover Behavior

Deliberately stop one backend and confirm traffic reroutes cleanly:

# On backend 10.0.0.12
sudo systemctl stop myapp

Then hit the load balancer repeatedly and confirm no requests fail — Nginx should detect the failure (based on max_fails/fail_timeout) and stop routing to that backend until it’s marked healthy again after recovering.

Session Persistence Considerations

If your application stores session state in server memory (rather than a shared store like Redis or a database), load balancing introduces a real problem: a user’s session on server A won’t exist on server B, and if round robin sends their next request to server B, they’ll appear logged out or lose cart contents, form progress, etc.

There are two real solutions, and I’d rank them in this order:

1. Externalize session state to Redis, Memcached, or a database, so any backend can serve any request statelessly from the load balancer’s perspective. This is the architecturally correct fix and scales better long-term.
2. Use ip_hash or a cookie-based hash directive to pin a given client to a specific backend for the duration of their session. This is simpler to implement but has downsides: it doesn’t help if that specific backend goes down (the user’s session is lost regardless), and it can create uneven load distribution.

I’d only reach for session pinning as a stopgap while migrating toward externalized session storage, not as a permanent architecture.

Troubleshooting Common Issues

Problem: Uneven distribution despite round robin.

Check for keepalive connection reuse patterns or unusually long-lived requests (like long-polling or streaming endpoints) that hold a connection open and skew apparent distribution. Also confirm you don’t have ip_hash or least_conn unintentionally left in the config from earlier testing.

Problem: 502 errors during backend restarts or deploys.

This suggests Nginx isn’t gracefully handling a backend going down mid-deploy. Make sure proxy_next_upstream is configured to retry on error and timeout, and consider a rolling deployment strategy where you take one backend out of rotation, deploy to it, verify it’s healthy, and only then move to the next — rather than restarting all backends simultaneously.

Problem: One backend is consistently overloaded while others sit idle.

If using round robin with backends of different capacity, switch to weight or least_conn. If request costs vary significantly (some requests are much more expensive than others), least_conn generally handles this better than static round robin or weighting.

Problem: Sessions keep dropping for logged-in users.

This is almost certainly the session-state problem described above. Verify whether your application handles sessions statelessly (via signed cookies/JWTs) or relies on server-side memory, and address accordingly.

Security Considerations

• Restrict backend servers to a private network not reachable directly from the internet; only the load balancer should have a public-facing address.
• Set proxy_connect_timeout to a reasonably low value (a few seconds) so a single unreachable backend doesn’t cause requests to hang for an extended period while Nginx waits to detect the failure.
• Be deliberate about what identifying information (like an internal X-Backend-Server header used for debugging) you strip before responses reach the client — don’t leak internal infrastructure details externally. Use proxy_hide_header if needed: proxy_hide_header X-Backend-Server;
• Apply rate limiting at the load balancer layer, since it’s the single choke point for all traffic and the most efficient place to catch abuse before it reaches any backend.

Performance Tips

• Use keepalive connections between Nginx and backends to avoid the overhead of establishing new TCP connections per request, especially important under high request rates.
• Choose least_conn over plain round robin whenever request processing times vary meaningfully between requests.
• Monitor actual backend response times and error rates (via Nginx logs or a metrics tool) rather than assuming your chosen algorithm is working well — measure it.
• Consider splitting read-heavy and write-heavy traffic across different upstream pools if your architecture supports it (e.g., routing GET requests to a larger pool of read replicas and POST/PUT/DELETE to a smaller, more tightly controlled pool).
• If running Nginx itself as a potential bottleneck at very high scale, consider running multiple Nginx load balancer instances behind a DNS round robin or a cloud provider’s own load balancer, creating a two-tier architecture.

Real-World Use Cases

• Horizontal scaling of stateless API backends, the most common and cleanest use case, where any request can go to any server without session concerns.
• Blue-green and canary deployments, using weighted upstreams to gradually shift traffic from an old version to a new one (e.g., weight=95 on the stable version and weight=5 on the canary, gradually adjusted).
• Multi-region failover, using backup servers in a secondary region that only receive traffic if the primary region’s servers become unreachable.
• Database read-replica routing, where a separate upstream pool of read-only database proxy endpoints handles GET-heavy traffic, distinct from the write path.

Best Practices Summary

• Choose your load balancing algorithm based on actual traffic characteristics, not habit — least_conn is a strong general-purpose default for variable request costs.
• Configure max_fails and fail_timeout deliberately rather than relying on defaults, and understand these are passive, traffic-driven checks in open-source Nginx.
• Externalize session state rather than relying on IP or cookie-based pinning as a long-term solution.
• Use keepalive between Nginx and backends for meaningful performance gains under load.
• Test failover behavior deliberately, not just happy-path traffic distribution.
• Keep backend servers on a private network, reachable only through the load balancer.

Combining Load Balancing With TLS Termination

In most real deployments, the load balancer is also where TLS gets terminated, so let’s put the pieces together into a single, complete example:

upstream backend_servers {
    least_conn;
    server 10.0.0.11:8080 weight=2 max_fails=3 fail_timeout=30s;
    server 10.0.0.12:8080 weight=1 max_fails=3 fail_timeout=30s;
    server 10.0.0.13:8080 weight=1 max_fails=3 fail_timeout=30s;
    keepalive 32;
}

server {
    listen 80;
    server_name example.com;
    return 301 https://$host$request_uri;
}

server {
    listen 443 ssl;
    server_name example.com;

    ssl_certificate     /etc/letsencrypt/live/example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;

    location / {
        proxy_pass http://backend_servers;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
    }
}

This is a common, solid production pattern: TLS terminates once at the load balancer, and traffic to backends flows over plain HTTP within your trusted private network, avoiding the overhead of re-encrypting for each internal hop while still protecting anything that crosses the public internet.

Monitoring Your Load Balancer

A load balancer that’s silently misbehaving is worse than no load balancer at all, since it creates a false sense of resilience. I keep an eye on a handful of specific signals:

• Per-backend request counts and error rates, ideally broken out individually rather than aggregated, so a single misbehaving backend doesn’t get hidden inside an overall-healthy-looking average.
• Upstream response times, tracked via $upstream_response_time in a custom log format, to catch a backend that’s technically up but responding slowly.
• upstream_status codes, to distinguish backend-generated errors from Nginx-generated ones (like a 502 from a genuinely failed connection versus a 500 the backend itself returned).

A useful custom log format for this:

log_format upstream_log '$remote_addr - [$time_local] "$request" '
                         '$status upstream_addr=$upstream_addr '
                         'upstream_status=$upstream_status '
                         'rt=$request_time uprt=$upstream_response_time';

access_log /var/log/nginx/lb_access.log upstream_log;

This gives you, per request, exactly which backend handled it and how long it took — invaluable when trying to figure out whether a performance problem is isolated to one server or affecting the whole pool.

Frequently Asked Questions

How many backend servers do I actually need to start?

Two is the practical minimum for real redundancy — one server alone gives you no failover protection at all. I generally recommend starting with three if budget allows, since it gives you tolerance for one server being down for maintenance while still having redundancy for an unexpected second failure.

Does Nginx load balancing work for WebSocket connections?

Yes, with the same Upgrade/Connection header handling covered in other guides in this series for WebSocket proxying, combined with your chosen load balancing algorithm. Just be aware that least_conn and similar algorithms consider a long-lived WebSocket connection as one ongoing “connection” for their accounting purposes, which is exactly the behavior you want.

What’s the difference between Nginx’s open-source load balancing and a cloud load balancer (like an AWS ALB)?

A cloud provider’s managed load balancer typically offers active health checks, deeper integration with auto-scaling groups, and managed high availability for the load balancer itself (since a single Nginx instance is, ironically, its own single point of failure unless you run multiple Nginx instances behind something like a floating IP or DNS-based failover). Nginx gives you more granular control and no vendor lock-in, at the cost of you being responsible for the load balancer’s own resilience.

Can I run multiple Nginx load balancers for redundancy?

Yes, and for anything serious, I’d recommend it — running two or more Nginx instances behind a floating/virtual IP (using something like keepalived) or behind your DNS provider’s own failover mechanism avoids the load balancer itself becoming a single point of failure.

How do I add or remove a backend server without downtime?

Edit the upstream block to add or remove the server line, then run nginx -s reload (or systemctl reload nginx), which reloads configuration gracefully without dropping active connections. For removal specifically, consider first marking the server with down; to drain it gracefully before removing the line entirely in a follow-up change.

Wrapping Up

Load balancing with Nginx upstreams turns a single point of failure into a genuinely resilient, horizontally scalable system, and the core configuration is approachable even for a first attempt. The parts that separate a toy setup from a production-grade one are the details: picking the right algorithm for your actual traffic pattern, handling session state correctly, configuring sensible failure detection and retry behavior, and testing failover scenarios deliberately rather than assuming they’ll just work. Once you’ve got those pieces in place, you’ll have a setup that can absorb traffic spikes and individual server failures without your users ever noticing.