The first time a backend server in my load-balanced cluster crashed at 2 AM, I got lucky — Apache had already been configured for failover, so traffic quietly shifted to the healthy servers and nobody noticed until I checked the logs the next morning. That experience convinced me that failover configuration isn’t optional for any production load-balanced setup. In this post, I’ll walk through exactly how I configure Apache to detect failing backends and automatically reroute traffic around them.
What Is Failover in Load Balancing?
Failover is the process by which traffic is automatically redirected away from a server that has become unavailable or unhealthy, toward servers that are still functioning. Without failover, a single crashed backend can mean broken pages, timeouts, or dropped connections for a portion of your users — even though other servers in the pool are perfectly capable of handling the load.
Apache handles failover primarily through mod_proxy_balancer, which continuously tracks the state of each backend and can mark members as failed based on connection errors.
Prerequisites
- Apache 2.4+ installed
mod_proxy,mod_proxy_http,mod_proxy_balancer, andmod_lbmethod_byrequests(or another lbmethod) enabled- At least two backend application servers
- Basic understanding of virtual host configuration
Step 1: Enable the Necessary Modules
sudo a2enmod proxy proxy_http proxy_balancer lbmethod_byrequests
sudo systemctl restart apache2
Step 2: Define a Balancer Pool with Failover Settings
Here’s a baseline configuration I use, with failover-specific parameters called out:
<Proxy "balancer://webcluster">
BalancerMember "http://192.168.1.10:8080" retry=60 timeout=5
BalancerMember "http://192.168.1.11:8080" retry=60 timeout=5
BalancerMember "http://192.168.1.12:8080" status=+H
ProxySet lbmethod=byrequests
</Proxy>
ProxyPass "/" "balancer://webcluster/"
ProxyPassReverse "/" "balancer://webcluster/"
Let’s break down the failover-relevant parameters:
retry— how many seconds Apache waits before trying a failed member again (default 60).timeout— how long Apache waits for a response from a backend before considering the request failed.status=+H— marks a member as a “hot standby,” only used when all primary members are down.
Step 3: Configure Connection Failure Handling
By default, Apache marks a backend as failed after a connection error, but you can tune this behavior with additional parameters:
BalancerMember "http://192.168.1.10:8080" retry=60 timeout=5 failonstatus=500,503
failonstatus tells Apache to treat specific HTTP response codes as failures, not just connection-level errors. I’ve found this critical — without it, a backend that’s technically “up” but returning 503s because its database connection pool is exhausted will keep receiving traffic.
Step 4: Set Up Failover with a Hot Standby Server
If you want a dedicated backup server that only receives traffic when all primary members are down, use the status=+H flag:
<Proxy "balancer://webcluster">
BalancerMember "http://192.168.1.10:8080" loadfactor=50
BalancerMember "http://192.168.1.11:8080" loadfactor=50
BalancerMember "http://192.168.1.99:8080" status=+H
ProxySet lbmethod=byrequests
</Proxy>
This is useful for disaster recovery scenarios — for example, keeping a smaller-capacity standby server in a different data center that only kicks in during a regional outage.
Step 5: Combine with Health Checks for Faster Detection
Failover based purely on request failures means a user has to hit the broken backend before Apache notices. Pairing failover with proactive health checks (covered in more depth in a separate post) speeds up detection significantly:
ProxySet hcmethod=GET hcuri=/health hcinterval=10
This actively probes /health on each backend every 10 seconds, marking members as failed before real user traffic ever reaches them.
Step 6: Test Your Failover Configuration
I always test failover deliberately before trusting it in production. Stop one of your backend services:
sudo systemctl stop myapp # on the backend server
Then send repeated requests through Apache and watch the access logs:
for i in {1..10}; do curl -s -o /dev/null -w "%{http_code}\n" http://your-lb-domain/; done
Check /balancer-manager (if enabled) to confirm the down member shows an “Error” or “Disabled” state, and that requests are being served entirely by the remaining healthy backends.
Real-World Use Cases
- Rolling deployments — failover settings let you take a server offline for updates without users noticing, as long as another member picks up the slack.
- Regional disaster recovery — a hot standby server in another region can absorb traffic if your primary data center has an outage.
- Database-dependent services — using
failonstatusto catch 503s from an app whose database connection has failed, rather than waiting for a hard connection error.
Troubleshooting Common Issues
Failover Isn’t Triggering — check that timeout isn’t set too high; a long timeout means Apache waits a long time before deciding a backend has failed, which can look like failover “not working” when it’s actually just slow.
Traffic Still Going to a Dead Server — verify the retry interval hasn’t expired and sent a probe request back to the still-broken server. Also confirm the balancer-manager status to rule out manual misconfiguration.
All Requests Fail Even Though One Server Is Healthy — check ProxyTimeout and connection limits; if Apache’s connection pool to the healthy backend is exhausted, new requests may fail even though failover technically worked.
Security Best Practices
- Don’t expose backend server IPs or hostnames in error pages sent to users.
- Use
ErrorDocumentdirectives to present a friendly error page rather than a raw proxy error if all backends fail. - Restrict access to
/balancer-managerso failover states can’t be tampered with by unauthorized users. - Log failover events distinctly so you can audit how often — and why — failover is triggering.
Performance Optimization Tips
- Keep
timeoutvalues realistic for your application’s normal response time; too aggressive a timeout causes false failovers under normal load spikes. - Use active health checks alongside passive failover to reduce the number of real users who hit a failing backend before it’s marked down.
- Consider
lbmethod=heartbeatfor very large clusters, which uses a separate heartbeat monitoring process instead of relying purely on request-based detection.
FAQs
Does failover cause dropped requests for users connected to the failed server? In-flight requests to a server that dies mid-request may fail, but subsequent requests will be routed to healthy backends automatically once failure is detected.
Can I set up failover across two different data centers? Yes, though latency between the load balancer and remote backends should be considered. A hot standby in another region is a common pattern for disaster recovery.
What’s the difference between failover and load balancing? Load balancing distributes traffic across multiple active servers; failover specifically handles what happens when one or more of those servers becomes unavailable.
Summary and Key Takeaways
Configuring Apache for failover comes down to a handful of parameters on your BalancerMember directives — retry, timeout, failonstatus, and status=+H — combined with proactive health checks for faster detection. Test your failover setup deliberately before you need it in a real incident, and always pair it with proper logging so you can review what happened after the fact.