How to Configure Health Checks for Load-Balanced Servers

For a long time, I relied purely on Apache’s passive failure detection — meaning a backend only got marked as failed after a real user’s request actually hit it and failed. That approach works, but it means someone always has to be the unlucky first request. Configuring proactive health checks changed that, letting Apache detect problems before they affect real traffic. Here’s how I set them up.

Passive vs. Active Health Checks

It’s worth understanding the distinction clearly:

• Passive health checks happen automatically based on real request failures (connection errors, timeouts, or specific HTTP status codes via failonstatus). No extra configuration is strictly needed beyond what mod_proxy_balancer already does.
• Active health checks proactively probe backend servers on a schedule, independent of real traffic, so failures are caught before users are affected.

I use both together — passive as a safety net, active as the primary detection mechanism.

Prerequisites

• Apache 2.4.10+ (active health check support via mod_proxy_hcheck was introduced in this version)
• mod_proxy, mod_proxy_balancer, mod_proxy_hcheck, and mod_watchdog enabled
• A health check endpoint on each backend application (e.g., /health or /status)

Step 1: Build a Proper Health Check Endpoint

Before touching Apache configuration, make sure your application actually has a meaningful health check endpoint. A basic “return 200” endpoint is a start, but I prefer ones that verify critical dependencies:

// Node.js/Express example
app.get('/health', async (req, res) => {
  try {
    await db.ping();
    res.status(200).json({ status: 'ok' });
  } catch (err) {
    res.status(503).json({ status: 'unhealthy', error: err.message });
  }
});

This way, Apache’s health check reflects whether the backend can actually serve requests properly — not just whether the process is technically running.

Step 2: Enable the Required Modules

sudo a2enmod proxy proxy_http proxy_balancer proxy_hcheck watchdog
sudo systemctl restart apache2

mod_watchdog runs the background timer that triggers periodic health checks, and mod_proxy_hcheck performs the actual checks.

Step 3: Configure Active Health Checks

<Proxy "balancer://appcluster">
    BalancerMember "http://192.168.1.10:8080" hcmethod=GET hcuri=/health hcinterval=10 hcpasses=2 hcfails=3
    BalancerMember "http://192.168.1.11:8080" hcmethod=GET hcuri=/health hcinterval=10 hcpasses=2 hcfails=3
    BalancerMember "http://192.168.1.12:8080" hcmethod=GET hcuri=/health hcinterval=10 hcpasses=2 hcfails=3
    ProxySet lbmethod=byrequests
</Proxy>

ProxyPass "/" "balancer://appcluster/"
ProxyPassReverse "/" "balancer://appcluster/"

Breaking down the parameters:

• hcmethod — the HTTP method used for checks (GET, HEAD, OPTIONS, CPING, TCP)
• hcuri — the endpoint to check
• hcinterval — how often (in seconds) to run the check
• hcpasses — number of consecutive successful checks needed to mark a member healthy again
• hcfails — number of consecutive failed checks before marking a member as down

Step 4: Choosing the Right Health Check Method

• TCP — simply confirms the port is accepting connections; lightweight but doesn’t verify application-level health.
• GET/HEAD — hits an actual HTTP endpoint, letting you check application-level status (like my database ping example above).
• CPING — sends an AJP CPING request, useful for AJP-based backends like Tomcat.

I default to GET against a dedicated /health endpoint for anything beyond the simplest setups, since TCP checks alone can’t catch an application that’s technically listening but internally broken.

Step 5: Setting Expected Response Conditions

You can also validate the response body, not just the status code, using an expression:

BalancerMember "http://192.168.1.10:8080" \
    hcmethod=GET \
    hcuri=/health \
    hcexpr=healthy_check

<ProxyHCExpr healthy_check "%{REQUEST_STATUS} == 200 && %{RESPONSE_BODY} =~ /ok/">

This adds an extra layer of confidence — the backend isn’t just returning 200, it’s returning the exact expected content.

Step 6: Monitor Health Check Results

Enable the balancer manager to watch health check results in real time:

<Location "/balancer-manager">
    SetHandler balancer-manager
    Require ip 127.0.0.1
</Location>

Members failing health checks will show as “Error” in the interface, letting you confirm your health check configuration is actually working as expected.

Step 7: Test Your Health Check Configuration

Deliberately break a backend’s health endpoint to confirm Apache reacts correctly:

# Temporarily rename or block the /health route on one backend
# Then watch balancer-manager and error logs
sudo tail -f /var/log/apache2/error.log

You should see the member get marked as failed after hcfails consecutive failures, and traffic should stop routing to it.

Real-World Use Cases

• Database-dependent applications — health checks that ping the database catch failures that a simple TCP check would completely miss.
• Rolling deployments — a backend mid-deployment can return a deliberately unhealthy status until it’s fully ready, preventing traffic from hitting a half-updated server.
• Multi-region setups — health checks that also verify connectivity to region-specific dependencies (like a regional cache) catch region-specific outages precisely.

Troubleshooting Common Issues

Health Checks Not Running at All — confirm mod_watchdog is enabled; mod_proxy_hcheck depends on it entirely and silently does nothing without it.

Member Marked Down Despite Being Healthy — check hcuri is reachable and doesn’t require authentication that Apache’s health check request doesn’t provide.

Flapping Between Healthy and Unhealthy — increase hcpasses and hcfails slightly to avoid reacting to transient blips, and check if the backend’s health endpoint itself has variable response times under load.

Security Best Practices

• Don’t expose sensitive information (stack traces, internal IPs) in health check responses.
• Restrict health check endpoints from public access where possible, or keep them lightweight enough that they can’t be abused for denial-of-service purposes.
• Log health check failures distinctly from regular application errors, making incident review easier.

Performance Optimization Tips

• Keep hcinterval reasonable — too frequent adds unnecessary load, too infrequent delays failure detection. I typically use 5-15 seconds depending on traffic criticality.
• Make health check endpoints lightweight; avoid running expensive queries on every single check.
• Use TCP checks for less critical backends and GET-based checks for anything where application-level health actually matters.

FAQs

Do I need mod_proxy_hcheck if I already have failonstatus configured? failonstatus is passive — it only reacts to real request failures. Active health checks via mod_proxy_hcheck catch problems proactively, before real users are affected. I recommend using both together.

Can health checks add noticeable load to backend servers? With reasonable intervals (5-15 seconds) and lightweight endpoints, the overhead is minimal. Just avoid pointing health checks at expensive operations.

What Apache version do I need for mod_proxy_hcheck? It was introduced in Apache 2.4.10. If you’re on an older version, you’ll need to rely on passive failure detection alone or upgrade.

Summary and Key Takeaways

Active health checks via mod_proxy_hcheck and mod_watchdog let Apache detect failing backends before real users do, rather than reacting after the fact. Building a meaningful health check endpoint that verifies actual application health — not just process uptime — combined with sensible hcinterval, hcpasses, and hcfails settings gives you a load-balanced cluster that’s genuinely self-healing.

References

• Apache mod_proxy_hcheck Documentation
• Apache mod_watchdog Documentation
• Apache mod_proxy_balancer Documentation