If you’ve ever run a Galera cluster in production, you already know the database part is only half the battle. The other half — the part that quietly determines whether your “highly available” cluster actually behaves that way during a failover — is the load balancer sitting in front of it. I’ve been burned before by a static load balancer config that kept sending traffic to a node that had silently dropped out of the cluster. That single mistake taught me why dynamic health-aware load balancing isn’t optional for Galera; it’s the whole point.
In this guide I’ll walk through why Galera needs special load balancing logic, how to set it up with HAProxy using clustercheck/xinetd health checks, and how to make the whole thing reconfigure itself automatically as nodes join, leave, or go into a non-primary state.
Why Galera Load Balancing Is Different From Regular MySQL
A standard master-replica MySQL setup usually routes writes to one master and reads to replicas. Galera is a multi-master, synchronous (virtually synchronous) cluster — every node can technically accept writes. But “can” isn’t “should”:
- Write conflicts: If you write to multiple nodes simultaneously, you increase the chance of certification conflicts and rollbacks under load.
- Flow control: A lagging node can throttle the whole cluster, so blindly balancing writes across all nodes evenly isn’t ideal.
- Node state matters: A node can be
Synced,Donor/Desynced, orJoining. Sending traffic to aJoiningnode means talking to a node with stale or incomplete data.
The standard pattern is: send all writes to a single node at a time (or a small write pool), and distribute reads across all healthy, synced nodes. The load balancer needs to know, in near real time, which nodes are actually fit to serve traffic.
Architecture Overview
+----------------+
Clients ---> | HAProxy |
+----------------+
| | |
node1 node2 node3
(Galera) (Galera) (Galera)
HAProxy polls each node’s health-check port. That port is served by clustercheck, a small script that queries wsrep_local_state and returns HTTP 200 (healthy) or 503 (unhealthy) accordingly. This is what makes the balancing “dynamic” — HAProxy doesn’t need a static list of “good” nodes; it discovers node health continuously.
Step 1: Install clustercheck on Each Galera Node
clustercheck ships with Percona XtraDB Cluster and MariaDB Galera packages. If you’re using it standalone:
# On each Galera node
sudo apt-get install -y percona-xtradb-cluster-client xinetd
Create a MySQL user the health check can use:
CREATE USER 'clustercheckuser'@'localhost' IDENTIFIED BY 'ClusterCheckPassword123!';
GRANT PROCESS ON *.* TO 'clustercheckuser'@'localhost';
FLUSH PRIVILEGES;
Edit /etc/sysconfig/clustercheck (or /etc/default/clustercheck on Debian-based systems):
MYSQL_USERNAME="clustercheckuser"
MYSQL_PASSWORD="ClusterCheckPassword123!"
MYSQL_HOST="localhost"
MYSQL_PORT="3306"
AVAILABLE_WHEN_DONOR=0
AVAILABLE_WHEN_READONLY=1
AVAILABLE_WHEN_DONOR_IS_MASTER=0
AVAILABLE_WHEN_DONOR=0 is important — you generally don’t want a node acting as an SST/IST donor to also be taking production traffic, since it’s under extra load and possibly desynced.
Configure xinetd to expose this as an HTTP-style health check on port 9200:
cat <<'EOF' | sudo tee /etc/xinetd.d/mysqlchk
service mysqlchk
{
disable = no
flags = REUSE
socket_type = stream
port = 9200
wait = no
user = nobody
server = /usr/bin/clustercheck
log_on_failure += USERID
only_from = 0.0.0.0/0
per_source = UNLIMITED
}
EOF
sudo systemctl restart xinetd
sudo systemctl enable xinetd
Test it locally:
curl -i http://localhost:9200/
Expected output for a healthy, synced node:
HTTP/1.1 200 OK
Content-Type: text/plain
Connection: close
Content-Length: 40
Percona XtraDB Cluster Node is synced.
An unhealthy node returns HTTP/1.1 503 Service Unavailable.
Step 2: Configure HAProxy for Dynamic Health-Based Routing
Install HAProxy on your load balancer host:
sudo apt-get install -y haproxy
/etc/haproxy/haproxy.cfg:
global
log /dev/log local0
maxconn 4096
defaults
log global
mode tcp
option tcplog
timeout connect 5s
timeout client 30s
timeout server 30s
retries 3
# Write traffic: single active node, automatic failover
frontend galera_write_front
bind *:3307
default_backend galera_write_back
backend galera_write_back
option httpchk
http-check expect status 200
default-server port 9200 inter 2s fall 3 rise 2
server node1 10.0.0.11:3306 check
server node2 10.0.0.12:3306 check backup
server node3 10.0.0.13:3306 check backup
# Read traffic: distributed across all synced nodes
frontend galera_read_front
bind *:3308
default_backend galera_read_back
backend galera_read_back
balance leastconn
option httpchk
http-check expect status 200
default-server port 9200 inter 2s fall 3 rise 2
server node1 10.0.0.11:3306 check
server node2 10.0.0.12:3306 check
server node3 10.0.0.13:3306 check
listen stats
bind *:8404
stats enable
stats uri /stats
stats refresh 5s
What’s happening here:
- Write backend:
node1is primary;node2andnode3arebackup, only receiving traffic ifnode1fails its health check. This avoids multi-node write conflicts. - Read backend:
balance leastconnspreads read connections across whichever nodes are currently passing theclustercheckhealth probe. inter 2s fall 3 rise 2means HAProxy checks every 2 seconds, marks a server down after 3 consecutive failures, and marks it back up after 2 consecutive successes — this is the “dynamic” part: no manual intervention needed when a node drops or rejoins.
Reload HAProxy after any config change:
sudo systemctl reload haproxy
Verify with the built-in stats page at http://<lb-ip>:8404/stats — you’ll see live UP/DOWN status per backend server.
Step 3: Simulate a Failover to Prove It’s Dynamic
Stop MySQL on the current write-primary:
# on node1
sudo systemctl stop mysql
Within a few seconds, curl http://node1:9200/ will return connection refused, HAProxy will mark node1 down, and traffic on port 3307 will automatically shift to node2 (the next available backup server) — no config reload needed.
Check HAProxy logs to confirm:
sudo tail -f /var/log/haproxy.log
Expected output snippet:
Server galera_write_back/node1 is DOWN, reason: Layer4 connection problem
Server galera_write_back/node2 is UP
Bring node1 back and let Galera perform IST/SST to resync it — once clustercheck reports 200 again, HAProxy will automatically re-add it to the pool.
Alternative: ProxySQL for Smarter Query-Aware Routing
HAProxy operates at TCP/HTTP-check level and doesn’t understand SQL. For more advanced routing (query splitting, read/write splitting based on query pattern, connection multiplexing), ProxySQL is the more common modern choice for Galera:
docker run -d --name proxysql \
-p 6033:6033 -p 6032:6032 \
-v proxysql-data:/var/lib/proxysql \
proxysql/proxysql:latest
ProxySQL maintains a mysql_galera_hostgroups table that natively understands wsrep_local_state, so it can dynamically detect donor/joining/synced states without needing an external xinetd health-check port at all. If you’re building a new deployment, it’s worth evaluating instead of HAProxy — but HAProxy remains the simpler, more transparent, and widely documented option, which is why it’s the default in most tutorials.
Security Considerations
- Restrict the
clustercheckhealth-check port (9200) to only the load balancer’s IP using firewall rules (ufw,iptables, or security groups) — don’t expose it publicly. - Use a dedicated, minimally privileged MySQL user for health checks (
PROCESSprivilege only, no data access). - Terminate TLS at HAProxy if clients connect over the internet, and use
sslmode=require(or stricter) for the backend MySQL connections if they traverse untrusted networks. - Keep the HAProxy stats page behind authentication:
listen stats
bind *:8404
stats enable
stats uri /stats
stats auth admin:StrongAdminPassword123!
Monitoring and Troubleshooting
| Symptom | Likely Cause | Fix |
|---|---|---|
| All backends show DOWN | xinetd/clustercheck not running or firewalled | systemctl status xinetd, check port 9200 reachability |
| Writes fail after node1 restarts | Node still in Joining state (SST in progress) | Wait for SST/IST to finish; check wsrep_local_state_comment |
| Reads return stale data | AVAILABLE_WHEN_DONOR=1 allowing donor traffic | Set AVAILABLE_WHEN_DONOR=0 |
| HAProxy flapping a node | inter/fall/rise too aggressive for network jitter | Increase inter to 3-5s, fall to 3-5 |
For ongoing monitoring, feed HAProxy’s stats socket into Prometheus using the haproxy_exporter, and track wsrep_cluster_size, wsrep_local_state, and wsrep_flow_control_paused from each node via mysqld_exporter for Galera-specific metrics.
Real-World Deployment Notes
In practice, I’ve found the single-writer pattern shown above works well for most OLTP workloads under moderate write volume, but it does concentrate write load on one box. For higher write throughput, some teams route writes to all three nodes and accept occasional certification-conflict rollbacks at the application layer (with retry logic on deadlock/conflict errors), trading a bit of complexity for horizontal write scaling. Whichever pattern you pick, keep the health-check interval tight enough to fail over quickly but not so tight that transient network blips cause flapping — 2 seconds with a 3-strike fall threshold has been a reliable middle ground across the clusters I’ve run. It’s also worth putting HAProxy itself behind Keepalived or a cloud load balancer with a floating VIP, so the load balancer isn’t a new single point of failure sitting in front of a cluster you built specifically to avoid single points of failure.
Summary
Dynamically load balancing a Galera cluster comes down to one core idea: never route traffic based on a static assumption of node health. By exposing wsrep_local_state through a lightweight clustercheck HTTP endpoint and letting HAProxy poll it continuously, your load balancer reacts to real cluster state — donor nodes get excluded from traffic, failed nodes get removed within seconds, and recovered nodes get automatically re-added. Combine that with a single-writer, multi-reader traffic split and you get a setup that survives node failures without a 2am page.