When I first tried running MySQL inside Kubernetes, I honestly underestimated how different stateful workloads are from the stateless microservices I was used to deploying. Kubernetes was built with ephemeral, disposable pods in mind, and a database is the exact opposite of that philosophy — it needs stable storage, a stable identity, and careful handling during restarts. In this guide, I want to walk you through everything I learned, from the fundamentals of MySQL architecture to running a production-grade MySQL cluster on Kubernetes, complete with YAML manifests, SQL commands, performance tuning, and troubleshooting tips.
This article is long because the topic deserves it. I’ll take you from the basics all the way to advanced operator-based deployments, so whether you’re a beginner or an experienced DBA moving into cloud-native infrastructure, you should find something useful here.
Table of Contents
- MySQL Architecture Fundamentals
- Why Running MySQL on Kubernetes Is Hard
- Kubernetes Primitives You Need to Know
- Deploying MySQL on Kubernetes Step by Step
- Using StatefulSets for MySQL
- Persistent Storage and Volume Management
- Configuring MySQL with ConfigMaps and Secrets
- Connecting Applications to MySQL in Kubernetes
- Replication and High Availability
- Using MySQL Operators
- Backup and Restore Strategies
- Security Best Practices
- Performance Tuning and Optimization
- Troubleshooting Common Issues
- Interview Questions
- FAQs
- Summary and Key Takeaways
- References
1. MySQL Architecture Fundamentals
Before touching Kubernetes at all, I think it’s important to understand what MySQL actually is under the hood, because that understanding directly informs how you should deploy it.
MySQL follows a layered architecture:
- Connection Layer – handles client authentication, connection pooling, and thread management.
- SQL Layer – parses queries, performs optimization, and decides the execution plan.
- Storage Engine Layer – this is where the actual data lives. InnoDB is the default engine and the one I recommend for almost every use case because it supports transactions, row-level locking, and crash recovery.
- File System Layer – InnoDB writes to tablespace files (
.ibd), redo logs, and the data dictionary.
graph TD
A[Client Application] --> B[Connection Layer]
B --> C[SQL Parser & Optimizer]
C --> D[Storage Engine - InnoDB]
D --> E[Redo Log]
D --> F[Tablespace Files]
D --> G[Buffer Pool - Memory]
G --> F
The buffer pool is the single most important memory structure in InnoDB. It caches data and index pages so that reads don’t always hit disk. When I run MySQL on Kubernetes, sizing this buffer pool correctly relative to the pod’s memory limit is one of the first tuning decisions I make.
2. Why Running MySQL on Kubernetes Is Hard
Kubernetes pods are designed to be killed and recreated at any time. A database can’t tolerate losing its data when that happens. Here are the challenges I ran into:
- Storage persistence – pod-local storage disappears when a pod is rescheduled.
- Stable network identity – replication requires nodes to know each other’s addresses reliably.
- Ordered startup/shutdown – a primary needs to come up before replicas try to sync.
- Resource contention – noisy neighbor pods can starve MySQL of CPU and I/O.
- Failover complexity – Kubernetes doesn’t understand MySQL replication topology natively.
This is why I never recommend running MySQL as a plain Deployment. You need StatefulSets, Persistent Volumes, and ideally an operator.
3. Kubernetes Primitives You Need to Know
| Primitive | Purpose in a MySQL Deployment |
|---|---|
| StatefulSet | Provides stable pod names and ordered deployment/scaling |
| PersistentVolume (PV) | Actual storage resource in the cluster |
| PersistentVolumeClaim (PVC) | Request for storage by a pod |
| ConfigMap | Stores my.cnf configuration |
| Secret | Stores database credentials |
| Service (Headless) | Gives each MySQL pod a stable DNS name |
| StorageClass | Defines how volumes are dynamically provisioned |
4. Deploying MySQL on Kubernetes Step by Step
Let me walk through a simple single-instance deployment first, then move to a full StatefulSet-based replica setup.
Step 1: Create a Namespace
kubectl create namespace mysql-demo
Step 2: Create a Secret for Credentials
kubectl create secret generic mysql-secret \
--from-literal=MYSQL_ROOT_PASSWORD=StrongPass123! \
--from-literal=MYSQL_DATABASE=appdb \
--from-literal=MYSQL_USER=appuser \
--from-literal=MYSQL_PASSWORD=AppPass123! \
-n mysql-demo
Step 3: Create a ConfigMap for MySQL Configuration
apiVersion: v1
kind: ConfigMap
metadata:
name: mysql-config
namespace: mysql-demo
data:
my.cnf: |
[mysqld]
innodb_buffer_pool_size=512M
max_connections=200
innodb_log_file_size=128M
character-set-server=utf8mb4
collation-server=utf8mb4_unicode_ci
Step 4: Create a PersistentVolumeClaim
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: mysql-pvc
namespace: mysql-demo
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 20Gi
storageClassName: standard
Step 5: Deploy MySQL
apiVersion: apps/v1
kind: Deployment
metadata:
name: mysql
namespace: mysql-demo
spec:
replicas: 1
selector:
matchLabels:
app: mysql
template:
metadata:
labels:
app: mysql
spec:
containers:
- name: mysql
image: mysql:8.0
envFrom:
- secretRef:
name: mysql-secret
ports:
- containerPort: 3306
volumeMounts:
- name: mysql-storage
mountPath: /var/lib/mysql
- name: mysql-config
mountPath: /etc/mysql/conf.d
volumes:
- name: mysql-storage
persistentVolumeClaim:
claimName: mysql-pvc
- name: mysql-config
configMap:
name: mysql-config
Apply everything:
kubectl apply -f mysql-deployment.yaml -n mysql-demo
Verify:
kubectl get pods -n mysql-demo
kubectl logs -f mysql-<pod-id> -n mysql-demo
Expected output once ready:
[Server] /usr/sbin/mysqld: ready for connections.
Version: '8.0.36' socket: '/var/run/mysqld/mysqld.sock' port: 3306
5. Using StatefulSets for MySQL
A single Deployment is fine for testing, but I never use it for anything real. For production, I switch to a StatefulSet, which gives each pod a stable, predictable name like mysql-0, mysql-1, mysql-2 — critical for replication.
apiVersion: v1
kind: Service
metadata:
name: mysql-headless
namespace: mysql-demo
spec:
clusterIP: None
selector:
app: mysql
ports:
- port: 3306
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: mysql
namespace: mysql-demo
spec:
serviceName: mysql-headless
replicas: 3
selector:
matchLabels:
app: mysql
template:
metadata:
labels:
app: mysql
spec:
containers:
- name: mysql
image: mysql:8.0
envFrom:
- secretRef:
name: mysql-secret
ports:
- containerPort: 3306
volumeMounts:
- name: data
mountPath: /var/lib/mysql
volumeClaimTemplates:
- metadata:
name: data
spec:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 20Gi
Each pod gets its own PVC automatically (data-mysql-0, data-mysql-1, etc.), which is exactly what I want for replicated nodes — they should never share the same disk.
6. Persistent Storage and Volume Management
I always pick my StorageClass based on the underlying cloud provider’s fastest reliable disk type — for example, gp3 on AWS EBS or Premium SSD on Azure Disk. Network-attached storage like NFS is usually too slow for InnoDB’s random I/O pattern under real load.
graph LR
Pod[MySQL Pod] --> PVC[PersistentVolumeClaim]
PVC --> PV[PersistentVolume]
PV --> Disk[Cloud Block Storage]
A mistake I made early on was using the default StorageClass without checking its reclaim policy. If it’s set to Delete, your data disappears the moment the PVC is removed. I now always verify:
kubectl get storageclass
kubectl describe storageclass standard
7. Configuring MySQL with ConfigMaps and Secrets
I keep configuration and secrets strictly separate. ConfigMaps are fine for non-sensitive tuning parameters, but credentials always go into Secrets, and in real production clusters I integrate with an external secret manager like HashiCorp Vault or AWS Secrets Manager rather than relying purely on base64-encoded Kubernetes Secrets, since base64 is encoding, not encryption.
8. Connecting Applications to MySQL in Kubernetes
Inside the cluster, an app connects using the service DNS name:
mysql-headless.mysql-demo.svc.cluster.local:3306
Example connection string for a Node.js app:
const mysql = require('mysql2');
const connection = mysql.createConnection({
host: 'mysql-headless.mysql-demo.svc.cluster.local',
user: 'appuser',
password: process.env.DB_PASSWORD,
database: 'appdb'
});
From outside the cluster, I expose MySQL only when absolutely necessary, typically through a LoadBalancer service restricted by network policy, or better, through a bastion/port-forward for admin tasks:
kubectl port-forward svc/mysql-headless 3306:3306 -n mysql-demo
9. Replication and High Availability
Once the StatefulSet is running, I configure classic MySQL replication or Group Replication depending on the consistency guarantees I need.
On the primary (mysql-0):
CREATE USER 'repl'@'%' IDENTIFIED BY 'ReplPass123!';
GRANT REPLICATION SLAVE ON *.* TO 'repl'@'%';
FLUSH PRIVILEGES;
SHOW MASTER STATUS;
Output:
+------------------+----------+--------------+------------------+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB |
+------------------+----------+--------------+------------------+
| binlog.000003 | 154 | | |
+------------------+----------+--------------+------------------+
On each replica (mysql-1, mysql-2):
CHANGE MASTER TO
MASTER_HOST='mysql-0.mysql-headless.mysql-demo.svc.cluster.local',
MASTER_USER='repl',
MASTER_PASSWORD='ReplPass123!',
MASTER_LOG_FILE='binlog.000003',
MASTER_LOG_POS=154;
START SLAVE;
SHOW SLAVE STATUS\G
I always check that Slave_IO_Running and Slave_SQL_Running both read Yes before trusting a replica.
sequenceDiagram
participant App
participant Primary as mysql-0 (Primary)
participant Replica1 as mysql-1 (Replica)
participant Replica2 as mysql-2 (Replica)
App->>Primary: Write transaction
Primary->>Primary: Write to binlog
Primary-->>Replica1: Stream binlog events
Primary-->>Replica2: Stream binlog events
App->>Replica1: Read query
App->>Replica2: Read query
10. Using MySQL Operators
Once I moved past hand-rolled StatefulSets, I started using Kubernetes Operators, which automate failover, backups, and scaling. The ones I’ve worked with most are:
- Percona XtraDB Cluster Operator — great for synchronous multi-primary clusters.
- Oracle MySQL Operator — official but less actively maintained.
- Bitpoke MySQL Operator — lightweight, good for GCP-based clusters.
Installing Percona’s operator with Helm:
helm repo add percona https://percona.github.io/percona-helm-charts/
helm install my-cluster percona/pxc-operator
Then applying a cluster custom resource:
apiVersion: pxc.percona.com/v1
kind: PerconaXtraDBCluster
metadata:
name: cluster1
spec:
pxc:
size: 3
image: percona/percona-xtradb-cluster:8.0
resources:
requests:
memory: 1G
cpu: 500m
An operator handles things I used to do manually — like promoting a new primary during a failure — through a controller loop that continuously reconciles the cluster’s actual state with the desired state.
11. Backup and Restore Strategies
I never rely on a single backup method. My usual approach combines logical and physical backups.
Logical backup with mysqldump, run as a Kubernetes CronJob:
apiVersion: batch/v1
kind: CronJob
metadata:
name: mysql-backup
namespace: mysql-demo
spec:
schedule: "0 2 * * *"
jobTemplate:
spec:
template:
spec:
containers:
- name: backup
image: mysql:8.0
command:
- /bin/sh
- -c
- >
mysqldump -h mysql-headless -u root -p$MYSQL_ROOT_PASSWORD
appdb > /backup/appdb-$(date +%F).sql
envFrom:
- secretRef:
name: mysql-secret
volumeMounts:
- name: backup-storage
mountPath: /backup
restartPolicy: OnFailure
volumes:
- name: backup-storage
persistentVolumeClaim:
claimName: backup-pvc
For larger datasets, I switch to Percona XtraBackup, which performs a physical, near-instant snapshot without locking tables for long periods.
Restoring:
mysql -h mysql-headless -u root -p appdb < appdb-2026-07-01.sql
12. Security Best Practices
- Never hardcode passwords in YAML manifests — always use Secrets or an external vault.
- Enforce TLS between application pods and MySQL using
require_secure_transport=ON. - Apply NetworkPolicies so only application namespaces can reach port 3306.
- Run the MySQL container as a non-root user where possible.
- Rotate credentials regularly and audit with
mysql.general_logsparingly, since it has a performance cost. - Restrict
GRANTprivileges — application users should never haveSUPERorGRANT OPTION.
CREATE USER 'appuser'@'%' IDENTIFIED BY 'AppPass123!';
GRANT SELECT, INSERT, UPDATE, DELETE ON appdb.* TO 'appuser'@'%';
FLUSH PRIVILEGES;
13. Performance Tuning and Optimization
I tune three layers when performance issues show up: the pod resource limits, the InnoDB engine settings, and the query patterns themselves.
Setting resource requests/limits so MySQL isn’t throttled or OOM-killed:
resources:
requests:
memory: "2Gi"
cpu: "1"
limits:
memory: "4Gi"
cpu: "2"
A good rule I follow: innodb_buffer_pool_size should be roughly 70-75% of the container’s memory limit, never 100%, because MySQL also needs memory for connections, sort buffers, and the OS page cache.
SHOW VARIABLES LIKE 'innodb_buffer_pool_size';
SHOW ENGINE INNODB STATUS\G
EXPLAIN SELECT * FROM orders WHERE customer_id = 105;
I also add indexes based on actual query patterns rather than guessing:
CREATE INDEX idx_customer_id ON orders(customer_id);
| Tuning Area | Recommendation |
|---|---|
| Buffer Pool | 70-75% of container memory |
| Connections | Match to app connection pool size, not unlimited |
| Disk | Use SSD-backed StorageClass |
| Logging | Keep slow query log on, general log off in production |
| Indexing | Index based on EXPLAIN output, not guesswork |
14. Troubleshooting Common Issues
| Symptom | Likely Cause | Fix |
|---|---|---|
Pod stuck in Pending | PVC can’t bind | Check StorageClass and available capacity |
CrashLoopBackOff | Bad config in ConfigMap | Check kubectl logs for syntax errors in my.cnf |
| Slow queries after restart | Cold buffer pool | Enable innodb_buffer_pool_dump_at_shutdown / innodb_buffer_pool_load_at_startup |
| Replica not syncing | Wrong binlog position | Re-run CHANGE MASTER TO with correct SHOW MASTER STATUS values |
Too many connections | Pool exhaustion | Raise max_connections or fix leaking app connections |
Useful debugging commands:
kubectl describe pod mysql-0 -n mysql-demo
kubectl exec -it mysql-0 -n mysql-demo -- mysql -u root -p
kubectl get events -n mysql-demo --sort-by='.lastTimestamp'
15. Interview Questions
- Why shouldn’t you run MySQL as a plain Kubernetes Deployment?
- What’s the difference between a StatefulSet and a Deployment for stateful workloads?
- How does a headless Service help with MySQL replication in Kubernetes?
- What is the role of a PersistentVolumeClaim versus a PersistentVolume?
- How would you perform a zero-downtime failover in a Kubernetes-hosted MySQL cluster?
- What’s the risk of setting
innodb_buffer_pool_sizetoo close to the pod’s memory limit? - How do MySQL Operators simplify cluster management compared to manual StatefulSets?
16. FAQs
Can I run MySQL on Kubernetes for production workloads? Yes, but I’d only do it with a StatefulSet, proper persistent storage, and ideally an operator that automates failover and backups.
Should I use a managed database instead of self-hosting on Kubernetes? If your team is small or you don’t have dedicated DBA expertise, a managed service like Amazon RDS or Azure Database for MySQL is usually less risky.
Is NFS suitable for MySQL storage in Kubernetes? Generally no. I’ve found NFS too slow and inconsistent for InnoDB’s I/O patterns; block storage is a better fit.
How do I scale reads in a Kubernetes MySQL deployment? Add read replicas via the StatefulSet and route read traffic to them using a separate Service or a proxy like ProxySQL.
17. Summary and Key Takeaways
Running MySQL on Kubernetes is absolutely doable, but it demands respect for what a database actually needs: stable storage, stable identity, and careful failover handling. I always reach for StatefulSets over Deployments, dedicated PersistentVolumeClaims per replica, and — for anything serious — an operator that automates the operational toil I used to do by hand. Tune the buffer pool relative to container memory, separate secrets from configuration, and never skip backups just because Kubernetes “feels” resilient. Kubernetes handles orchestration; it does not understand your data.