How to Use MySQL database with Kubernetes

How to Use MySQL database with Kubernetes

When I first tried running MySQL inside Kubernetes, I honestly underestimated how different stateful workloads are from the stateless microservices I was used to deploying. Kubernetes was built with ephemeral, disposable pods in mind, and a database is the exact opposite of that philosophy — it needs stable storage, a stable identity, and careful handling during restarts. In this guide, I want to walk you through everything I learned, from the fundamentals of MySQL architecture to running a production-grade MySQL cluster on Kubernetes, complete with YAML manifests, SQL commands, performance tuning, and troubleshooting tips.

This article is long because the topic deserves it. I’ll take you from the basics all the way to advanced operator-based deployments, so whether you’re a beginner or an experienced DBA moving into cloud-native infrastructure, you should find something useful here.

Table of Contents

  1. MySQL Architecture Fundamentals
  2. Why Running MySQL on Kubernetes Is Hard
  3. Kubernetes Primitives You Need to Know
  4. Deploying MySQL on Kubernetes Step by Step
  5. Using StatefulSets for MySQL
  6. Persistent Storage and Volume Management
  7. Configuring MySQL with ConfigMaps and Secrets
  8. Connecting Applications to MySQL in Kubernetes
  9. Replication and High Availability
  10. Using MySQL Operators
  11. Backup and Restore Strategies
  12. Security Best Practices
  13. Performance Tuning and Optimization
  14. Troubleshooting Common Issues
  15. Interview Questions
  16. FAQs
  17. Summary and Key Takeaways
  18. References

1. MySQL Architecture Fundamentals

Before touching Kubernetes at all, I think it’s important to understand what MySQL actually is under the hood, because that understanding directly informs how you should deploy it.

MySQL follows a layered architecture:

  • Connection Layer – handles client authentication, connection pooling, and thread management.
  • SQL Layer – parses queries, performs optimization, and decides the execution plan.
  • Storage Engine Layer – this is where the actual data lives. InnoDB is the default engine and the one I recommend for almost every use case because it supports transactions, row-level locking, and crash recovery.
  • File System Layer – InnoDB writes to tablespace files (.ibd), redo logs, and the data dictionary.
graph TD
    A[Client Application] --> B[Connection Layer]
    B --> C[SQL Parser & Optimizer]
    C --> D[Storage Engine - InnoDB]
    D --> E[Redo Log]
    D --> F[Tablespace Files]
    D --> G[Buffer Pool - Memory]
    G --> F

The buffer pool is the single most important memory structure in InnoDB. It caches data and index pages so that reads don’t always hit disk. When I run MySQL on Kubernetes, sizing this buffer pool correctly relative to the pod’s memory limit is one of the first tuning decisions I make.

2. Why Running MySQL on Kubernetes Is Hard

Kubernetes pods are designed to be killed and recreated at any time. A database can’t tolerate losing its data when that happens. Here are the challenges I ran into:

  • Storage persistence – pod-local storage disappears when a pod is rescheduled.
  • Stable network identity – replication requires nodes to know each other’s addresses reliably.
  • Ordered startup/shutdown – a primary needs to come up before replicas try to sync.
  • Resource contention – noisy neighbor pods can starve MySQL of CPU and I/O.
  • Failover complexity – Kubernetes doesn’t understand MySQL replication topology natively.

This is why I never recommend running MySQL as a plain Deployment. You need StatefulSets, Persistent Volumes, and ideally an operator.

3. Kubernetes Primitives You Need to Know

PrimitivePurpose in a MySQL Deployment
StatefulSetProvides stable pod names and ordered deployment/scaling
PersistentVolume (PV)Actual storage resource in the cluster
PersistentVolumeClaim (PVC)Request for storage by a pod
ConfigMapStores my.cnf configuration
SecretStores database credentials
Service (Headless)Gives each MySQL pod a stable DNS name
StorageClassDefines how volumes are dynamically provisioned

4. Deploying MySQL on Kubernetes Step by Step

Let me walk through a simple single-instance deployment first, then move to a full StatefulSet-based replica setup.

Step 1: Create a Namespace

kubectl create namespace mysql-demo

Step 2: Create a Secret for Credentials

kubectl create secret generic mysql-secret \
  --from-literal=MYSQL_ROOT_PASSWORD=StrongPass123! \
  --from-literal=MYSQL_DATABASE=appdb \
  --from-literal=MYSQL_USER=appuser \
  --from-literal=MYSQL_PASSWORD=AppPass123! \
  -n mysql-demo

Step 3: Create a ConfigMap for MySQL Configuration

apiVersion: v1
kind: ConfigMap
metadata:
  name: mysql-config
  namespace: mysql-demo
data:
  my.cnf: |
    [mysqld]
    innodb_buffer_pool_size=512M
    max_connections=200
    innodb_log_file_size=128M
    character-set-server=utf8mb4
    collation-server=utf8mb4_unicode_ci

Step 4: Create a PersistentVolumeClaim

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mysql-pvc
  namespace: mysql-demo
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi
  storageClassName: standard

Step 5: Deploy MySQL

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mysql
  namespace: mysql-demo
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mysql
  template:
    metadata:
      labels:
        app: mysql
    spec:
      containers:
        - name: mysql
          image: mysql:8.0
          envFrom:
            - secretRef:
                name: mysql-secret
          ports:
            - containerPort: 3306
          volumeMounts:
            - name: mysql-storage
              mountPath: /var/lib/mysql
            - name: mysql-config
              mountPath: /etc/mysql/conf.d
      volumes:
        - name: mysql-storage
          persistentVolumeClaim:
            claimName: mysql-pvc
        - name: mysql-config
          configMap:
            name: mysql-config

Apply everything:

kubectl apply -f mysql-deployment.yaml -n mysql-demo

Verify:

kubectl get pods -n mysql-demo
kubectl logs -f mysql-<pod-id> -n mysql-demo

Expected output once ready:

[Server] /usr/sbin/mysqld: ready for connections.
Version: '8.0.36'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306

5. Using StatefulSets for MySQL

A single Deployment is fine for testing, but I never use it for anything real. For production, I switch to a StatefulSet, which gives each pod a stable, predictable name like mysql-0, mysql-1, mysql-2 — critical for replication.

apiVersion: v1
kind: Service
metadata:
  name: mysql-headless
  namespace: mysql-demo
spec:
  clusterIP: None
  selector:
    app: mysql
  ports:
    - port: 3306
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mysql
  namespace: mysql-demo
spec:
  serviceName: mysql-headless
  replicas: 3
  selector:
    matchLabels:
      app: mysql
  template:
    metadata:
      labels:
        app: mysql
    spec:
      containers:
        - name: mysql
          image: mysql:8.0
          envFrom:
            - secretRef:
                name: mysql-secret
          ports:
            - containerPort: 3306
          volumeMounts:
            - name: data
              mountPath: /var/lib/mysql
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: [ "ReadWriteOnce" ]
        resources:
          requests:
            storage: 20Gi

Each pod gets its own PVC automatically (data-mysql-0, data-mysql-1, etc.), which is exactly what I want for replicated nodes — they should never share the same disk.

6. Persistent Storage and Volume Management

I always pick my StorageClass based on the underlying cloud provider’s fastest reliable disk type — for example, gp3 on AWS EBS or Premium SSD on Azure Disk. Network-attached storage like NFS is usually too slow for InnoDB’s random I/O pattern under real load.

graph LR
    Pod[MySQL Pod] --> PVC[PersistentVolumeClaim]
    PVC --> PV[PersistentVolume]
    PV --> Disk[Cloud Block Storage]

A mistake I made early on was using the default StorageClass without checking its reclaim policy. If it’s set to Delete, your data disappears the moment the PVC is removed. I now always verify:

kubectl get storageclass
kubectl describe storageclass standard

7. Configuring MySQL with ConfigMaps and Secrets

I keep configuration and secrets strictly separate. ConfigMaps are fine for non-sensitive tuning parameters, but credentials always go into Secrets, and in real production clusters I integrate with an external secret manager like HashiCorp Vault or AWS Secrets Manager rather than relying purely on base64-encoded Kubernetes Secrets, since base64 is encoding, not encryption.

8. Connecting Applications to MySQL in Kubernetes

Inside the cluster, an app connects using the service DNS name:

mysql-headless.mysql-demo.svc.cluster.local:3306

Example connection string for a Node.js app:

const mysql = require('mysql2');
const connection = mysql.createConnection({
  host: 'mysql-headless.mysql-demo.svc.cluster.local',
  user: 'appuser',
  password: process.env.DB_PASSWORD,
  database: 'appdb'
});

From outside the cluster, I expose MySQL only when absolutely necessary, typically through a LoadBalancer service restricted by network policy, or better, through a bastion/port-forward for admin tasks:

kubectl port-forward svc/mysql-headless 3306:3306 -n mysql-demo

9. Replication and High Availability

Once the StatefulSet is running, I configure classic MySQL replication or Group Replication depending on the consistency guarantees I need.

On the primary (mysql-0):

CREATE USER 'repl'@'%' IDENTIFIED BY 'ReplPass123!';
GRANT REPLICATION SLAVE ON *.* TO 'repl'@'%';
FLUSH PRIVILEGES;
SHOW MASTER STATUS;

Output:

+------------------+----------+--------------+------------------+
| File             | Position | Binlog_Do_DB | Binlog_Ignore_DB |
+------------------+----------+--------------+------------------+
| binlog.000003    |      154 |              |                  |
+------------------+----------+--------------+------------------+

On each replica (mysql-1, mysql-2):

CHANGE MASTER TO
  MASTER_HOST='mysql-0.mysql-headless.mysql-demo.svc.cluster.local',
  MASTER_USER='repl',
  MASTER_PASSWORD='ReplPass123!',
  MASTER_LOG_FILE='binlog.000003',
  MASTER_LOG_POS=154;
START SLAVE;
SHOW SLAVE STATUS\G

I always check that Slave_IO_Running and Slave_SQL_Running both read Yes before trusting a replica.

sequenceDiagram
    participant App
    participant Primary as mysql-0 (Primary)
    participant Replica1 as mysql-1 (Replica)
    participant Replica2 as mysql-2 (Replica)
    App->>Primary: Write transaction
    Primary->>Primary: Write to binlog
    Primary-->>Replica1: Stream binlog events
    Primary-->>Replica2: Stream binlog events
    App->>Replica1: Read query
    App->>Replica2: Read query

10. Using MySQL Operators

Once I moved past hand-rolled StatefulSets, I started using Kubernetes Operators, which automate failover, backups, and scaling. The ones I’ve worked with most are:

  • Percona XtraDB Cluster Operator — great for synchronous multi-primary clusters.
  • Oracle MySQL Operator — official but less actively maintained.
  • Bitpoke MySQL Operator — lightweight, good for GCP-based clusters.

Installing Percona’s operator with Helm:

helm repo add percona https://percona.github.io/percona-helm-charts/
helm install my-cluster percona/pxc-operator

Then applying a cluster custom resource:

apiVersion: pxc.percona.com/v1
kind: PerconaXtraDBCluster
metadata:
  name: cluster1
spec:
  pxc:
    size: 3
    image: percona/percona-xtradb-cluster:8.0
    resources:
      requests:
        memory: 1G
        cpu: 500m

An operator handles things I used to do manually — like promoting a new primary during a failure — through a controller loop that continuously reconciles the cluster’s actual state with the desired state.

11. Backup and Restore Strategies

I never rely on a single backup method. My usual approach combines logical and physical backups.

Logical backup with mysqldump, run as a Kubernetes CronJob:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: mysql-backup
  namespace: mysql-demo
spec:
  schedule: "0 2 * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
            - name: backup
              image: mysql:8.0
              command:
                - /bin/sh
                - -c
                - >
                  mysqldump -h mysql-headless -u root -p$MYSQL_ROOT_PASSWORD
                  appdb > /backup/appdb-$(date +%F).sql
              envFrom:
                - secretRef:
                    name: mysql-secret
              volumeMounts:
                - name: backup-storage
                  mountPath: /backup
          restartPolicy: OnFailure
          volumes:
            - name: backup-storage
              persistentVolumeClaim:
                claimName: backup-pvc

For larger datasets, I switch to Percona XtraBackup, which performs a physical, near-instant snapshot without locking tables for long periods.

Restoring:

mysql -h mysql-headless -u root -p appdb < appdb-2026-07-01.sql

12. Security Best Practices

  • Never hardcode passwords in YAML manifests — always use Secrets or an external vault.
  • Enforce TLS between application pods and MySQL using require_secure_transport=ON.
  • Apply NetworkPolicies so only application namespaces can reach port 3306.
  • Run the MySQL container as a non-root user where possible.
  • Rotate credentials regularly and audit with mysql.general_log sparingly, since it has a performance cost.
  • Restrict GRANT privileges — application users should never have SUPER or GRANT OPTION.
CREATE USER 'appuser'@'%' IDENTIFIED BY 'AppPass123!';
GRANT SELECT, INSERT, UPDATE, DELETE ON appdb.* TO 'appuser'@'%';
FLUSH PRIVILEGES;

13. Performance Tuning and Optimization

I tune three layers when performance issues show up: the pod resource limits, the InnoDB engine settings, and the query patterns themselves.

Setting resource requests/limits so MySQL isn’t throttled or OOM-killed:

resources:
  requests:
    memory: "2Gi"
    cpu: "1"
  limits:
    memory: "4Gi"
    cpu: "2"

A good rule I follow: innodb_buffer_pool_size should be roughly 70-75% of the container’s memory limit, never 100%, because MySQL also needs memory for connections, sort buffers, and the OS page cache.

SHOW VARIABLES LIKE 'innodb_buffer_pool_size';
SHOW ENGINE INNODB STATUS\G
EXPLAIN SELECT * FROM orders WHERE customer_id = 105;

I also add indexes based on actual query patterns rather than guessing:

CREATE INDEX idx_customer_id ON orders(customer_id);
Tuning AreaRecommendation
Buffer Pool70-75% of container memory
ConnectionsMatch to app connection pool size, not unlimited
DiskUse SSD-backed StorageClass
LoggingKeep slow query log on, general log off in production
IndexingIndex based on EXPLAIN output, not guesswork

14. Troubleshooting Common Issues

SymptomLikely CauseFix
Pod stuck in PendingPVC can’t bindCheck StorageClass and available capacity
CrashLoopBackOffBad config in ConfigMapCheck kubectl logs for syntax errors in my.cnf
Slow queries after restartCold buffer poolEnable innodb_buffer_pool_dump_at_shutdown / innodb_buffer_pool_load_at_startup
Replica not syncingWrong binlog positionRe-run CHANGE MASTER TO with correct SHOW MASTER STATUS values
Too many connectionsPool exhaustionRaise max_connections or fix leaking app connections

Useful debugging commands:

kubectl describe pod mysql-0 -n mysql-demo
kubectl exec -it mysql-0 -n mysql-demo -- mysql -u root -p
kubectl get events -n mysql-demo --sort-by='.lastTimestamp'

15. Interview Questions

  1. Why shouldn’t you run MySQL as a plain Kubernetes Deployment?
  2. What’s the difference between a StatefulSet and a Deployment for stateful workloads?
  3. How does a headless Service help with MySQL replication in Kubernetes?
  4. What is the role of a PersistentVolumeClaim versus a PersistentVolume?
  5. How would you perform a zero-downtime failover in a Kubernetes-hosted MySQL cluster?
  6. What’s the risk of setting innodb_buffer_pool_size too close to the pod’s memory limit?
  7. How do MySQL Operators simplify cluster management compared to manual StatefulSets?

16. FAQs

Can I run MySQL on Kubernetes for production workloads? Yes, but I’d only do it with a StatefulSet, proper persistent storage, and ideally an operator that automates failover and backups.

Should I use a managed database instead of self-hosting on Kubernetes? If your team is small or you don’t have dedicated DBA expertise, a managed service like Amazon RDS or Azure Database for MySQL is usually less risky.

Is NFS suitable for MySQL storage in Kubernetes? Generally no. I’ve found NFS too slow and inconsistent for InnoDB’s I/O patterns; block storage is a better fit.

How do I scale reads in a Kubernetes MySQL deployment? Add read replicas via the StatefulSet and route read traffic to them using a separate Service or a proxy like ProxySQL.

17. Summary and Key Takeaways

Running MySQL on Kubernetes is absolutely doable, but it demands respect for what a database actually needs: stable storage, stable identity, and careful failover handling. I always reach for StatefulSets over Deployments, dedicated PersistentVolumeClaims per replica, and — for anything serious — an operator that automates the operational toil I used to do by hand. Tune the buffer pool relative to container memory, separate secrets from configuration, and never skip backups just because Kubernetes “feels” resilient. Kubernetes handles orchestration; it does not understand your data.

18. References

Total
3
Shares

Leave a Reply

Previous Post
How to Use MySQL Database with Jenkins

How to Use MySQL Database with Jenkins

Next Post
How to Create a Basic Spreadsheet in Excel

How to Create a Basic Spreadsheet in Excel

Related Posts