How to Set Up Pod Disruption Budgets with Thanos in Kubernetes

Thanos extends Prometheus with long-term storage, global querying, and high-availability capabilities across multiple Prometheus instances — and like any distributed system running on Kubernetes, it needs protection from routine cluster maintenance taking down too many components at once. In this guide, I’ll walk through configuring Pod Disruption Budgets (PDBs) for a Thanos deployment, component by component, since Thanos isn’t a single monolith but a set of cooperating services each with different availability requirements.

Why Thanos Needs Careful PDB Planning

Thanos is composed of several distinct components, each with different failure characteristics:

• Sidecar: Runs alongside each Prometheus Pod, uploads blocks to object storage, and serves Prometheus’s local data to Queriers. Tightly coupled to its Prometheus instance.
• Querier: Stateless, fans out queries across Sidecars and Store Gateways, deduplicates results.
• Store Gateway: Serves historical data from object storage; often stateful with local caching.
• Compactor: Downsamples and compacts blocks in object storage — typically a single replica, since concurrent compaction against the same storage bucket can cause corruption.
• Receiver: Accepts remote-write traffic in HA setups; loses data if too many replicas go down before flushing.

Because the Compactor must run as a singleton, and the Receiver/Store Gateway have real availability requirements, a one-size-fits-all PDB strategy doesn’t work here — each component needs its own budget tuned to its actual replication and failure tolerance.

Kubernetes Architecture Refresher: How PDBs Are Enforced

A PDB doesn’t block direct kubectl delete pod calls. It only governs the Eviction API, which is what kubectl drain, the Cluster Autoscaler, and managed node upgrades use. When an eviction request comes in:

1. The API server checks all PDBs whose selector matches the target Pod.
2. If evicting would drop available replicas below minAvailable (or exceed maxUnavailable), the request is denied with 429 Too Many Requests.
3. The evicting controller retries later, once conditions allow.

This means PDBs protect you specifically during voluntary cluster operations — node drains, cordoning for upgrades, and autoscaler scale-downs — which is exactly when a multi-component system like Thanos is most at risk of losing more replicas at once than it can tolerate.

Step 1: Deploy Thanos via Helm

Using the community kube-prometheus-stack or standalone thanos chart:

helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo update
helm install thanos bitnami/thanos -n monitoring --create-namespace \
  --set query.replicaCount=2 \
  --set storegateway.replicaCount=2 \
  --set receive.replicaCount=3 \
  --set compactor.enabled=true

Check what’s running:

kubectl get pods -n monitoring -l app.kubernetes.io/name=thanos

Step 2: PDB for the Querier (Stateless, Safe to Disrupt Gradually)

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: thanos-query-pdb
  namespace: monitoring
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: thanos-query

With 2 replicas and maxUnavailable: 1, at least one Querier always stays up to serve Grafana dashboards during node maintenance.

Step 3: PDB for the Store Gateway

Store Gateways cache index data locally and can be expensive to “warm up” again after a restart, so it’s worth being conservative:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: thanos-storegateway-pdb
  namespace: monitoring
spec:
  minAvailable: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: thanos-storegateway

With minAvailable: 1 on a 2-replica StatefulSet, only one can be evicted at a time — the other keeps serving historical query traffic while its sibling restarts and re-warms its cache.

Step 4: PDB for the Receiver (Data-Loss Sensitive)

Receivers accept remote-write traffic; taking down too many at once during a rolling node upgrade can cause dropped samples if your remote-write clients don’t buffer well. With 3 replicas and a typical replication factor of 2 in the Receiver hashring:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: thanos-receive-pdb
  namespace: monitoring
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app.kubernetes.io/name: thanos-receive

This guarantees the hashring never drops below the minimum needed to satisfy its own replication factor during voluntary disruptions.

Step 5: The Compactor — Deliberately No PDB (or a Trivial One)

Since the Compactor typically runs as a single replica by design (running more than one against the same bucket risks corrupting compacted blocks), a minAvailable: 1 PDB would make it impossible to ever drain the node it’s on. For a singleton like this, it’s usually better to not create a PDB at all, and instead rely on it simply restarting elsewhere after eviction — accept the short gap in compaction rather than blocking cluster maintenance indefinitely.

# Deliberately omitted: no PDB for thanos-compactor
# Rationale: single replica by design; a minAvailable:1 PDB would
# permanently block node drains for the node it's scheduled on.

If you want some protection without blocking drains forever, use maxUnavailable: 0 combined with a PriorityClass and tight pod anti-affinity instead — but understand this still allows involuntary disruption, it just discourages voluntary ones without hard-blocking them indefinitely (this requires care and isn’t a universal recommendation).

Step 6: Apply and Verify

kubectl apply -f thanos-pdbs.yaml
kubectl get pdb -n monitoring

NAME                       MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE
thanos-query-pdb           N/A             1                 1                     2m
thanos-storegateway-pdb    1               N/A               1                     2m
thanos-receive-pdb         2               N/A               1                     2m

Step 7: Simulate a Drain and Observe

kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data

Watch how eviction proceeds component by component — Queriers and Store Gateways drain one at a time per their budgets, while Receiver Pods stop being evicted once only 2 remain, with the drain command retrying automatically until the third becomes safe to move (once a replacement is healthy elsewhere).

High Availability and Disaster Recovery

• Spread Thanos components across multiple availability zones with topology spread constraints — PDBs limit disruption during maintenance, but zone-level redundancy protects against an entire zone outage, which PDBs cannot do anything about.
• Ensure object storage (S3/GCS/Azure Blob) used by Thanos has its own durability guarantees and versioning — PDBs protect compute availability, not the underlying data.
• Regularly test full restores from the object storage bucket to confirm your Store Gateways and Compactor can rebuild state if the cluster itself is lost.

Common Mistakes

• Applying the same PDB template to every component without considering each one’s actual replication semantics — this is the single most common mistake with Thanos specifically, since it’s not architecturally uniform like a simple stateless web app.
• Forgetting the Compactor is a singleton and accidentally blocking node drains with an overly strict PDB.
• Setting minAvailable on the Receiver lower than its hashring replication factor, silently risking dropped writes during maintenance even though the PDB “passes.”

Summary

Thanos isn’t one workload — it’s several, each with distinct availability semantics, and your PDB strategy needs to reflect that. Give the Querier and Store Gateway conservative budgets, protect the Receiver hashring’s replication factor explicitly, and think carefully before applying any PDB at all to the singleton Compactor. Done right, this lets you drain and upgrade nodes confidently without silently degrading your monitoring stack’s own reliability — which is the last thing you want to fail quietly.

References

• Kubernetes Documentation: Pod Disruption Budgets
• Thanos Documentation
• Thanos GitHub Repository
• CNCF Thanos Project Page