Prometheus is great until it isn’t. It runs beautifully for a single cluster with a few weeks of retention, and then one day someone asks “can we see six months of trends across all four clusters?” and you realize Prometheus was never designed for that. That’s the exact wall I hit while running monitoring for a multi-cluster setup, and it’s what pushed me toward Thanos.
This article covers what Thanos actually solves, its architecture, and a full walkthrough of deploying it on Kubernetes.
Why Prometheus Alone Isn’t Enough
Prometheus stores data locally on disk, has no built-in long-term storage, and doesn’t natively support querying across multiple Prometheus instances. That’s fine for a single small cluster. It breaks down when you need:
- Long-term retention (months or years) without blowing up local disk.
- A single query interface across multiple clusters or regions.
- Highly available Prometheus without duplicate alerts firing.
- Downsampling of old data to keep long-range queries fast.
Thanos solves all four without requiring you to rip out Prometheus.
Thanos Architecture Overview
Thanos is a set of components that attach to your existing Prometheus deployments:
- Sidecar — runs alongside each Prometheus pod, uploads blocks to object storage, and exposes a StoreAPI so the Querier can read live data directly from Prometheus.
- Querier (Query) — the global query layer. It talks to sidecars, store gateways, and rulers, deduplicating data from HA Prometheus pairs.
- Store Gateway — serves historical data from object storage (S3, GCS, Azure Blob) so the Querier can reach back further than local retention.
- Compactor — downsamples and compacts blocks in object storage over time (raw → 5m → 1h resolutions).
- Ruler — evaluates recording/alerting rules against the global view, useful when rules need data from multiple Prometheus instances.
The data flow looks like this: Prometheus scrapes → Sidecar uploads 2-hour blocks to object storage → Compactor merges and downsamples them → Store Gateway serves them → Querier federates everything, live and historical, into one API.
Prerequisites
- A running Kubernetes cluster (v1.28+ recommended)
kubectlandhelminstalled and configured- An object storage bucket (S3, GCS, or MinIO for self-hosted)
- Prometheus already deployed, or willingness to deploy it via the kube-prometheus-stack
Step 1: Deploy Prometheus with Thanos Sidecar Enabled
The simplest path is via the kube-prometheus-stack Helm chart, which supports Thanos sidecar natively.
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
Create a values file:
# values-prometheus.yaml
prometheus:
prometheusSpec:
retention: 6h
thanos:
objectStorageConfig:
existingSecret:
name: thanos-objstore-config
key: objstore.yml
externalLabels:
cluster: prod-us-east
replica: $(POD_NAME)
The retention: 6h is intentional — Thanos sidecar handles long-term storage, so local Prometheus retention only needs to cover the sidecar’s upload interval plus a safety margin.
Step 2: Configure Object Storage
Create the object storage config secret. Example for S3:
# objstore.yml
type: S3
config:
bucket: my-thanos-metrics
endpoint: s3.us-east-1.amazonaws.com
region: us-east-1
access_key: <YOUR_ACCESS_KEY>
secret_key: <YOUR_SECRET_KEY>
kubectl create secret generic thanos-objstore-config \
--from-file=objstore.yml=./objstore.yml \
-n monitoring
In production, avoid embedding raw credentials — use IAM roles for service accounts (IRSA on EKS) or workload identity (GKE) instead, and drop the access_key/secret_key fields entirely.
Step 3: Install Prometheus
kubectl create namespace monitoring
helm install kube-prometheus prometheus-community/kube-prometheus-stack \
-n monitoring \
-f values-prometheus.yaml
Verify the sidecar is running:
kubectl get pods -n monitoring -l app.kubernetes.io/name=prometheus
NAME READY STATUS RESTARTS AGE
prometheus-kube-prometheus-prometheus-0 3/3 Running 0 2m
The 3/3 confirms Prometheus, the config-reloader, and the Thanos sidecar are all up.
Step 4: Deploy the Thanos Query Component
# thanos-query.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: thanos-query
namespace: monitoring
spec:
replicas: 2
selector:
matchLabels:
app: thanos-query
template:
metadata:
labels:
app: thanos-query
spec:
containers:
- name: thanos-query
image: quay.io/thanos/thanos:v0.36.1
args:
- query
- --http-address=0.0.0.0:9090
- --grpc-address=0.0.0.0:10901
- --store=dnssrv+_grpc._tcp.prometheus-kube-prometheus-prometheus-thanos-sidecar.monitoring.svc.cluster.local
- --store=dnssrv+_grpc._tcp.thanos-store-gateway.monitoring.svc.cluster.local
- --query.replica-label=replica
ports:
- containerPort: 9090
name: http
- containerPort: 10901
name: grpc
---
apiVersion: v1
kind: Service
metadata:
name: thanos-query
namespace: monitoring
spec:
selector:
app: thanos-query
ports:
- name: http
port: 9090
targetPort: 9090
kubectl apply -f thanos-query.yaml
kubectl port-forward -n monitoring svc/thanos-query 9090:9090
Now open localhost:9090 — you get a Prometheus-compatible query UI that federates across every sidecar and store gateway you registered.
Step 5: Deploy the Store Gateway
The Store Gateway lets the Querier reach historical data sitting in object storage.
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: thanos-store-gateway
namespace: monitoring
spec:
serviceName: thanos-store-gateway
replicas: 1
selector:
matchLabels:
app: thanos-store-gateway
template:
metadata:
labels:
app: thanos-store-gateway
spec:
containers:
- name: thanos-store
image: quay.io/thanos/thanos:v0.36.1
args:
- store
- --data-dir=/data
- --objstore.config-file=/etc/thanos/objstore.yml
- --grpc-address=0.0.0.0:10901
- --http-address=0.0.0.0:10902
volumeMounts:
- name: objstore-config
mountPath: /etc/thanos
- name: data
mountPath: /data
volumes:
- name: objstore-config
secret:
secretName: thanos-objstore-config
volumeClaimTemplates:
- metadata:
name: data
spec:
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 20Gi
---
apiVersion: v1
kind: Service
metadata:
name: thanos-store-gateway
namespace: monitoring
spec:
clusterIP: None
selector:
app: thanos-store-gateway
ports:
- name: grpc
port: 10901
Step 6: Deploy the Compactor
Only run one Compactor instance ever — running multiples against the same bucket corrupts data.
apiVersion: apps/v1
kind: Deployment
metadata:
name: thanos-compactor
namespace: monitoring
spec:
replicas: 1
selector:
matchLabels:
app: thanos-compactor
template:
metadata:
labels:
app: thanos-compactor
spec:
containers:
- name: thanos-compactor
image: quay.io/thanos/thanos:v0.36.1
args:
- compact
- --data-dir=/data
- --objstore.config-file=/etc/thanos/objstore.yml
- --retention.resolution-raw=30d
- --retention.resolution-5m=90d
- --retention.resolution-1h=365d
- --wait
volumeMounts:
- name: objstore-config
mountPath: /etc/thanos
- name: data
mountPath: /data
volumes:
- name: objstore-config
secret:
secretName: thanos-objstore-config
- name: data
emptyDir: {}
Multi-Cluster Setup
For multiple clusters feeding one global view, each cluster’s Prometheus sidecar needs a unique externalLabels.cluster value, and the central Querier needs network reachability to every sidecar (commonly via a Thanos Receive gateway for clusters that can’t be reached directly, or via a service mesh / VPN peering).
externalLabels:
cluster: prod-eu-west
This label is what lets the Querier disambiguate and deduplicate series coming from different clusters or HA replicas.
Grafana Integration
Point Grafana’s Prometheus data source at the Thanos Querier instead of Prometheus directly:
apiVersion: 1
datasources:
- name: Thanos
type: prometheus
access: proxy
url: http://thanos-query.monitoring.svc.cluster.local:9090
isDefault: true
Dashboards work unmodified since Thanos Querier speaks the PromQL-compatible API.
Security and RBAC Considerations
- Restrict access to the object storage bucket using least-privilege IAM policies — write access for sidecars/compactor, read-only for the store gateway.
- Put Thanos Query behind an Ingress with authentication (OAuth2 proxy or mTLS) if exposed outside the cluster.
- Use Kubernetes
NetworkPolicyto restrict gRPC traffic (port 10901) to only the Querier and relevant Thanos components.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: thanos-grpc-policy
namespace: monitoring
spec:
podSelector:
matchLabels:
app: thanos-store-gateway
ingress:
- from:
- podSelector:
matchLabels:
app: thanos-query
ports:
- protocol: TCP
port: 10901
Thanos Receive for Push-Based Ingestion
The sidecar-based architecture described above works well when the central Querier can reach every Prometheus instance’s gRPC endpoint directly. In practice, this breaks down for clusters behind restrictive firewalls, in different cloud accounts, or on-prem clusters without a direct network path to a central monitoring stack. Thanos Receive solves this with a push model instead:
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: thanos-receive
namespace: monitoring
spec:
serviceName: thanos-receive
replicas: 3
selector:
matchLabels:
app: thanos-receive
template:
metadata:
labels:
app: thanos-receive
spec:
containers:
- name: thanos-receive
image: quay.io/thanos/thanos:v0.36.1
args:
- receive
- --tsdb.path=/data
- --objstore.config-file=/etc/thanos/objstore.yml
- --grpc-address=0.0.0.0:10901
- --http-address=0.0.0.0:10902
- --remote-write.address=0.0.0.0:19291
- --label=receive_cluster="central"
volumeMounts:
- name: objstore-config
mountPath: /etc/thanos
- name: data
mountPath: /data
volumes:
- name: objstore-config
secret:
secretName: thanos-objstore-config
volumeClaimTemplates:
- metadata:
name: data
spec:
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 50Gi
Each remote cluster’s Prometheus is then configured with remote_write pointing at the central Receive endpoint, rather than requiring the central Querier to reach out to every remote sidecar:
# Remote cluster's Prometheus config
remote_write:
- url: https://thanos-receive.central-monitoring.example.com/api/v1/receive
This inverts the connectivity requirement — remote clusters only need outbound HTTPS access to the central Receive endpoint, which is almost always easier to arrange across security boundaries than opening inbound gRPC access for a central Querier to reach into every remote environment.
Querier Federation Across Both Patterns
A production setup with a mix of directly-reachable clusters (using the Sidecar pattern) and firewalled clusters (using Receive) can combine both under one Querier — it doesn’t have to be one pattern or the other cluster-wide:
--store=dnssrv+_grpc._tcp.prometheus-thanos-sidecar.monitoring.svc.cluster.local
--store=dnssrv+_grpc._tcp.thanos-receive.monitoring.svc.cluster.local
--store=dnssrv+_grpc._tcp.thanos-store-gateway.monitoring.svc.cluster.local
The Querier treats every registered --store endpoint identically at query time, regardless of whether the underlying data arrived via sidecar scraping, remote-write push, or historical object storage — this uniformity is a large part of why Thanos scales well as an organization’s cluster footprint grows across regions and network boundaries.
Common Mistakes
- Running multiple Compactors against the same bucket — causes data corruption. Always exactly one.
- Forgetting
externalLabels— without a distinguishing label, the Querier can’t tell replicas or clusters apart, and deduplication breaks. - Setting local Prometheus retention too high, defeating the point of offloading to object storage and wasting local disk.
- No lifecycle policy on the object storage bucket — raw, uncompacted blocks pile up if the Compactor falls behind.
- Ignoring downsampling — querying a year of raw 15s-resolution data across many series is what fries your Querier’s memory; downsampled resolutions (5m, 1h) exist specifically to keep long-range dashboards fast.
Disaster Recovery
Because metrics live in object storage rather than only on local PVCs, cluster rebuilds are much less painful — redeploy the Thanos components pointing at the same bucket, and historical data is immediately queryable again. Still, back up your object storage bucket with cross-region replication if metrics history is business-critical (e.g., for SLA reporting).
Summary
Thanos extends Prometheus into a globally queryable, long-term-retained, highly available monitoring system without replacing your existing setup. The Sidecar uploads data, the Querier federates across clusters and time ranges, the Store Gateway serves historical blocks, and the Compactor keeps storage lean through downsampling. For any team running more than one cluster, or needing retention beyond a few weeks, Thanos is close to the default answer in the CNCF ecosystem.
