I’ve never trusted a Kubernetes cluster I couldn’t see into. Pods can restart, nodes can silently run low on memory, and a slow creep in latency can go unnoticed for days without proper monitoring. Prometheus has become the de facto standard for Kubernetes observability, largely because its pull-based model and label-based data model map so naturally onto how Kubernetes itself works. In this guide, I’ll walk through installing Prometheus, understanding what it scrapes, writing queries, setting alerts, and visualizing everything in Grafana.
Why Prometheus Fits Kubernetes So Well
Prometheus works by periodically scraping HTTP endpoints (typically /metrics) that expose data in a simple text-based format. Kubernetes components — the kubelet, API server, and many applications — already expose metrics this way. Combined with Kubernetes’ dynamic service discovery, Prometheus can automatically find and scrape new Pods as they’re created or destroyed, without manual reconfiguration.
Step 1: Install kube-prometheus-stack
The easiest path to a fully working setup — Prometheus, Alertmanager, Grafana, and pre-built dashboards — is the community Helm chart:
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install kube-prometheus prometheus-community/kube-prometheus-stack \
--namespace monitoring \
--create-namespace
Check what got installed:
kubectl get pods -n monitoring
NAME READY STATUS RESTARTS AGE
kube-prometheus-grafana-6f8b7c9d-2kqpl 3/3 Running 0 2m
kube-prometheus-kube-prome-operator-7d9c8f6b5-9xqwt 1/1 Running 0 2m
kube-prometheus-kube-state-metrics-5f8d9c7b6-4mzp1 1/1 Running 0 2m
prometheus-kube-prometheus-kube-prome-prometheus-0 2/2 Running 0 90s
alertmanager-kube-prometheus-kube-prome-alertmanager-0 2/2 Running 0 90s
This chart deploys the Prometheus Operator, which manages Prometheus configuration declaratively through Kubernetes custom resources (ServiceMonitor, PodMonitor, PrometheusRule) rather than editing a static config file by hand — a huge quality-of-life improvement over manual Prometheus setups.
Step 2: Access the Prometheus UI
kubectl port-forward -n monitoring svc/kube-prometheus-kube-prome-prometheus 9090:9090
Open http://localhost:9090 and check Status > Targets to confirm Prometheus is successfully scraping the kubelet, API server, node exporter, and kube-state-metrics.
Step 3: Understanding What’s Being Collected
Three components do most of the heavy lifting:
- node-exporter (a DaemonSet running on every node) exposes host-level metrics: CPU, memory, disk, network.
- kube-state-metrics exposes the state of Kubernetes objects: how many Pods are pending, Deployment replica counts, Pod restart counts.
- cAdvisor, built into the kubelet, exposes per-container resource usage.
Together these give you both infrastructure-level and Kubernetes-object-level visibility.
Step 4: Writing PromQL Queries
Let’s look at a few practically useful queries. CPU usage per Pod over the last 5 minutes:
sum(rate(container_cpu_usage_seconds_total{namespace="production"}[5m])) by (pod)
Memory usage as a percentage of the limit:
sum(container_memory_working_set_bytes{namespace="production"}) by (pod)
/
sum(kube_pod_container_resource_limits{namespace="production", resource="memory"}) by (pod)
* 100
Pods that have restarted in the last hour:
increase(kube_pod_container_status_restarts_total[1h]) > 0
Nodes under memory pressure:
kube_node_status_condition{condition="MemoryPressure", status="true"} == 1
Number of Pods stuck in Pending:
sum(kube_pod_status_phase{phase="Pending"}) by (namespace)
Step 5: Setting Up Alerts
Alerts are defined declaratively as PrometheusRule custom resources, picked up automatically by the Prometheus Operator:
# high-restart-alert.yaml
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: pod-restart-alerts
namespace: monitoring
labels:
release: kube-prometheus
spec:
groups:
- name: pod-health
rules:
- alert: PodCrashLooping
expr: increase(kube_pod_container_status_restarts_total[15m]) > 3
for: 5m
labels:
severity: warning
annotations:
summary: "Pod {{ $labels.pod }} is crash looping"
description: "{{ $labels.pod }} in {{ $labels.namespace }} has restarted more than 3 times in 15 minutes."
- alert: NodeMemoryPressure
expr: kube_node_status_condition{condition="MemoryPressure", status="true"} == 1
for: 5m
labels:
severity: critical
annotations:
summary: "Node {{ $labels.node }} under memory pressure"
kubectl apply -f high-restart-alert.yaml
Note the release: kube-prometheus label — the Prometheus Operator only picks up PrometheusRule objects matching the label selector configured on the Prometheus instance, so this label must match your Helm release name or the rule silently gets ignored.
Step 6: Configuring Alertmanager Routing
Alerts need somewhere to go — Slack, PagerDuty, email:
# alertmanager-config.yaml
apiVersion: v1
kind: Secret
metadata:
name: alertmanager-kube-prometheus-kube-prome-alertmanager
namespace: monitoring
type: Opaque
stringData:
alertmanager.yaml: |
route:
receiver: slack-notifications
group_by: ['alertname', 'namespace']
group_wait: 30s
repeat_interval: 4h
receivers:
- name: slack-notifications
slack_configs:
- api_url: https://hooks.slack.com/services/XXX/YYY/ZZZ
channel: '#k8s-alerts'
send_resolved: true
kubectl apply -f alertmanager-config.yaml
Step 7: Visualizing in Grafana
Get the Grafana admin password and access it:
kubectl get secret -n monitoring kube-prometheus-grafana \
-o jsonpath="{.data.admin-password}" | base64 --decode
kubectl port-forward -n monitoring svc/kube-prometheus-grafana 3000:80
The chart ships pre-built dashboards for cluster overview, node resource usage, and Pod-level metrics out of the box — I’d start there rather than building dashboards from scratch. From Dashboards > Browse, look for “Kubernetes / Compute Resources / Namespace (Pods)” as a good daily-driver view.
Step 8: Scraping Your Own Application’s Metrics
For custom app metrics (using a client library like prometheus-client for Python or prom-client for Node), expose a /metrics endpoint and create a ServiceMonitor so the Operator picks it up automatically:
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: myapp-monitor
namespace: production
labels:
release: kube-prometheus
spec:
selector:
matchLabels:
app: myapp
endpoints:
- port: metrics
interval: 30s
path: /metrics
This requires the target Service to have a named port called metrics:
apiVersion: v1
kind: Service
metadata:
name: myapp-service
labels:
app: myapp
spec:
selector:
app: myapp
ports:
- name: metrics
port: 9100
targetPort: 9100
Debugging Common Issues
kubectl get servicemonitors -n production
kubectl logs -n monitoring prometheus-kube-prometheus-kube-prome-prometheus-0 -c prometheus
If a target doesn’t appear in Prometheus’s target list, double-check the release label matches what the Prometheus CR expects (kubectl get prometheus -n monitoring -o yaml shows the serviceMonitorSelector), and confirm the Service’s port name matches the ServiceMonitor‘s endpoints.port.
Best Practices
- Set retention and storage sizing deliberately — high-cardinality labels (like raw user IDs) can blow up Prometheus’s memory usage fast.
- Use recording rules for expensive queries dashboards run frequently, pre-computing them instead of recalculating on every load.
- Federate or use Thanos/Mimir for long-term storage and multi-cluster aggregation once a single Prometheus instance isn’t enough.
- Alert on symptoms (latency, error rate) as much as on causes (CPU, memory) — the former tells you when users are actually affected.
- Keep Alertmanager routing simple at first; overly clever routing rules become their own maintenance burden.
Common Mistakes
- Forgetting the
releaselabel onServiceMonitor/PrometheusRuleresources, so they’re silently ignored. - Alerting on every possible metric, leading to alert fatigue and ignored pages.
- Not setting resource limits on Prometheus itself, letting it consume unbounded memory on a high-cardinality cluster.
- Relying only on Grafana dashboards without alerts — dashboards require someone to be looking at the right time.
Long-Term Storage and Multi-Cluster Setups with Thanos
A single Prometheus instance has two structural limitations worth knowing about before you’re surprised by them in production: local storage isn’t durable beyond the retention window you configure, and it only sees a single cluster. Once you’re running more than one cluster, or need retention beyond a few weeks for compliance or trend analysis, Thanos (or Grafana Mimir, a similar alternative) becomes the natural next step. Thanos sits alongside your existing Prometheus instances via a sidecar, continuously uploading blocks of metric data to durable object storage (S3, GCS), and provides a global query layer that can transparently query across both recent local data and older data in object storage, and across multiple clusters simultaneously:
apiVersion: apps/v1
kind: Deployment
metadata:
name: prometheus
spec:
template:
spec:
containers:
- name: prometheus
image: prom/prometheus:latest
- name: thanos-sidecar
image: thanosio/thanos:latest
args:
- sidecar
- --tsdb.path=/prometheus
- --prometheus.url=http://localhost:9090
- --objstore.config-file=/etc/thanos/objstore.yaml
I wouldn’t reach for this on day one — it’s meaningfully more operational complexity than a single Prometheus instance — but it’s worth planning for once you outgrow a single cluster’s local retention.
Dashboards as Code
Manually clicking through Grafana’s UI to build dashboards works fine for one-off exploration, but for anything you want to keep and version-control, define dashboards as JSON and provision them via ConfigMap, which Grafana automatically picks up when configured with the sidecar provisioning pattern the kube-prometheus-stack chart ships by default:
apiVersion: v1
kind: ConfigMap
metadata:
name: myapp-dashboard
namespace: monitoring
labels:
grafana_dashboard: "1"
data:
myapp-dashboard.json: |
{
"title": "MyApp Overview",
"panels": [
{
"title": "Request Rate",
"type": "graph",
"targets": [
{ "expr": "sum(rate(http_requests_total{app=\"myapp\"}[5m]))" }
]
}
]
}
This keeps dashboards in git alongside the application code they monitor, reviewable in pull requests just like any other change, rather than living only as tribal knowledge inside someone’s Grafana account.
Practical Alerting Philosophy
Beyond the mechanics of PrometheusRule, the philosophy behind what to alert on matters more than most people initially expect. I try to follow a simple rule: every alert that pages someone should be actionable and should represent (or reliably predict) real user impact. Symptom-based alerts — elevated error rate, elevated latency, failed health checks — are almost always better first-line alerts than cause-based ones like raw CPU usage, because high CPU on its own often isn’t actually a problem, while elevated error rate always is. I keep cause-based metrics (CPU, memory, disk) as dashboards for investigation after a symptom-based alert fires, rather than as page-worthy alerts on their own. This single change — moving from “alert on everything that looks abnormal” to “alert on what actually hurts users, dashboard the rest” — has done more for alert fatigue on every team I’ve worked with than any amount of threshold tuning.
Resource Sizing for Prometheus Itself
Prometheus’s memory usage scales primarily with the number of unique time series (cardinality), not raw data volume, so it’s worth being deliberate about labels that could explode cardinality — raw user IDs, request IDs, or full URLs as label values are the classic mistakes that quietly turn a healthy Prometheus instance into one that OOMs regularly:
resources:
requests:
cpu: "1"
memory: 4Gi
limits:
cpu: "2"
memory: 8Gi
Start with generous limits, watch prometheus_tsdb_head_series over a few weeks of real traffic, and size down (or up) from observed reality rather than guessing up front.
Summary
Prometheus gives Kubernetes clusters the observability layer they need to catch problems before users do. The Prometheus Operator and kube-prometheus-stack make the setup largely declarative — install the chart, let it discover kubelet/node/kube-state-metrics targets automatically, add ServiceMonitor resources for your own apps, and layer PrometheusRule alerts with Alertmanager routing on top. Once it’s running, PromQL becomes the language you reach for whenever something in the cluster feels off.
