Offensive Security Researcher & Quantum Cryptography Analyst

How to Set Up Kubernetes Monitoring with Grafana

Abdul Wahab Junaid
How to Set Up Kubernetes Monitoring with Grafana

Nothing exposes an under-monitored cluster like an outage nobody saw coming. Kubernetes will happily reschedule crashing Pods for you all night without a single alert firing if there’s no monitoring stack in place. Grafana, paired with Prometheus as its data source, is the de facto standard for visualizing that data — this guide sets up the full pipeline from metrics collection to dashboards to alerts.

The Monitoring Stack, Conceptually

Grafana itself does not collect metrics — it’s a visualization and alerting layer on top of a time-series database. The standard combination is:

Installing the Stack via Helm

The kube-prometheus-stack chart bundles all of the above:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

kubectl create namespace monitoring

helm install kube-prometheus-stack prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --set grafana.adminPassword=changeme123 \
  --set prometheus.prometheusSpec.retention=15d

Verify the rollout:

kubectl -n monitoring get pods

Output:

NAME                                                     READY   STATUS
kube-prometheus-stack-grafana-6d7f9c8b7f-abcde           3/3     Running
kube-prometheus-stack-kube-state-metrics-7d9c8-fghij     1/1     Running
kube-prometheus-stack-prometheus-node-exporter-klmno     1/1     Running
prometheus-kube-prometheus-stack-prometheus-0            2/2     Running
alertmanager-kube-prometheus-stack-alertmanager-0        2/2     Running

Accessing Grafana

kubectl -n monitoring port-forward svc/kube-prometheus-stack-grafana 3000:80

Then log in at http://localhost:3000 with admin / the password set above. For production, expose it via an Ingress instead of port-forwarding:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: grafana
  namespace: monitoring
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
  ingressClassName: nginx
  tls:
    - hosts: ["grafana.example.com"]
      secretName: grafana-tls
  rules:
    - host: grafana.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: kube-prometheus-stack-grafana
                port:
                  number: 80

Prometheus as a Data Source

The Helm chart wires this up automatically, but the manual equivalent (Grafana provisioning YAML) looks like:

apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    url: http://kube-prometheus-stack-prometheus.monitoring.svc:9090
    access: proxy
    isDefault: true

Importing Dashboards

The stack ships several prebuilt dashboards, but the two most-used community dashboards worth importing by ID (via Dashboards → Import in the Grafana UI) are:

# No CLI equivalent for import — use the Grafana UI, entering the dashboard ID

Writing Custom PromQL Queries

A few examples that map directly to common day-to-day questions:

# Pod restart count over the last hour
increase(kube_pod_container_status_restarts_total[1h])

# CPU usage per pod, in cores
sum(rate(container_cpu_usage_seconds_total{namespace="production"}[5m])) by (pod)

# Memory usage vs limit ratio
sum(container_memory_working_set_bytes{namespace="production"}) by (pod)
  / sum(kube_pod_container_resource_limits{resource="memory", namespace="production"}) by (pod)

# Nodes not ready
kube_node_status_condition{condition="Ready", status="true"} == 0

Alerting

Prometheus Alertmanager (bundled with the stack) handles alert routing. A PrometheusRule CRD defines the alert conditions:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: pod-crashlooping
  namespace: monitoring
  labels:
    release: kube-prometheus-stack
spec:
  groups:
    - name: pod-health
      rules:
        - alert: PodCrashLooping
          expr: increase(kube_pod_container_status_restarts_total[15m]) > 3
          for: 5m
          labels:
            severity: warning
          annotations:
            summary: "Pod {{ $labels.pod }} is restarting frequently"
            description: "Namespace {{ $labels.namespace }}, container {{ $labels.container }}"

kubectl apply -f pod-crashlooping-rule.yaml

Route alerts to Slack via Alertmanager config:

apiVersion: v1
kind: Secret
metadata:
  name: alertmanager-config
  namespace: monitoring
stringData:
  alertmanager.yaml: |
    route:
      receiver: slack-notifications
    receivers:
      - name: slack-notifications
        slack_configs:
          - api_url: https://hooks.slack.com/services/XXX/YYY/ZZZ
            channel: "#alerts"

Monitoring Ingress and Services

If using NGINX Ingress, its own metrics endpoint can be scraped for request rate, latency, and error rate dashboards — useful for correlating application-level errors with cluster events:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: nginx-ingress
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: ingress-nginx
  endpoints:
    - port: metrics
      interval: 30s

High Availability for the Monitoring Stack Itself

A monitoring stack that goes down along with the cluster it’s watching defeats its own purpose. For real HA:

grafana:
  replicas: 2
  persistence:
    enabled: true
  env:
    GF_DATABASE_TYPE: postgres
    GF_DATABASE_HOST: postgres.monitoring.svc:5432

Dashboard Variables for Reusable Views

Hardcoding a namespace or Deployment name into a dashboard means building a new one for every application. Grafana template variables solve this:

Variable name: namespace
Type: Query
Data source: Prometheus
Query: label_values(kube_pod_info, namespace)

Once defined, any panel query can reference $namespace instead of a literal string:

sum(rate(container_cpu_usage_seconds_total{namespace="$namespace"}[5m])) by (pod)

A dropdown appears at the top of the dashboard, letting anyone switch between namespaces without touching a single query — the difference between one dashboard per team building their own copy-pasted variant, and one dashboard the whole organization shares.

Annotations for Correlating Deploys with Metrics

One of the most useful things a monitoring setup can show is when something changed, layered directly onto the metrics graphs. Grafana annotations, pushed automatically from a CI/CD pipeline, do exactly this:

curl -X POST http://grafana.example.com/api/annotations \
  -H "Authorization: Bearer $GRAFANA_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "text": "Deployed myapp v1.2.0",
    "tags": ["deploy", "myapp"],
    "time": '"$(date +%s000)"'
  }'

Adding this as a final CI stage means every dashboard viewer can instantly see “the error rate spiked right after this deploy” instead of separately cross-referencing deploy timestamps against a graph by hand.

Long-Term Storage Beyond Local Retention

Local Prometheus storage is deliberately not meant to hold years of history — it’s optimized for fast, recent queries. For longer retention (compliance, trend analysis, year-over-year comparisons), Thanos or Grafana Mirror are the standard extensions:

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: main
  namespace: monitoring
spec:
  retention: 6h
  thanos:
    objectStorageConfig:
      key: thanos.yaml
      name: thanos-objstore-config

Thanos sidecars ship blocks to object storage (S3, GCS) once they age out of local retention, and a Thanos Querier presents a unified view spanning both recent local data and older archived data — all queryable from the same Grafana data source without the dashboard author needing to know which layer actually holds a given time range.

Avoiding Alert Fatigue with SLO-Based Alerting

A monitoring stack that alerts on every minor blip trains people to ignore alerts entirely, which is arguably worse than having no alerting at all. A more durable approach ties alerts to Service Level Objectives — a defined error budget the service is allowed to consume before it’s actually a problem worth waking someone up for, rather than alerting on every individual metric threshold in isolation:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: myapp-slo
  namespace: monitoring
spec:
  groups:
    - name: slo-burn-rate
      rules:
        - alert: ErrorBudgetBurnFast
          expr: |
            (
              sum(rate(http_requests_total{status=~"5..", app="myapp"}[5m]))
              /
              sum(rate(http_requests_total{app="myapp"}[5m]))
            ) > (14.4 * 0.001)
          for: 2m
          labels:
            severity: critical
          annotations:
            summary: "myapp burning error budget fast — page immediately"

The 14.4 * 0.001 figure follows the widely used multi-window burn-rate approach popularized in Google’s SRE workbook — a fast burn rate over a short window pages immediately, while a slower burn rate over a longer window (a separate rule, typically checked hourly rather than every 5 minutes) generates a lower-urgency ticket instead. This two-tier structure is what separates a monitoring setup people actually trust from one that’s eventually muted at the OS notification level out of sheer alert volume.

Correlating Logs Alongside Metrics

Grafana increasingly serves as a single pane for logs too, when paired with Loki rather than a separate log-viewing tool entirely:

helm repo add grafana https://grafana.github.io/helm-charts
helm install loki grafana/loki-stack --namespace monitoring

Once Loki is added as a second Grafana data source, a dashboard panel showing a CPU or error-rate spike can link directly to the matching log lines for that exact time window and namespace — turning “the graph looks wrong” into “here’s the actual stack trace from that moment” without leaving Grafana at all.

Common Mistakes

Summary

Grafana turns Prometheus’s raw time-series data into something a human can actually act on: dashboards for at-a-glance health, PromQL for ad-hoc investigation, and Alertmanager-backed rules for catching problems before a customer does. The kube-prometheus-stack Helm chart gets the whole pipeline running in minutes; the real work is in tuning retention, dashboards, and alert thresholds to match what actually matters for your workloads.

References

Exit mobile version