You can’t operate a Kubernetes cluster responsibly without visibility into what’s actually happening inside it — CPU pressure, memory leaks, pod restarts, API server latency, all of it. Prometheus has become the de facto standard for Kubernetes monitoring, largely because it was built with the same “pull-based, label-oriented” philosophy as Kubernetes itself. In this guide, I’ll walk through installing Prometheus properly on EKS, understanding how it discovers targets, writing alerting rules, and connecting it to Grafana for visualization.
Why Prometheus Fits Kubernetes So Well
Prometheus scrapes metrics over HTTP from targets that expose a /metrics endpoint in its text-based exposition format. Combined with Kubernetes service discovery, Prometheus can automatically find every pod, service, and node in your cluster without hardcoded target lists — as pods come and go (which happens constantly in Kubernetes), Prometheus’s discovery mechanism keeps its scrape targets current.
The core components you’ll typically run:
- Prometheus Server — scrapes and stores time-series metrics
- Alertmanager — routes and deduplicates alerts, handles silencing/grouping
- kube-state-metrics — exposes Kubernetes object state (deployment replica counts, pod status, etc.) as metrics, since the API server itself doesn’t expose Prometheus-format metrics for object state
- node-exporter — a DaemonSet exposing host-level metrics (CPU, memory, disk, network) from every node
- Grafana — visualization layer on top of Prometheus’s data
Installing via the kube-prometheus-stack Helm Chart
Rather than assembling all these pieces manually, the community-maintained kube-prometheus-stack chart bundles Prometheus, Alertmanager, Grafana, kube-state-metrics, and node-exporter with sane defaults and pre-built dashboards.
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
kubectl create namespace monitoring
A production-oriented values.yaml:
prometheus:
prometheusSpec:
retention: 15d
resources:
requests:
cpu: 500m
memory: 2Gi
limits:
cpu: 1
memory: 4Gi
storageSpec:
volumeClaimTemplate:
spec:
storageClassName: gp3
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 50Gi
alertmanager:
alertmanagerSpec:
storage:
volumeClaimTemplate:
spec:
storageClassName: gp3
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 5Gi
grafana:
adminPassword: "changeme-use-a-secret"
persistence:
enabled: true
storageClassName: gp3
size: 10Gi
nodeExporter:
enabled: true
kubeStateMetrics:
enabled: true
helm install kube-prometheus-stack prometheus-community/kube-prometheus-stack \
--namespace monitoring \
-f values.yaml
Verify everything came up:
kubectl get pods -n monitoring
NAME READY STATUS RESTARTS AGE
alertmanager-kube-prometheus-stack-alertmanager-0 2/2 Running 0 2m
kube-prometheus-stack-grafana-6d9f8b7c5d-abc12 3/3 Running 0 2m
kube-prometheus-stack-kube-state-metrics-7b9c6d4f8-xy 1/1 Running 0 2m
kube-prometheus-stack-operator-5f6d8b9c7-mn0op 1/1 Running 0 2m
kube-prometheus-stack-prometheus-node-exporter-abcde 1/1 Running 0 2m
prometheus-kube-prometheus-stack-prometheus-0 2/2 Running 0 2m
For production, don’t store the Grafana admin password in values.yaml in plaintext — reference a Kubernetes Secret instead, or better, integrate with AWS Secrets Manager via the External Secrets Operator.
Accessing Prometheus and Grafana
kubectl port-forward -n monitoring svc/kube-prometheus-stack-prometheus 9090:9090
kubectl port-forward -n monitoring svc/kube-prometheus-stack-grafana 3000:80
For persistent access, expose via Ingress with the AWS Load Balancer Controller:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: grafana
namespace: monitoring
annotations:
kubernetes.io/ingress.class: alb
alb.ingress.kubernetes.io/scheme: internal
alb.ingress.kubernetes.io/target-type: ip
alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:us-east-1:123456789012:certificate/xxxx
spec:
rules:
- host: grafana.internal.company.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: kube-prometheus-stack-grafana
port:
number: 80
I’ve marked this scheme: internal deliberately — monitoring dashboards should almost never be exposed to the public internet directly; put them behind a VPN or internal load balancer.
Understanding Kubernetes Service Discovery
The Prometheus Operator (installed by the chart) introduces CRDs — ServiceMonitor and PodMonitor — that declaratively tell Prometheus what to scrape, instead of hand-editing prometheus.yml.
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: web-app
namespace: production
labels:
release: kube-prometheus-stack
spec:
selector:
matchLabels:
app: web-app
namespaceSelector:
matchNames:
- production
endpoints:
- port: metrics
interval: 30s
path: /metrics
This assumes your application’s Service exposes a port named metrics:
apiVersion: v1
kind: Service
metadata:
name: web-app
namespace: production
labels:
app: web-app
spec:
selector:
app: web-app
ports:
- name: metrics
port: 9100
targetPort: 9100
The release: kube-prometheus-stack label on the ServiceMonitor is important — by default, the Prometheus Operator only picks up ServiceMonitors matching the label selector configured on the Prometheus CRD, which the Helm chart sets to match its own release name.
Writing Custom Alerting Rules
PrometheusRule CRDs define alerting logic that gets loaded into Prometheus automatically:
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: web-app-alerts
namespace: monitoring
labels:
release: kube-prometheus-stack
spec:
groups:
- name: web-app.rules
rules:
- alert: HighErrorRate
expr: |
sum(rate(http_requests_total{app="web-app",status=~"5.."}[5m]))
/
sum(rate(http_requests_total{app="web-app"}[5m])) > 0.05
for: 5m
labels:
severity: critical
annotations:
summary: "High error rate on web-app"
description: "Error rate has exceeded 5% for 5 minutes."
- alert: PodMemoryUsageHigh
expr: |
container_memory_working_set_bytes{namespace="production",pod=~"web-app.*"}
/ container_spec_memory_limit_bytes{namespace="production",pod=~"web-app.*"} > 0.9
for: 10m
labels:
severity: warning
annotations:
summary: "Pod {{ $labels.pod }} approaching memory limit"
- alert: KubePodCrashLooping
expr: |
rate(kube_pod_container_status_restarts_total{namespace="production"}[15m]) > 0
for: 5m
labels:
severity: warning
annotations:
summary: "Pod {{ $labels.pod }} is crash looping"
Apply it and verify Prometheus picked it up:
kubectl apply -f web-app-alerts.yaml
kubectl port-forward -n monitoring svc/kube-prometheus-stack-prometheus 9090:9090
# then visit http://localhost:9090/alerts
Configuring Alertmanager Routing
apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig
metadata:
name: web-app-routing
namespace: monitoring
labels:
alertmanagerConfig: main
spec:
route:
groupBy: ["alertname", "namespace"]
groupWait: 30s
groupInterval: 5m
repeatInterval: 4h
receiver: slack-critical
routes:
- matchers:
- name: severity
value: critical
receiver: slack-critical
- matchers:
- name: severity
value: warning
receiver: slack-warnings
receivers:
- name: slack-critical
slackConfigs:
- apiURL:
name: slack-webhook
key: url
channel: "#alerts-critical"
- name: slack-warnings
slackConfigs:
- apiURL:
name: slack-webhook
key: url
channel: "#alerts-warnings"
Application Instrumentation Example (Node.js)
For your own applications to be scrapable, expose a /metrics endpoint using a client library:
const client = require('prom-client');
const express = require('express');
const app = express();
const register = new client.Registry();
client.collectDefaultMetrics({ register });
const httpRequestDuration = new client.Histogram({
name: 'http_request_duration_seconds',
help: 'Duration of HTTP requests in seconds',
labelNames: ['method', 'route', 'status'],
});
register.registerMetric(httpRequestDuration);
app.get('/metrics', async (req, res) => {
res.set('Content-Type', register.contentType);
res.end(await register.metrics());
});
app.listen(9100);
Key Cluster-Level Queries to Know
# Nodes under memory pressure
kube_node_status_condition{condition="MemoryPressure",status="true"}
# Pods pending for more than 5 minutes
(time() - kube_pod_created{phase="Pending"}) > 300
# Cluster-wide CPU utilization vs allocatable
sum(rate(container_cpu_usage_seconds_total[5m])) / sum(kube_node_status_allocatable{resource="cpu"})
# Top 5 pods by memory usage
topk(5, container_memory_working_set_bytes{namespace!="kube-system"})
Long-Term Storage and Cost Control on EKS
Prometheus’s local storage isn’t meant for long-term retention at scale — 15 days is a reasonable default. For longer retention or multi-cluster aggregation, integrate with a remote-write-compatible backend:
prometheus:
prometheusSpec:
remoteWrite:
- url: "https://aps-workspaces.us-east-1.amazonaws.com/workspaces/ws-xxxx/api/v1/remote_write"
sigv4:
region: us-east-1
This example points at Amazon Managed Service for Prometheus (AMP), which is often the pragmatic choice on EKS — it removes the operational burden of scaling Prometheus storage yourself and integrates with IAM for auth.
Federation and Multi-Cluster Monitoring
Once you’re running more than one EKS cluster, a common question is how to get a single-pane-of-glass view without duplicating dashboards per cluster. Two common approaches:
Prometheus federation — a central Prometheus scrapes aggregated metrics from each cluster’s local Prometheus:
- job_name: 'federate-cluster-a'
scrape_interval: 30s
honor_labels: true
metrics_path: '/federate'
params:
'match[]':
- '{job="kubernetes-pods"}'
- '{__name__=~"kube_.*"}'
static_configs:
- targets:
- 'prometheus.cluster-a.internal:9090'
Remote write to a shared backend — each cluster’s Prometheus (or the Prometheus Operator’s remoteWrite config shown earlier) pushes to a central store like Amazon Managed Service for Prometheus, Thanos, or Cortex, which handles multi-cluster aggregation and long-term storage natively. For most teams running more than 2-3 clusters, remote write to AMP is simpler operationally than maintaining federation hierarchies by hand.
Dashboards as Code
Manually clicking together Grafana dashboards doesn’t scale, and dashboards built by hand in the UI are easy to lose. Store dashboard JSON in Git and provision it via ConfigMap, which the kube-prometheus-stack chart’s Grafana sidecar automatically picks up:
apiVersion: v1
kind: ConfigMap
metadata:
name: web-app-dashboard
namespace: monitoring
labels:
grafana_dashboard: "1"
data:
web-app-dashboard.json: |
{
"title": "Web App Overview",
"panels": [
{
"title": "Request Rate",
"targets": [{"expr": "sum(rate(http_requests_total{app=\"web-app\"}[5m]))"}]
}
]
}
The grafana_dashboard: "1" label is what the sidecar watches for — any ConfigMap carrying it gets auto-imported into Grafana without manual UI work, and because it’s a ConfigMap, it’s naturally versioned alongside the rest of your manifests in Git.
Troubleshooting
# Check Prometheus targets and their scrape health
kubectl port-forward -n monitoring svc/kube-prometheus-stack-prometheus 9090:9090
# visit http://localhost:9090/targets
# Check Prometheus Operator logs if ServiceMonitors aren't being picked up
kubectl logs -n monitoring -l app=kube-prometheus-stack-operator
# Confirm a target's /metrics endpoint is actually reachable
kubectl run -it --rm debug --image=curlimages/curl --restart=Never -- curl http://web-app.production.svc:9100/metrics
A very common failure mode: a ServiceMonitor created without the release label the Operator expects, so it’s silently ignored. Always check kubectl get prometheus -n monitoring -o yaml for the serviceMonitorSelector to confirm what label it’s actually watching for.
Best Practices
- Set explicit resource requests/limits on the Prometheus pod itself — an unbounded Prometheus under high cardinality can OOM and take monitoring down exactly when you need it most.
- Watch cardinality carefully — unbounded label values (like raw user IDs in a metric label) can silently explode Prometheus’s memory usage.
- Use
for:durations on alerts to avoid flapping/noisy pages from transient blips. - Route alerts by severity to different channels so critical pages don’t get lost in a warnings firehose.
- Consider Amazon Managed Service for Prometheus for multi-cluster or long-retention needs rather than self-managing Prometheus storage scaling.
Summary
Prometheus, paired with kube-state-metrics and node-exporter, gives you comprehensive visibility into both Kubernetes object state and node-level resource usage, and its label-based pull model maps naturally onto Kubernetes’ own dynamic, ephemeral workload model. The kube-prometheus-stack Helm chart is the fastest path to a working, production-capable setup, and the Prometheus Operator’s CRDs (ServiceMonitor, PodMonitor, PrometheusRule, AlertmanagerConfig) let you manage scrape configs and alerting rules declaratively alongside your application manifests. On EKS, consider Amazon Managed Service for Prometheus once your retention or multi-cluster aggregation needs outgrow a single in-cluster Prometheus instance.
