How to Set Up Prometheus Alerting in Kubernetes

There’s a meaningful difference between “monitoring” and “alerting,” and I didn’t fully appreciate it until I was staring at a beautiful Grafana dashboard, fifteen minutes into an outage, realizing nothing had told me to look at it. Metrics existing somewhere is not the same as being notified when they cross a line that matters. This article is specifically about the Prometheus half of that pipeline — writing good alerting rules — with Alertmanager as the delivery mechanism downstream.

The Alerting Pipeline in Kubernetes

The flow has three distinct layers, and it’s worth being clear about where each one lives:

1. Prometheus scrapes metrics and continuously evaluates alerting rules (defined as PromQL expressions) against them.
2. When a rule’s condition is true for the configured duration, Prometheus fires an alert and sends it to Alertmanager.
3. Alertmanager groups, deduplicates, routes, and delivers notifications (Slack, PagerDuty, email, etc.).

This article focuses on step 1 and 2 — writing rules that are actually useful, not just technically correct.

Prerequisites

• Prometheus running in-cluster (via kube-prometheus-stack or standalone) with the Prometheus Operator, if using PrometheusRule CRDs
• Alertmanager configured and reachable (see the companion Alertmanager setup process)

Anatomy of an Alerting Rule

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: application-alerts
  namespace: monitoring
  labels:
    release: kube-prometheus
spec:
  groups:
  - name: application.rules
    rules:
    - alert: HighErrorRate
      expr: |
        sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)
        /
        sum(rate(http_requests_total[5m])) by (service)
        > 0.05
      for: 10m
      labels:
        severity: critical
      annotations:
        summary: "High error rate on {{ $labels.service }}"
        description: "{{ $labels.service }} has a {{ $value | humanizePercentage }} error rate over the last 5 minutes."

Breaking this down:

• expr — the PromQL expression evaluated on every rule-evaluation interval.
• for — the condition must be continuously true for this duration before the alert actually fires (state moves from pending to firing). This prevents alerting on brief blips.
• labels — used for routing decisions in Alertmanager (e.g., severity).
• annotations — human-readable context, supports templating with $labels and $value.

Alert States

Understanding the three states matters for reading dashboards correctly:

• Inactive — the expression isn’t true; nothing happening.
• Pending — the expression became true, but for duration hasn’t elapsed yet.
• Firing — the expression has been true for at least for duration; the alert is sent to Alertmanager.

kubectl port-forward -n monitoring svc/prometheus-operated 9090:9090

Check the Alerts tab in the Prometheus UI at localhost:9090/alerts to see live state for every rule.

Essential Kubernetes Alerting Rules

Here’s a practical, production-tested set covering the most common failure modes.

Pod and Container Health

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: kubernetes-pod-alerts
  namespace: monitoring
  labels:
    release: kube-prometheus
spec:
  groups:
  - name: kubernetes-pods
    rules:
    - alert: PodCrashLooping
      expr: rate(kube_pod_container_status_restarts_total[15m]) * 60 * 15 > 3
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "Pod {{ $labels.namespace }}/{{ $labels.pod }} crash looping"
        description: "Container {{ $labels.container }} has restarted more than 3 times in 15 minutes."

    - alert: PodStuckPending
      expr: kube_pod_status_phase{phase="Pending"} == 1
      for: 15m
      labels:
        severity: warning
      annotations:
        summary: "Pod {{ $labels.namespace }}/{{ $labels.pod }} stuck Pending"
        description: "Pod has been Pending for over 15 minutes — check scheduling constraints or resource capacity."

    - alert: ContainerOOMKilled
      expr: kube_pod_container_status_last_terminated_reason{reason="OOMKilled"} == 1
      for: 0m
      labels:
        severity: warning
      annotations:
        summary: "Container {{ $labels.container }} was OOMKilled"
        description: "Pod {{ $labels.namespace }}/{{ $labels.pod }} container {{ $labels.container }} was killed due to OOM."

    - alert: DeploymentReplicasMismatch
      expr: kube_deployment_spec_replicas != kube_deployment_status_replicas_available
      for: 15m
      labels:
        severity: warning
      annotations:
        summary: "Deployment {{ $labels.namespace }}/{{ $labels.deployment }} replica mismatch"
        description: "Deployment has {{ $labels.spec_replicas }} desired but fewer available for 15+ minutes."

Node Health

    - alert: NodeNotReady
      expr: kube_node_status_condition{condition="Ready", status="true"} == 0
      for: 10m
      labels:
        severity: critical
      annotations:
        summary: "Node {{ $labels.node }} not ready"
        description: "Node has been NotReady for over 10 minutes."

    - alert: NodeHighDiskUsage
      expr: (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}) < 0.1
      for: 10m
      labels:
        severity: warning
      annotations:
        summary: "Node {{ $labels.instance }} low on disk"
        description: "Less than 10% disk space remaining on root filesystem."

    - alert: NodeHighMemoryPressure
      expr: kube_node_status_condition{condition="MemoryPressure", status="true"} == 1
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "Node {{ $labels.node }} under memory pressure"

Resource Saturation

    - alert: NamespaceCPUQuotaHigh
      expr: |
        sum(rate(container_cpu_usage_seconds_total[5m])) by (namespace)
        /
        sum(kube_resourcequota{resource="cpu", type="hard"}) by (namespace)
        > 0.9
      for: 10m
      labels:
        severity: warning
      annotations:
        summary: "Namespace {{ $labels.namespace }} approaching CPU quota"
        description: "CPU usage is above 90% of the namespace's resource quota."

    - alert: PersistentVolumeUsageHigh
      expr: (kubelet_volume_stats_used_bytes / kubelet_volume_stats_capacity_bytes) > 0.9
      for: 10m
      labels:
        severity: warning
      annotations:
        summary: "PVC {{ $labels.persistentvolumeclaim }} almost full"
        description: "Volume is over 90% full in namespace {{ $labels.namespace }}."

API Server and Control Plane

    - alert: APIServerHighLatency
      expr: |
        histogram_quantile(0.99, sum(rate(apiserver_request_duration_seconds_bucket{verb!="WATCH"}[5m])) by (le, verb))
        > 1
      for: 10m
      labels:
        severity: warning
      annotations:
        summary: "Kubernetes API server high latency"
        description: "99th percentile request latency for {{ $labels.verb }} exceeds 1 second."

    - alert: APIServerErrorsHigh
      expr: |
        sum(rate(apiserver_request_total{code=~"5.."}[5m]))
        /
        sum(rate(apiserver_request_total[5m]))
        > 0.05
      for: 10m
      labels:
        severity: critical
      annotations:
        summary: "High API server 5xx error rate"

Apply everything:

kubectl apply -f kubernetes-pod-alerts.yaml

Confirm Prometheus picked the rules up:

kubectl port-forward -n monitoring svc/prometheus-operated 9090:9090

Check localhost:9090/rules to see all loaded rule groups.

Recording Rules: Precomputing Expensive Queries

Complex PromQL expressions evaluated repeatedly (by dashboards, alerts, or both) are wasteful to recompute from scratch each time. Recording rules precompute them into new time series:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: recording-rules
  namespace: monitoring
  labels:
    release: kube-prometheus
spec:
  groups:
  - name: recording.rules
    interval: 30s
    rules:
    - record: service:http_error_rate:ratio_5m
      expr: |
        sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)
        /
        sum(rate(http_requests_total[5m])) by (service)

Now your alerting rule (and any Grafana dashboard) can reference service:http_error_rate:ratio_5m directly, cheaper to query and consistent across every consumer.

    - alert: HighErrorRate
      expr: service:http_error_rate:ratio_5m > 0.05
      for: 10m
      labels:
        severity: critical

Testing Alerting Rules Before Deploying

Use promtool to unit test rules without waiting for real conditions to occur in production:

# tests.yaml
rule_files:
  - kubernetes-pod-alerts.yaml
evaluation_interval: 1m
tests:
  - interval: 1m
    input_series:
      - series: 'kube_pod_status_phase{namespace="default", pod="test-pod", phase="Pending"}'
        values: '1x20'
    alert_rule_test:
      - eval_time: 16m
        alertname: PodStuckPending
        exp_alerts:
          - exp_labels:
              severity: warning
              namespace: default
              pod: test-pod
            exp_annotations:
              summary: "Pod default/test-pod stuck Pending"

promtool test rules tests.yaml

This catches broken for durations, wrong thresholds, or PromQL syntax errors before they ever hit a real cluster.

CI/CD Integration

Lint and validate rules as part of your pipeline before merging:

promtool check rules kubernetes-pod-alerts.yaml

# .github/workflows/validate-alerts.yaml
name: Validate Prometheus Rules
on: [pull_request]
jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v4
    - name: Install promtool
      run: |
        wget https://github.com/prometheus/prometheus/releases/download/v2.53.0/prometheus-2.53.0.linux-amd64.tar.gz
        tar xzf prometheus-2.53.0.linux-amd64.tar.gz
        sudo mv prometheus-2.53.0.linux-amd64/promtool /usr/local/bin/
    - name: Validate rules
      run: promtool check rules monitoring/*.yaml

Multi-Window, Multi-Burn-Rate Alerts (SLO-Based Alerting)

Simple threshold alerts work fine for infrastructure-level issues, but for services with a defined SLO (Service Level Objective), a more sophisticated technique — multi-window, multi-burn-rate alerting — catches problems faster while generating fewer false positives than a flat threshold. The idea: alert when your error budget is burning fast enough over a short window AND a longer window simultaneously, which distinguishes real incidents from brief blips far better than either window alone.

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: slo-burn-rate-alerts
  namespace: monitoring
  labels:
    release: kube-prometheus
spec:
  groups:
  - name: slo.burnrate
    rules:
    - alert: ErrorBudgetBurnRateHigh
      expr: |
        (
          sum(rate(http_requests_total{status=~"5..", service="checkout"}[1h]))
          /
          sum(rate(http_requests_total{service="checkout"}[1h]))
          > (14.4 * 0.001)
        )
        and
        (
          sum(rate(http_requests_total{status=~"5..", service="checkout"}[5m]))
          /
          sum(rate(http_requests_total{service="checkout"}[5m]))
          > (14.4 * 0.001)
        )
      labels:
        severity: critical
      annotations:
        summary: "Checkout service burning error budget fast"
        description: "Error budget burn rate over both 1h and 5m windows exceeds 14.4x — at this rate the monthly SLO budget will be exhausted in hours, not weeks."

The 14.4 multiplier is a standard reference value (borrowed from widely published SRE burn-rate alerting guidance) representing a burn rate that would exhaust an entire month’s error budget in about 2 days if sustained — tune it against your own SLO target and budget window rather than copying it verbatim.

Silencing Known Issues During Deployments

A practical pattern for CI/CD pipelines is automatically creating a short-lived Alertmanager silence during a deployment window, so expected transient errors (brief 5xx spikes during a rolling update, for instance) don’t trigger a page for something already known and being actively managed:

amtool silence add \
  service="checkout" \
  --duration="10m" \
  --comment="Rolling deployment in progress - build ${CI_BUILD_ID}" \
  --alertmanager.url=http://alertmanager.monitoring.svc:9093

# .gitlab-ci.yml snippet
deploy:
  script:
    - ./scripts/create-deploy-silence.sh
    - kubectl apply -f deployment.yaml
    - kubectl rollout status deployment/checkout --timeout=300s

This is a deliberate, time-boxed, auditable silence tied to a specific deployment — very different from someone manually muting an alert indefinitely and forgetting to unmute it, which is one of the more common ways real incidents go unnoticed in practice.

Common Mistakes

• No for duration, causing alerts to fire on transient, self-resolving blips — extremely common cause of alert fatigue.
• Thresholds copied from a blog post without tuning to your actual traffic patterns — 5% error rate might be catastrophic for a payments API and completely normal noise for a best-effort internal tool.
• Missing severity labels, breaking Alertmanager’s routing tree entirely.
• Alerting on symptoms without annotations that explain next steps — a summary with no description leaves whoever’s paged guessing at 3 AM.
• Not testing rules with promtool before deploying, leading to silently broken alerts nobody notices until an incident happens with zero notification.
• Recording rule sprawl — creating precomputed series for queries that are cheap enough to compute directly just adds unnecessary cardinality.

Best Practices

• Alert on symptoms (error rate, latency, saturation) rather than causes when possible — causes change, symptoms are what actually hurt users.
• Every alert should be actionable — if there’s nothing a human can do in response, it probably belongs in a dashboard, not a page.
• Use consistent severity labels across all rule groups so Alertmanager routing stays predictable.
• Version-control all PrometheusRule manifests and review threshold changes like code, because they are code.

Summary

Good Prometheus alerting comes down to writing PromQL expressions that reflect real user-facing symptoms, giving them appropriate for durations to avoid noise, labeling them consistently for routing, and testing them with promtool before they ever reach production. Combined with recording rules for efficiency and CI validation to catch mistakes early, this turns your metrics pipeline into something that actually tells the right people the right thing at the right time — which is the entire point of collecting metrics in the first place.

References

• Prometheus Alerting Rules Documentation
• Prometheus Operator: PrometheusRule
• promtool Documentation
• CNCF Prometheus Project